Geody Labs


# Main Index: Debian Linux Magic Spells Cheat Sheet (one liners, how to, tips and tricks)

# File and Directory Operations

File status and information:
file PATH/FILE # identify file type
stat PATH/FILE # Show file properties (file status)
stat --printf="%U" PATH/FILE # Get user name of the owner of the file
stat --printf="%u" PATH/FILE # Get user ID of the owner of the file
stat --printf="%s" PATH/FILE # Get file size
wc -c PATH/FILE | awk '{print $1}' | tr -d '\n' # Get file size
basename PATH/FILE # return the file name (FILE) of the specified FILE
s1='PATH/FILE'; echo ${s1#$(dirname "$s1")/}; # return the file name (FILE) of the specified FILE
dirname PATH/FILE # return the path (from root) of the specified FILE
realpath PATH/FILE # Get the full path of a file
s1='DIR1/DIRN/FILE'; echo ${s1#$(dirname "$(dirname "$s1")")/}; # return the last directory (DIRN) and the file name (FILE) of the specified FILE

Search for a file:
find PATH -type f # find all files in PATH and its subdirectories
find PATH -type d # find all subdirectories in PATH and its subdirectories
find PATH -name FILE_MASK # find all files and directories with given mask inside PATH and its subdirectories. Example: find / -name a* # find all files and directories starting with a in the server
find PATH -type f -name FILE_MASK # find all files with given mask inside PATH and its subdirectories. Example: find / -type f -name *.php # find any PHP file in the server
find PATH -type d -name FILE_MASK # find all directories with given mask inside PATH and its subdirectories. Example: find / -type d -name test* # find all test directories in the server
find PATH -type f -regex ".*\.EXT$" # Find all files in PATH and its subdirectories with given EXTension
find PATH -type f -regex ".*\.[0-9]*$" # Find all files in PATH and its subdirectories with a numeric extension
whereis FILENAME # return the paths of files with matching FILENAME
locate NAME # return all paths and files with a matching NAME string
grep -r "STRING" PATH # search for all files containing the given STRING, within the PATH and its subdirectories (-r)
grep -r -l "STRING" PATH # search for all files containing the given STRING, within the PATH and its subdirectories (-r) and return just matching files name and not each occurrence of the string within every file (-l)
find PATH | xargs grep "STRING" -sl # search for all files containing the given STRING, within the PATH and its subdirectories, returns only filenames with path
find PATH -type f -newermt "2019-01-01" \! -newermt "2020-09-01" -exec ls -ld {} \; # Find all files in PATH and its subdirectories created between the two given dates
find PATH -type f -newermt "2019-01-01" \! -newermt "2020-09-01" -exec ls -ld {} \; # Find all subdirectories in PATH and its subdirectories created between the two given dates

Create an empty file:
touch PATH/NAME # Create an empty file. If the given file already exists, it updates its access date
touch PATH/FILENAME_$(date +"%Y-%m-%d").log # Create an empty file appending current date in YYYY-MM-DD format to the file name, and .log extension (you can add any extension or no extension at all)

Create a dummy file:
fallocate -l 1K PATH/NAME # generate a dummy file sized 1Kbyte (doesn't work on ext3 and ZFS, and could contain sensible information from previously unallocated data of deleted files)
dd if=/dev/zero of=PATH/NAME bs=1K count=2 # generate a dummy file sized 2Kbyte and filled with zeroes (slower than fallocate, but it does work on ext3 and ZFS and the output file doesn't contain unwanted data)
tr '\000' $'\xff' < /dev/zero | dd of=PATH/NAME bs=1K count=1 # generate a dummy file sized 1Kbyte and filled with hex FF (slower than fallocate, but it does work on ext3 and ZFS and the output file doesn't contain unwanted data)
dd if=/dev/random of=PATH/NAME bs=1K count=2 # generate a dummy file sized 2Kbyte and filled with random data (slower than fallocate, but it does work on ext3 and ZFS and the output file doesn't contain unwanted data)

Create a unique temporary file or directory with a pseudo random name:
mktemp "tmp_XXXXXX" # Create an empty file with a random name generated from the given mask, for example tmp_92pIKB or tmp_Z1Pdz1
mktemp -d "tmp_XXXXXX" # Create an empty directory with a random name generated from the given mask, for example tmp_olD9rA or tmp_5Fw0JI
Sample usage of mktemp (from mktemp man):
TMPFILE=`mktemp /tmp/example.XXXXXX` || exit 1
echo "program output" >> $TMPFILE

Symlinks
# Symlinks / Sym links / Symbolic links / Soft links
ln -s /PATH/TARGET_FILE /PATH/SYMLINK # Create a SYMLINK to TARGET_FILE (actual file). Permissions for the file will be the ones for TARGET_FILE, regardless of the ones for SYMLINK. Use rm to remove the SYMLINK (TARGET_FILE will not be affected)
find . -type l ! -exec test -e {} \; -print # Find broken symlinks

Copy a file or a directory:
cp -rp SOURCE DESTINATION # copy a file or a directory tree preserving owners and properties
mkdir -p PATH && cp -rp SOURCE $_ # copy a file or a directory tree (preserving owners and properties) and create destination path if it doesn't exist

Move a file or a directory tree (or rename):
mv SOURCEPATH/NAME DESTPATH/NAME
for a in *; do mv -v "$a" "${a//STRING_SEARCH/STRING_REPLACE}"; done  # batch rename all files in the current directory
for a in *; do mv -v "$a" "${a}_`date --iso-8601`"; done  # append date as YYYY-MM-DD to all file names in the current directory (note that it will be appended after any extension)
find PATH -type f -name "*" -exec mv "{}" "{}".txt \; # add an extension (.txt in this example) to all files in given PATH
find PATH -type f -not -name "*.*" -exec mv "{}" "{}".txt \; # Add an extension (.txt in this example) to all files without an extension in given PATH
for i in `ls PATH -1`; do mv $i "${i,,}" ; done # rename all files in the given PATH to lowercase
for i in `ls PATH -1`; do mv $i "${i^^}" ; done # rename all files in the given PATH to uppercase
for f in * ; do mv "$f" "$f.txt"; done # add .txt extension (suffix) to all files (in the current directory)
for f in *.txt ; do mv "$f" "${f%txt}asc"; done # change extension (suffix) from .txt to .asc
for f in *.txt ; do mv "$f" "${f%.txt}"; done # remove .txt extension (suffix)
for f in * ; do mv "$f" "prefix_$f"; done # add _prefix prefix to all files (in the current directory)
for f in prefix_* ; do mv "$f" "${f#prefix_}"; done; # remove prefix_
for f in prefix_* ; do mv "$f" "x_${f#prefix_}"; done; # replace prefix_ with x_

Synchronize two directories:
apt-get install rsync
rsync -a -v /SOURCE_PATH/* /DEST_PATH/ # copy all files from /SOURCE_PATH that doesn't exist in /DEST_PATH
rsync -a --delete -v /SOURCE_PATH/* /DEST_PATH/ # copy all files from /SOURCE_PATH that doesn't exist in /DEST_PATH, and delete all files in /DEST_PATH that doesn't (no longer) exist in /SOURCE_PATH
rsync -a --delete -v rsync://www.example.com/dir/ /DEST_PATH # synchronize from an external rsync directory

Empty files:
# Delete the content of a file without removing the file. It can be useful to don't break programs which expect to find the file, or before to delete a very large file.
cp /dev/null > PATH/NAME # Empty a file. It will also work as "> PATH/NAME" (without quotes)
echo -n '' > PATH/NAME # Empty a file
find PATH -type f -exec cp /dev/null {} \; # Empty all files in PATH and its subdirectories
find PATH -type f -regex ".*\.EXT$" -exec cp /dev/null {} \; # Empty all files in PATH and its subdirectories with given EXTension
find PATH -type f -regex ".*\.[0-9]*$" -exec cp /dev/null {} \; # Empty all files in PATH and its subdirectories with a numeric extension

Delete files:
rm PATH/NAME
rm -i PATH/NAME # ask for confirmation before to delete
find PATH -maxdepth 1 -type f -empty -delete # Delete all empty files in the given PATH (also: find PATH -maxdepth 1 -type f -size 0 -delete )
find PATH -maxdepth 1 -type f ! -empty -delete # Delete all non empty files in the given PATH (also: find PATH -maxdepth 1 -type f -size +0 -delete )
find PATH -type f -empty -delete # Delete all empty files in the given PATH and its subdirectories (also: find PATH -type f -size 0 -delete )
find PATH -type f ! -empty -delete # Delete all non empty files in the given PATH and its subdirectories (also: find PATH -type f -size +0 -delete )
find /var/www/test/t5 -type f -delete # Delete all files in /var/www/test/t5 and its subdirectories
find /var/www/test/t5 -type f -regex ".*\.EXT$" -delete # Delete all files in /var/www/test/t5 and its subdirectories with given EXTension
find /var/www/test/t5 -type f -regex ".*\.[0-9]*$" -delete # Delete all files in /var/www/test/t5 and its subdirectories with a numeric extension

Wipe a file (secure file deletion):
# Important note: wiping overwrites a file one or more time before to delete it, to make it unrecoverable. This technique does not work with journaled file systems, like ReiserFS, Reiser4, or ext3 in journaled mode. It also doesn't work with solid state drives (such as SD cards).
apt-get install wipe # if it doesn't work, try apt-get install secure-delete
wipe PATH/FILE # wipe a file (asks for confirmation, write protected files will be spared)
wipe -c -f PATH/FILE # wipe a file
wipe -c -f -s PATH/FILE # wipe a file, quiet mode (no output returned except for fatal errors like "No such file or directory")
wipe -c -f -k PATH/FILE # wipes the content of a file, but keeps the file in place (with a lenght of 0 bytes)
wipe -c -f -e PATH/FILE # wipes the file up to its exact size, without wiping extra data in the last used block

Undelete a file:
apt-get install testdisk # PhotoRec
photorec MEDIA # attempt to recover deleted files from given MEDIA (for example /dev/sda2 ). You'll have then to confirm the media, choose partition, select recovery options, specify the file type of the files you want to recover, specify the filesystem type where the files were stored, choose if you want to search for deleted files in the empty space or the whole partition, choose the destination directory (in a separate disk) where you want to save the recovered files.

Create a directory:
mkdir PATH
mkdir -p PATH # Create the whole path (not just the final directory in the path) if it doesn't exist

Delete a directory:
rmdir PATH # Delete an empty directory with no subdirectories
rmdir -r PATH # Delete an empty directory with empty subdirectories (recursive)
find PATH -depth -type d -empty -exec rmdir {} \; # Delete an empty directory with empty subdirectories (recursive)

Prune a directory tree (remove a directory with all its subdirectories, even if not empty, recursive):
rm -rf PATH ; rm -rf PATH/.* # Prune a directory tree
mkdir /tmp/empty/ ; rsync -a --delete /tmp/empty/ PATH/DIRECTORY_TO_REMOVE/ ; rmdir PATH/DIRECTORY_TO_REMOVE # Delete a directory or prune a directory tree with a large number (thousands and more) of files. Note: you can use any empty directory, just make sure it's actually empty.

Wipe a directory tree (recursive secure file deletion):
apt-get install wipe
wipe -c -r PATH # wipe a directory tree (asks for confirmation)
wipe -c -f -r PATH # wipe a directory tree
wipe -c -f -r -s PATH # wipe a directory tree, quiet mode (no output returned except for fatal errors like "No such file or directory")

Show current directory:
pwd

Change directory:
cd PATH
cd .. # go to the parent directory (that is, one directory up)
cd - # go to the previous directory (which is stored in $OLDPWD)
echo $OLDPWD # see previous directory
pushd PATH # change directory and remember the current directory in the dirs stack
popd # return to the previous directory in the dirs stack
dirs # (or dirs -l ) list directories in the dirs stack keeping the output in a single line
dirs -p # list directories in the dirs stack showing the output in multiple lines
dirs -v # list directories in the dirs stack showing the output in multiple lines and line numbers
dirs -c # clear the dirs stack

Show the content of a directory:
ls PATH
ls -l -d PATH # show only the properties of a directory, not its content.
ls -laF --color=auto PATH # extended information and colored output
ls -laFXB PATH # sort by extension
ls -laFSr PATH # sort by size, small to large
ls -laFS PATH # sort by size, large to small
ls -laFtr PATH # sort by size, old to recent
ls -laFt PATH # sort by date, recent to old
ls -laFtur PATH # sort by access date, old to recent
ls -laFtu PATH # sort by access date, recent to old
ls -A # show hidden files
ls --group-directories-first # show directories first
ls -d */. # list only directories within current directory
ls -d PATH/*/. # list only directories within given PATH
ls -R PATH | grep ":$" | sed -e 's/:$//' -e 's/[^-][^\/]*\//--/g' -e 's/^/ /' -e 's/-/|/' # show directory tree (add | less at the end of the line to navigate the tree up and down)
find PATH -ls # The -ls parameters make find return results as the output of ls . You may also add the -print0 parameter and pipe to xargs -0 ls -lt (that is: -print0 | xargs -0 ls -lt )
find PATH -type f -mtime -30 # list files modified within last 30 days
find PATH -depth -type d -empty -exec echo {} \; # list empty subdirectories within given PATH (recursive)
find PATH -maxdepth 1 -type f -empty # Find all empty files in the given PATH (also: find PATH -maxdepth 1 -type f -size 0 )
find PATH -maxdepth 1 -type f ! -empty # Find all non empty files in the given PATH (also: find PATH -maxdepth 1 -type f -size +0 )
find PATH -type f -empty # Find all empty files in the given PATH and its subdirectories (also: find PATH -type f -size 0 )
find PATH -type f ! -empty # Find all non empty files in the given PATH and its subdirectories (also: find PATH -type f -size +0 )
find PATH/DIRECTORY -type f | perl -ne 'print $1 if m/\.([^.\/]+)$/' | sort -u # return all different file extensions in given DIRECTORY

Compare and show differences between two directories:
diff PATH/DIR1 PATH/DIR2 # Report different files between two directories
diff -q PATH/DIR1 PATH/DIR2 # Report different files between two directories without showing the different content of files with identical file names
diff -s PATH/DIR1 PATH/DIR2 # Report different files and identical files between two directories
diff --ignore-file-name-case PATH/DIR1 PATH/DIR2 # Report different files between two directories ignoring case differences in file names
diff -i PATH/DIR1 PATH/DIR2 # Report different files between two directories ignoring case differences in the content of text files
diff -r PATH/DIR1 PATH/DIR2 # Report different files between two directories and all their subdirectories
dir1="PATH/DIR1"; dir2="PATH/DIR2"; diff -q "$dir1" "$dir2" | grep "$dir2" | sed "s/$dir2\///g" | awk '{print $4}' # Report different files and files that are only present in DIR2

Pick a random file from a directory:
ls DIR | shuf -n 1 # Pick a random file from the given DIR directory
ls DIR -pR | grep -v / | sed '/^$/d' | shuf -n 1 # Pick a random file from the given DIR directory and its subdirectories
find DIR -maxdepth 1 -type f | shuf -n 1 # Pick a random file (including path) from the given DIR directory
find DIR -type f | shuf -n 1 # Pick a random file (including path) from the given DIR directory and its subdirectories

Count files and subdirectories within directories:
ls -1 /PATH/ | wc -l # count files and subdirectories within a directory
ls -1R /PATH/ | wc -l # count files and subdirectories within a directory including subdirectories (recursively)
ls -1 /PATH/ | grep -i .*.EXT | wc -l # count files and subdirectories with a given EXTension within a directory
ls -1R /PATH/ | grep -i .*.EXT | wc -l # count files and subdirectories with a given EXTension within a directory including subdirectories (recursively)

Count files within directories:
find PATH -maxdepth 1 -type f | wc -l # count files within a directory
find PATH -type f | wc -l # count files within a directory including subdirectories (recursively)
find PATH -maxdepth 1 -type f | grep -i .*.EXT | wc -l # count files with a given EXTension within a directory
find PATH -type f | grep -i .*.EXT | wc -l # count files with a given EXTension within a directory including subdirectories (recursively)

Show open files:
apt-get install lsof
lsof # Show all open files
lsof PATH # Show all open files within PATH
lsof / | awk '{ if($7 > 1048576) print $7/1048576 "MB" " " $9 " " $1 }' | sort -n -u | tail --lines=10 # show the 10 largest open files with their size in MB and the processes that keep them open

Show deleted files locked by a running process:
# If you can't see more free space in your disk after deleting a large file it's probably locked by a running process so that it has been unlinked but not actually deleted yet. In this case you'll have to kill, stop, or restart the process to unlock the file and free its disk space
lsof +L1

Ownership and Permissions

Change access permissions: # Users: uuugggooo (ugo = u: owner user, g: all the users of the group, o: all other users) # Modes: 0: ---, 1: --x, 2: -w-, 3: -wx, 4: r--, 5: r-x, 6: rw- , 7: rwx chmod MOD FILE # Change permissions for a file chmod -v MOD FILE # Change permissions for a file, with verbose output chmod -R MOD DIRECTORY # Change permissions for a directory tree (recursive) chmod -vR MOD DIRECTORY # Change permissions for a directory tree (recursive), with verbose output find PATH -type f -exec chmod MOD {} + # Change permissions for all files (only files, not directories) in a directory tree (recursive) find PATH -type d -exec chmod MOD {} + # Change permissions for all directories (only directories, not files) in a directory tree (recursive) Change owner: chown OWNER FILE # Change the owner of a file chown OWNER:GROUP FILE # Change the owner and the group of a file chown -R OWNER DIRECTORY # Change the owner of a directory tree (recursive) chown -R OWNER:GROUP DIRECTORY # Change the owner and a group of a directory tree (recursive) Change group: chgrp OWNER FILE # Change the group of a file chgrp -R OWNER DIRECTORY # Change the group of a directory tree (recursive) File Permissions Backup and Restore getfacl -R / > PATH/fsperms.txt # Back up file system permissions to a text file setfacl --restore=PATH/fsperms.txt # Restore file system permissions from a text file

Operations with Files and Directories on a remote system

Copy a file over a network

apt install netcat-openbsd # this is a rewrite of netcat, which provides more features, including support for IPv6, proxies, and Unix sockets. # apt install netcat-traditional # this is the classic version of netcat that has less features than netcat-openbsd # On the Destination computer (listener). Make sure the PORT you are listening isn't blocked. Timeout (-w) is set to 3 seconds: nc -vvn -l -p PORT -w 3 > DEST_PATH/FILE # On the Source computer (sender). Make sure the PORT you are sending to isn't blocked. It will quit 2 seconds after receiving an EOF. If -q is not specified, then netcat must be killed manually: nc -vvn DESTINATION_IP PORT -q 2 < SOURCE_PATH/FILE

Copy a file securely (through SSH) over a network

# You will have to authenticate on the remote host. # Use blowfish encryption (-c blowfish) instead of the default TripleDES (3DES) because it's more secure and faster. scp -c blowfish SOURCEPATH/NAME DESTUSER@DESTHOST:DESTPATH/NAME # copies a file from the local system to a remote host scp -c blowfish SOURCEUSER@SOURCEHOST:DSOURCEPATH/NAME DESTPATH/NAME # copies a file from a remote host to the local system scp -c blowfish SOURCEUSER@SOURCEHOST:DSOURCEPATH/NAME DESTUSER@DESTHOST:DESTPATH/NAME # copies a file from a remote host to another remote host scp -c blowfish -r SOURCEUSER@SOURCEHOST:DSOURCEPATH DESTUSER@DESTHOST:DESTPATH/ # copies a directory tree from a remote host to another remote host

How to mount a remote directory tree

apt-get install sshfs fuse-utils To let other users to handle the remote directory tree: # create a group called fuse containing users allowed to mount the remote directory tree chgrp fuse /usr/bin/fusermount chmod u+s /usr/bin/fusermount adduser USER fuse Create the local directory where the remote directory tree will be mounted: mkdir /mnt/sshdir chown USER /mnt/sshdir Mount a remote directory tree: sshfs USER@HOST:/PATH /mnt/sshdir




Please DONATE to support the development of Free and Open Source Software (PayPal, Credit Card, Bitcoin, Ether)

Page issued on 25-Sep-2022 05:10 GMT
Copyright (c) 2022 Geody - Legal notices: copyright, privacy policy, disclaimer