Geody Labs


# Main Index: Debian Linux Magic Spells Cheat Sheet (one liners, how to, tips and tricks)

# File Manipulation

Split a file in chunks:
split -b=BYTES PATH/FILE PREFIX # split FILE in chunks of BYTES bytes, using PREFIX to name generated chunks
split -l LINES PATH/FILE.txt PREFIX # split PATH/FILE.txt in chunks of LINES lines, using PREFIX to name generated chunks

Rebuild a split file (concatenate files):
cat PREFIX* > PATH/FILE
sed -e '$a\' test?.txt > testx1b.txt # concatenate text files appending an empty line at the end of every line if there isn't already one
(for r in PREFIX*;do sed s/\$/\ $r/ < "$r";done) > PATH/FILE # concatenate text files appending the file name after each line
cat $(ls PREFIX* -t) > PATH/FILE # concatenate files sorted by date and time (newer first, older last)
cat $(ls PREFIX* -tr) > PATH/FILE # concatenate files sorted by date and time (older first, newer last)

Truncate a file:
cp /dev/null PATH/FILE # empty FILE. If FILE doesn't exist, it will be created.
truncate -s 0 PATH/FILE # truncate FILE to 0 bytes (empty FILE). If FILE doesn't exist, it will be created.
truncate -s 0 -c PATH/FILE # truncate FILE to 0 bytes (empty FILE). If FILE doesn't exist, it will not be created.
truncate -s 100 PATH/FILE # truncate FILE to 100 bytes. If FILE doesn't exist, or it's shorter than the given size, it will be filled by zeroes (0).
truncate -r PATH/REF_FILE PATH/FILE # truncate FILE to the size of REF_FILE.

Split or limit lines in a text file:
cut -c 80 PATH/FILE.txt # cut every line of FILE.txt to 80 characters (extra characters will be lost)
fold -w 80 PATH/FILE.txt # split every line of FILE.txt after 80 columns (extra characters will be wrapped to the next line)
fold -w 80 -s PATH/FILE.txt # split every line of FILE.txt after 80 columns, breaking at spaces (full words will be preserved)

Extrapolate strings of text from binary files:
strings -a -n 4 PATH/FILE # extrapolate strings of text from binary files (a string of text is considered at least 4 bytes long, you can change this value with the option -n) (get it with apt-get install binutils )
strings -a -t x PATH/FILE # extrapolate strings of text from binary files and return the offset of the string (-t x specifies the offset as hex, -t d decimal, -t o octal)

Search for data in a CSV (Comma Separated Values) / XSV (Character Separated Values) / SSV (Separator Separated Values) file:
awk -F',' '$1 == "TEXT"' /PATH/FILE.csv # If the first item (field) matches with the given TEXT in a comma (',') separated file, then return the whole line (row / record)
awk -F',' '{ if ($1 == "TEXT") { print $2 } }' /PATH/FILE.csv # If the first item (field) matches with the given TEXT in a comma (',') separated file, then return the second record in the line (row / record)

Count lines in text files:
wc PATH/FILE # line/word/byte count (handy options: --lines or -l print only the newline counts, --words print only the word counts, --bytes print only the byte counts, --chars print the character counts)
wc -L PATH/FILE # print the lenght of the longest line of FILE
grep "STRING" PATH/FILE|wc --lines # count how many lines in FILE contains STRING
wc -l PATH/*.txt # count the lines of all files with .txt extension in a directory
find PATH -name "*.txt" | xargs wc -l # count the lines of all files with .txt extension in a directory and its subdirectories

Count the occurrences of all the words in a text file:
tr -c a-zA-Z '\n' < PATH/FILE.txt | sed '/^$/d' | sort -n | uniq -ci | sort -nr # case insensitive
tr -c a-zA-Z '\n' < PATH/FILE.txt | sed '/^$/d' | sort -n | uniq -c | sort -nr # case sensitive

Sort a text file:
sort PATH/FILE.txt # sort FILE.txt alphabetically, case sensitive
sort -i PATH/FILE.txt # sort FILE.txt alphabetically, cosindering only printable characters
sort -d PATH/FILE.txt # sort FILE.txt alphabetically, cosindering only dictionary characters (alphanumeric characters and blanks)
sort -f PATH/FILE.txt # sort FILE.txt alphabetically, case insensitive
sort -fs PATH/FILE.txt # sort FILE.txt alphabetically, case insensitive and stable (sort according which case comes first for each letter)
sort -b PATH/FILE.txt # sort FILE.txt ignoring leading blank spaces
sort -r PATH/FILE.txt # sort FILE.txt in reverse alphabetical order
sort -n PATH/FILE.txt # sort numbers in FILE.txt
sort -h PATH/FILE.txt # sort human readable numbers (like 5K, 2G) in FILE.txt
sort -M PATH/FILE.txt # sort months (like Jan, Feb, Mar) in FILE.txt
sort -R PATH/FILE.txt # sort lines randomly in FILE.txt
sort -u PATH/FILE.txt # sort lines in FILE.txt and keep only unique lines
sort -k 2 PATH/FILE.txt # sort lines in FILE.txt using characters from column 2 (any blank character separates columns)
sort -t : -k 2 PATH/FILE.txt # sort lines in FILE.txt using characters from column 2 using ":" as a separator for columns
sort PATH/FILE.txt -o "PATH/DEST_FILE.txt" # sort FILE.txt and save the output in DEST_FILE.txt

Extract items from a delimited text file:
cut -d: -f1 PATH/SOURCE_FILE>PATH/DESTINATION_FILE  # choose the first item of a series delimited by ":"

Remove consecutive duplicate lines from a text file:
uniq PATH/FILE
uniq -u PATH/FILE # remove only successive duplicated lines (all occurrences)

Reverse lines of a file
rev PATH/FILE.txt > PATH/REVFILE.txt # Reverse the lines of a given file(s)

Delete lines from a text file:
sed -i '1d' PATH/TEXT_FILE # Remove the first line from a file
cp PATH/TEXT_FILE PATH/TEXT_FILE.tmp ; sed '1d' PATH/TEXT_FILE.tmp > PATH/TEXT_FILE ; rm -f PATH/TEXT_FILE.tmp # sed older than v3.95 doesn't have the option -i and requires a temporary file
sed -i '$ d' PATH/TEXT_FILE # Remove the last line from a file
cp PATH/TEXT_FILE PATH/TEXT_FILE.tmp ; sed '$ d' PATH/TEXT_FILE.tmp > PATH/TEXT_FILE ; rm -f PATH/TEXT_FILE.tmp # sed older than v3.95 doesn't have the option -i and requires a temporary file
sed -e '10d' PATH/FILE.txt > PATH/DESTFILE.txt # remove line 10
sed -e '5,15d' PATH/FILE.txt > PATH/DESTFILE.txt # remove lines 5 to 15

Replace multiple spaces with a single space in a text file:
sed "s/  */ /g" PATH/SOURCE_FILE>PATH/DESTINATION_FILE

Replace multiple spaces with a single space, and add a space before of every character in a text file:
sed "s/ */ /g" PATH/SOURCE_FILE>PATH/DESTINATION_FILE

Remove comments from a text file:
sed "/ *#/d;" PATH/SOURCE_FILE>PATH/DESTINATION_FILE

Remove blank lines from a text file:
sed "/^ *$/d" PATH/SOURCE_FILE>PATH/DESTINATION_FILE

Remove comments and blank lines from a text file:
sed "/ *#/d; /^ *$/d" PATH/SOURCE_FILE>PATH/DESTINATION_FILE

Add a text at the end of each line:
sed "s/$/\;/g" PATH/SOURCE_FILE>PATH/DESTINATION_FILE  # Append a semicolon (";") at the end of each line

Add a text at the beginning of each line:
sed "s/^/LINE : /g" PATH/SOURCE_FILE>PATH/DESTINATION_FILE  # Add "LINE : " at the beginning of each line

Remove blank lines (empty lines) in a text file:
sed '/^$/d' PATH/SOURCE_FILE>PATH/DESTINATION_FILE
sed '/^[[:space:]]*$/d' PATH/SOURCE_FILE>PATH/DESTINATION_FILE # Remove all blank lines, including the ones that are not actually empty but contains spaces and/or tabs

Add blank lines after every line in a text file:
sed "G" PATH/SOURCE_FILE>PATH/DESTINATION_FILE # Add 1 blank line after every line in a text file
sed "G;G" PATH/SOURCE_FILE>PATH/DESTINATION_FILE # Add 2 blank lines after every line in a text file

Append a line of text to a file:
echo "line 1" >> PATH/FILE.txt

Append multiple lines of text to a file (in scripts):
echo "line 1
line 2" >> PATH/FILE.txt

Append multiple lines of text to a file (in scripts):
cat <> PATH/FILE.txt
line 1
line 2
EOT

Compare and show differences between two binary files:
cmp -b -l PATH/FILE1 PATH/FILE2 # compare two files and returns the position of the different bytes followed by the value and the corresponding ASCII character for each file
cmp -b -l --bytes=10 PATH/FILE1 PATH/FILE2 # limit comparison to the first 10 bytes

Compare and show differences between two text files:
diff -abB PATH/FILE1 PATH/FILE2 # Report differences between two text files
diff -abBi PATH/FILE1 PATH/FILE2 # Report differences between two text files ignoring case differences in their content

Compare and show differences between three text files:
diff3 -e FILE1 FILE2 FILE3 # output unmerged changes from FILE2 to FILE3 into FILE1

Return the average lenght in lines for all text files in the tree:
find PATH -name "*.txt" -exec wc -l {} \; | awk 'BEGIN {x=0;y=0} {x+=1; y+=$1} END {print y/x}'

More file operations:
# All these commands should work also with "gawk" (GNU awk) and "nawk" (new awk by AT&T)
awk '1; { print "" }' SOURCE>DESTINATION # Add an empty line after each line
awk 'NF { print $0 "\n" }' SOURCE>DESTINATION # Add an empty line after each line, except when an empty line is already present
awk '{ print FNR ".\t" $0 }' SOURCE>DESTINATION # Add line numbers followed by dot and tab before than each line
awk '{sub(/^[ \t]+/, "")};1' SOURCE>DESTINATION # Remove leading spaces from each line
awk '{sub(/[ \t]+$/, "")};1' SOURCE>DESTINATION # Remove trailing spaces from each line
sed "s/[ \t]*$//" PATH/SOURCE_FILE>PATH/DESTINATION_FILE # Remove trailing spaces and tabs from a text file
awk '{gsub(/^[ \t]+|[ \t]+$/,"")};1' SOURCE>DESTINATION # Remove leading and trailing spaces from each line
awk '{sub(/A/,"B")}; 1' SOURCE>DESTINATION # Replace the first occurrence of A with B in each line
awk '{gsub(/A/,"B")}; 1' SOURCE>DESTINATION # Replace every occurrence of A with B in each line
awk '/X/{gsub(/A/, "B")}; 1' SOURCE>DESTINATION # Replace every occurrence of A with B in each line that contains X
awk '!/X/{gsub(/A/, "B")}; 1' SOURCE>DESTINATION # Replace every occurrence of A with B in each line that does not contain X
awk 'BEGIN{while (a++<100) s=s "."; print s}' >DESTINATION # Create a file containing 100 dots (.)
awk 'END{print NR}' SOURCE # count lines (emulates "wc -l")

Indentation

Indent C code (or similarly formatted code): apt-get install indent indent PATH/SOURCE -o PATH/DEST indent -nprs PATH/SOURCE -o PATH/DEST # Doesn't add a space after an open bracket and before a closed one Indent HTML code: apt-get install hindent hindent PATH/SOURCE.HTML>PATH/DEST.HTML hindent -i 0 PATH/SOURCE.HTML>PATH/DEST.HTML # Remove indentation hindent -c PATH/SOURCE.HTML>PATH/DEST.HTML # Force all tags to lowercase (otherwise all tags are forced to UPPERCASE by default) hindent -c -i 2 PATH/SOURCE.HTML>PATH/DEST.HTML # Set indentation level to 2 and force all tags to lowercase hindent -c PATH/SOURCE.HTML>PATH/DEST.HTML # Force all tags to lowercase hindent -f PATH/SOURCE.HTML>PATH/DEST.HTML # Flow: dump only HTML tags discarding any data between hindent -l PATH/SOURCE.HTML>PATH/DEST.HTML # List all HTML tags contained in the source document Remove Indentation: sed -i 's/^[[:space:]]*//g' PATH/FILE # Remove indentation from any text file




Please DONATE to support the development of Free and Open Source Software (PayPal, Credit Card, Bitcoin, Ether)

Page issued on 25-Sep-2022 04:31 GMT
Copyright (c) 2022 Geody - Legal notices: copyright, privacy policy, disclaimer