Geody Labs

# The Debian Linux Magic Spells File Cheat Sheet
# by Elf Qrin - http://labs.geody.com/
v1.8 r2018-03-02 fr2005-09-xx


# "One man cast a lingering spell of awe and wonder, of magical innocence overcoming evil, of simple courage conquering fear" -- J.R.R. Tolkien's The Lord of the Rings (1978 Ralph Bakshi animated film)
# 'someone who thinks of code as magical incantations and asks only "what do I need to type to make this happen?"' -- from the definition of Script Kiddies in the Jargon File


-----

Notes:

- In this file, jed is used as the default editor for it's more intuitive than others. You'll have to install it ( apt-get install jed ). Alternatively, you might want to use vi or vim instead.
- Most commands work also on Ubuntu / Kubuntu / Xubuntu / Edubuntu, Mint and Xandros. Some commands work on all Linux distributions.
- You may send contributions and suggestions to (see: http://www.geody.com/contacts.php )

- You can access the terminal in most Debian based distributions (including Debian itself and Ubuntu) by pressing CTRL+ALT+F1 (actually you can switch between terminals with CTRL+ALT+function keys. With CTRL+ALT+F7 you normally return to the graphic interface in Ubuntu). In Xandros, you have to press CTRL+ALT+T.
- To access a remote terminal via SSH from Windows use the free and open source software PuTTY.
- To access a remote terminal via SSH from Android use the free and open source software ConnectBot.

-----

Install a software package using APT (Advanced Package Tool):

apt-get install base-config
apt-setup

Set APT sources (repositories) manually:
jed /etc/apt/sources.list
The normal way of accessing the mirror is by putting these lines in /etc/apt/sources.list :

deb http://http.us.debian.org/debian stable main contrib non-free
deb-src http://http.us.debian.org/debian stable main contrib non-free

deb http://security.debian.org stable/updates main contrib non-free
deb-src http://security.debian.org stable/updates main contrib non-free

You can use a local repository instead of the US Debian one, for example debian.fastweb.it instead of http.us.debian.org
Use unstable instead of stable if you want bleeding edge technology. Not very recommended in production environments.

To keep an old version up to date:
deb http://archive.debian.org/debian/ DEBIAN_VERSION main non-free contrib
deb-src http://archive.debian.org/debian/ DEBIAN_VERSION main non-free contrib

deb http://archive.debian.org/debian-security/ DEBIAN_VERSION/updates main non-free contrib
deb-src http://archive.debian.org/debian-security/ DEBIAN_VERSION/updates main non-free contrib

Update the list of packages:
# After adding a new apt-get source, or before installing new packages, enter this command
apt-get update
apt update

Should a "Segmentation fault" error occur when updating the list of packages, you can fix it with these commands:
rm -r /var/lib/apt/lists/*
rm /var/cache/apt/*.bin # will delete pkgcache.bin and srcpkgcache.bin

Show upgradable packages without upgrading them (after an update)
apt-get -u upgrade --assume-no
apt list --upgradable

Upgrade installed packages:
apt-get upgrade # upgrade already installed packages but don't remove anything or install anything new
apt upgrade

Upgrade a distribution:
# This is just a basic procedure to upgrade a Debian release. Check https://www.debian.org/releases/stable/amd64/release-notes/ch-upgrading.html for mode detailed information.
apt-get update
apt-get upgrade
aptitude search '~i(!~ODebian)' # List third-party packages that may be lost after a dist-upgrade
dpkg --audit # List Half-Installed or Failed-Config packages
apt-get autoremove # remove packages installed as dependency of already uninstalled packages
jed /etc/apt/sources.list # Set APT sources (repositories) for the new Debian release
apt-get update
apt-get upgrade
apt-get dist-upgrade # upgrade everything removing/adding packages as required to resolve dependencies
apt-get autoremove
reboot

Make sure the system is up to date after a fresh installation:
dpkg --audit
apt-get update
apt-get upgrade
apt-get dist-upgrade
dpkg --audit

Prevent a package to be upgraded:
apt-get install wajig
wajig hold PACKAGE # the specified package will no longer be upgraded (for example with an apt-get upgrade)
wajig unhold PACKAGE # the package will be upgraded again
wajig hold mysql-server-5.0 # a common problem with Kernel 2.4 is that latest releases of MySQL server 5 are not compatible with such kernel version, interfering with the upgrade process

Search for a package:
apt-cache search TEXT # search for a package containing the specified text or keyword in its name or description

List installed packages:
dpkg-query -l # List all installed packages
dpkg --list # List all installed packages
dpkg --get-selections # List all installed packages
dpkg -l # List all installed packages with version and description
dpkg --get-selections | wc --lines # Count installed packages
grep install /var/log/dpkg.log | tail -10 # Last 10 installed packages
dpkg --listfiles PACKAGE # List all files "owned" by PACKAGE
cat /var/log/dpkg.log | grep "status installed " # Show last installed packages
cat /var/log/apt/history.log # Show last apt commands (installed and upgraded packages)
aptitude search '~i(!~ODebian)' # List third-party packages only

Check if a package is installed:
dpkg -s PACKAGE
dpkg-query -l PACKAGE
dpkg-query -W -f='${Status} ${Version}\n' PACKAGE # If the package is installed, show status and version 
dpkg-query -l "TEXT*" # Show all installed packages starting with TEXT
dpkg-query -l "*TEXT" # Show all installed packages ending with TEXT
dpkg-query -l "*TEXT*" # Show all installed packages containing TEXT
cat /var/log/dpkg.log | grep "status installed PACKAGE" # Search for a specific package among the last installed ones

Install a package:
apt-get install PACKAGE # get and install a PACKAGE from a repository specified in /etc/apt/sources.list
dpkg -i PACKAGE.deb # manually install a package available locally (already downloaded)
apt-get install --reinstall PACKAGE # reinstall a PACKAGE

Show information about a package and all its dependencies
apt-cache show PACKAGE
aptitude show PACKAGE # similar to apt-cache show PACKAGE but shows extra information
apt-cache showpkg PACKAGE # show dependencies and reverse dependencies

Verify if all packages are correctly installed and configured:
dpkg --audit # List Half-Installed or Failed-Config packages. If everything is fine, it doesn't return anything.

Check for broken dependencies:
apt-get check

Uninstall a package:
apt-get remove PACKAGE
aptitude purge PACKAGE # remove PACKAGE and its configuration files

Show a random non installed package:
aptitude search ~g | while read ; do echo "$RANDOM $REPLY" ; done | sort -n | head -1 | cut -d' ' -f2-

Store/Restore Packages:
dpkg --get-selections > installedpackages.txt # Store the list of installed packages into the file installedpackages.txt
dpkg --set-selections < installedpackages.txt # Restore the list of installed packages from the file installedpackages.txt
apt-get install `cat installedpackages.txt` # Restore the list of installed packages from the file installedpackages.txt

Clean up packages:
apt-get autoremove # remove packages installed as dependency of already uninstalled packages
apt-get clean # delete all downloaded packages (that is, all files stored in /var/cache/apt/archives/ and /var/cache/apt/archives/partial/ , except for lock files). If you want to reinstall one of such packages, apt-get will have to download them again.
apt-get autoclean # only delete packages which is no longer possible to download (obsolete).

If you get one of the following error messages:

E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)
E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?

dpkg: status database area is locked by another process

or anyway if something goes wrong when upgrading packages, you may fix it this way:

rm /var/lib/apt/lists/lock
rm /var/lib/dpkg/lock
rm /var/cache/apt/archives/lock
dpkg --configure -a # to fix interrupted configure (may take a while)
apt-get -f install # to fix broken dependencies

If you still get error messages there are probably running processes working with installation files, you either have to find and kill them all or reboot before than attempting to recover the upgrade process again.

After upgrading the Linux kernel, it's better to perform a update-grub and then reboot the system.

Debian Linux (Sarge, Kernel 2.6): workaround for
E: This installation run will require temporarily removing the essential package e2fsprogs due to a Conflicts/Pre-Depends loop. This is often bad, but if you really want to do it, activate the APT::Force-LoopBreak option.
apt-get remove e2fsprogs
apt-get install e2fsprogs sysvinit initscripts
upgrade glibc when requested.
# note that this is a workaround, it should be fixed with apt-get dist-upgrade

Note: when the Debian stable changes, if you don't upgrade your system accordingly, you'll have to modify the /etc/apt/sources.list file to specify you are using an old version of Debian (that is, you are no longer using the "current stable" version of Debian). For example, if you are using Sarge, and Debian Etch is released as stable, and you are not upgrading to Etch, you have to change all references to stable as sarge in your source.list file.
Which means, you have to change "deb http://debian.example.com/debian stable main contrib non-free" as "deb http://debian.example.com/debian sarge main contrib non-free".
If you don't make this change, when you'll try to install new packages, you'll receive the following message:
The following packages will be REMOVED:
  base-config initrd-tools kernel-image-2.4.27-2-386
This is an example, the kernel image can be different in your case, however you'll not be able to install the package and it may damage your system. In fact, if you procede, you will be warned by this message:
You are running a kernel (version 2.4.27-2-386) and attempting to remove the same version.
This is a potentially disastrous action. Not only will /boot/vmlinuz-2.4.27-2-386 be removed, making it impossible to boot it, (you will have to take action to change your boot loader to boot a new kernel), it will also remove all modules under the directory /lib/modules/2.4.27-2-386.
Just having a copy of the kernel image is not enough, you will have to replace the modules too.
I repeat, this is very dangerous. If at all in doubt, answer no. If you know exactly what you are doing, and are prepared to hose your system, then answer Yes.

If you get an error like
'Template parse error near `Description-sr@latin.UTF-8', in stanza #X of /PATH/FILE.templates
then edit the specified file removing the lines containing the given text (and, if present, the following line with the description) from /PATH/FILE.templates

If you get an error with locales, like
locale: Cannot set LC_CTYPE to default locale
you might be able to fix it with
apt-get install locales

If you get a message ending with the following request after a power outage:

Give root password for maintenance
(or type Control-D to continue):

enter the root password
then check the file system of the faulty device (read the whole message to identify the faulty driver, generally is /dev/hda0 on a single hard drive system or /dev/md0 on a RAID system):
fsck -f FAULTY_DEVICE
reply yes (y) to all questions. Note that if it asks to connect to lost+found more than once some of your files might be seriously damaged. However you can also use fsck -f -y FAULTY_DEVICE to reply automatically "yes" to all questions.
then
reboot

If you can't perform a simple operation that always used to work (for example, copying a file) make sure the disk is not full (check with df).

Set keyboard layout
# Enter this command to localize the keyboard for your country
apt-get install console-common

You can also use the apt command instead of apt-get for most common tasks:
apt install : apt-get install
apt remove : apt-get remove
apt purge : apt-get purge
apt update : apt-get update
apt upgrade : apt-get upgrade
apt autoremove : apt-get autoremove
apt full-upgrade : apt-get dist-upgrade
apt search : apt-cache search
apt show : apt-cache show

-----

Install a software package from the source (compile a package source):

Packages in form of source code are generally stored into .tar.gz (.tgz) or .tar.bz2 (.tbz) files.

You may need to download them from the Internet:
wget http://www.example.com/source/package.tgz

Once you have the package (from the Internet, a CD, or otherwise) in your work directory, you'll have to decompress it:

tar -xjvf package.tbz # extract all files from a TAR+BZIP2 compressed archive
or
tar -xzvf package.tgz # extract all files from a TAR+GZIP compressed archive

it should decompress inside a new directory. Move inside this new directory:
cd PACKAGE_SOURCE_DIRECTORY_NAME

and execute these commands:
./configure
make
su ROOT_PASSWORD # (if you're not logged as root)
make install

The executable file should be created inside /usr/local/bin ,  and you should be able to invoke it typing its name in the console and then hitting ENTER.

To uninstall a package, you have to move inside the directory containing the source (if you kept it):
cd PACKAGE_SOURCE_DIRECTORY_NAME
and uninstall it:
make uninstall
If you haven't kept the source, you'll have to locate all files (try locate PACKAGE_NAME) and delete them manually.

-----

Change root password without knowing the existing one
You need physical access to the system.

Add init=/bin/bash to boot parameters
mount -o remount,rw # mount file system
passwd # set new password. WARNING: sudo passwd will change root password, regardless of the current account.

Recover a deleted password file:
# Debian makes regular backups of the password file in /var/backups/
cp /var/backups/passwd.bak /etc/passwd
chmod 644 /etc/passwd

-----

Writing text files for Linux using an editor running on another Operating System:
When you edit a system file, a shell script, a fortune cookie text file, or another file meant to be processed by a Linux shell command or application, make sure that the lines are divided UNIX style (LF only, x0A; not CR/LF, x0D0A as in Windows), otherwise the line breaks will not be recognized as such and the whole file will be treated as a single line.

-----

Install SSH:
apt-get install ssh

Get SSH version:
ssh -V

Start a SSH connection (port 22):
ssh HOST

Configure SSH:
jed /etc/ssh/sshd_config
# If your client can't authenticate to your server, it might be sending the password as a cleartext and your server is not accepting it. In this case you have to enable tunnelled clear text passwords setting PasswordAuthentication yes

# Make sure SSH server is not using the outdated Protocol 1. This line should be present in the configuration file instead:
Protocol 2

# Set SSH timeout (in seconds). A logged user will be logged out after idling past specified time.
ClientAliveInterval 600
ClientAliveCountMax 0

/etc/init.d/ssh restart # Restart SSH to make changes take effect. However note that they will apply to next SSH session.


Start a Telnet (unencrypted) connection:
telnet HOST # start a telnet connection on HOST at default port 23
telnet HOST:PORT # start a telnet connection on HOST at the specified PORT

-----

Bash shells:

Show BASH version:
echo $BASH_VERSION

Command Line Navigation:
CURSOR LEFT / CURSOR RIGHT # move cursor one character left or right on command line
LEFT ALT+[B/F] # move cursor to previous / next word on command line
CURSOR UP / CURSOR DOWN # move to previous / next command in history
CTRL+R # search for a command in History: hit CTRL+R, type the string you want to search, hit CTRL+R again to find further matches

Switch bash shells:
LEFT ALT+[F1-F6]

Cycle between bash shells:
LEFT ALT+[CURSOR LEFT / CURSOR RIGHT]

Show current terminal:
tty

Reset a "corrupted" shell:
reset

Clear shell terminal window:
clear

Set Terminal Window Size:
echo -e "\e[8;LINES;COLUMNSt" # resize the terminal window to LINES x COLUMNS . For example, tlins=40; tcols=160; echo -e "\e[8;$tlins;${tcols}t"; (or echo -e "\e[8;40;160t"; ) resizes the window to 40 x 160 characters.

Get Terminal Window Size:
tput lines # return the height in characters (number of lines) of the terminal window
tput cols # return the width in characters (number of columns) of the terminal window
echo "$(tput cols) x $(tput lines)"; # show terminal window size
# The environmental variables $LINES and $COLUMNS contain the values of lines and colums of the terminal window, but their value is not preserved within scripts, so that you'll have to add this line to your script assign them the expected values: COLUMNS="$(tput cols)"; LINES="$(tput lines)";

Open a new Bash shell:
bash

Exit from a non login shell:
exit

Duplicate a terminal:
apt-get install conspy
conspy N # N can range from 1 to 6, then execute another conspy N with the same number on another bash shell

Show current command line:
cat /proc/self/cmdline # typical output: cat/proc/self/cmdline

Change prompt:
export PS1="[\t] \u@\h:\w\\\$ " # show time in 24 hours format between square brackets, the logged user name, the host name, the current path, and the root status ("#" if root, "$" otherwise)
export PS1="\[\e[1;32m\][\t] \u@\h:\w\\\$ \[\e[0m\] " # same prompt, but colored in light green
export PS1="\[\e[1;32m\][\$(date +%a\ %d%b%Y\ %H:%M:%S)] \u@\h:\w\\\$\[\e[0m\] " # full date and time between square brackets, the logged user name, the host name, the current path, and the root status ("#" if root, "$" otherwise), colored in light green. Note that the shell command date is used, invoked with \$() .  If $() (without the leading backslash) were used, the date would be generated only the first time and then stored into the variable and wouldn't be updated every time the prompt is shown. Note: if you want to set Windows Command Prompt to match your Linux Shell Prompt you have to set the %prompt% environment variable. For example: set prompt=[$d$s$t]$s%username%@%computername%:$p$+$g$s
export PS1="\[\e[1;32m\][\$(date +%Y-%m-%d\ %H:%M:%S\ %a)] \u@\h:\w\\\$\[\e[0m\] " # full date and time (in a sortable format) between square brackets, the logged user name, the host name, the current path, and the root status ("#" if root, "$" otherwise), colored in light green.
echo $PS1 # See current prompt

Edit the script executed at the shell start-up (user login shell):
# Custom Prompt and Aliases can be set here
jed ~/.bash_profile

Sample bash profile script:
(see http://labs.geody.com/systatus/ )
---
# ~/.bash_profile

# BASH Shell Start-up
# r2018-01-27 fr2016-10-18
# by Valerio Capello - http://labs.geody.com/ - License: GPL v3.0

# Config
tshilon="\e[0;33m"; tshilof="\e[0m";
tsalerton="\e[0;31m"; tsalertof="\e[0m";

# Set Terminal Window Size
# tlins=40; tcols=160; echo -e "\e[8;$tlins;${tcols}t";

# Get Terminal Window Size
COLUMNS="$(tput cols)"; LINES="$(tput lines)";

# Prompt
export PS1="\[\e[1;32m\][\$(date +%Y-%m-%d\ %H:%M:%S\ %a)] \u@\h:\w\\\$\[\e[0m\] "

# History Date Format
export HISTTIMEFORMAT="%F %T  "

# Aliases
alias l="ls -laF --group-directories-first --color=auto"
alias d="ls -aF --group-directories-first --color=auto"

# Message
echo; clear
date "+%a %d %b %Y %H:%M:%S %Z (UTC%:z)"
echo -n "Hello "; echo -ne "$tshilon"; echo -n "$(whoami)"; echo -ne "$tshilof";
if [ "$SSH_CONNECTION" ]; then
echo -n " ("; echo -ne "$tshilon"; echo -n "`echo $SSH_CLIENT | awk '{print $1}'`"; echo -ne "$tshilof)";
fi
echo -n ", ";
echo -n "welcome to "; echo -ne "$tshilon"; echo -n "$(hostname)"; echo -ne "$tshilof";
echo -n " ("; echo -ne "$tshilon"; echo -n "$(hostname -i)"; echo -ne "$tshilof)";
echo ".";
echo -n "Machine ID: "; echo -n "$(cat /etc/machine-id) ";
echo -n "Boot ID: "; echo -n "$(cat /proc/sys/kernel/random/boot_id) ";
echo -n "Session ID: "; echo "$(cat /proc/self/sessionid)";
echo -n "You are ";
if [ -n "$SSH_CONNECTION" ]; then
echo -n "connected remotely via SSH";
elif [[ "${DISPLAY%%:0*}" != "" ]]; then
echo -n "connected remotely "; echo -ne "$tsalerton"; echo -n "NOT"; echo -ne "$tsalertof"; echo " via SSH (which is Bad)";
else
echo -n "connected locally";
fi
echo ". Your Terminal Window Size is $COLUMNS x $LINES"
if [ $EUID -eq 0 ]; then
echo -ne "$tsalerton"; echo -n "You have ROOT superpowers!"; echo -e "$tsalertof";
fi
echo

# Software version
uname -a
echo "Bash version: $BASH_VERSION"
# Webserver version
echo -n "$(/usr/sbin/apache2 -v|head --lines=1) "; echo "$(/usr/sbin/apache2 -v|tail --lines=1)";
openssl version -v
php -v|head --lines=1
mysql -V
# echo
# echo -n "Installed Packages: "; dpkg --get-selections | wc -l;
# echo "Last installed packages:"; grep install /var/log/dpkg.log | tail --lines=5;
echo

# System status
# echo -n "Vendor: "; echo "$(cat /sys/class/dmi/id/sys_vendor)";
echo -n "Machine: "; echo "$(cat /sys/class/dmi/id/product_name)";
# echo -n "Machine Type: "; echo "$MACHTYPE";
# echo -n "Board: "; echo "$(cat /sys/class/dmi/id/board_vendor) $(cat /sys/class/dmi/id/board_name)";
# echo -n "BIOS: "; echo "$(cat /sys/class/dmi/id/bios_vendor) $(cat /sys/class/dmi/id/bios_vendor) $(cat /sys/class/dmi/id/bios_version) $(cat /sys/class/dmi/id/bios_date)";
echo -n "CPU: "; echo -n "$(grep 'model name' /proc/cpuinfo | head -1). ";
echo -n "Cores: "; grep -c 'processor' /proc/cpuinfo
free -mh | xargs | awk '{print "Memory: Size: " $8 " Used: " $9 " Free: " $10}';
df -h | xargs | awk '{print "Disk ("$8"): Size: " $9 " Used: " $11 " Free: " $10}';
# df -P -h | awk '(0+$5 < 90 && 0+$5 > 0) {print "FS: "$1" ("$6") Size: "$2" Used: "$3" ("$5") Free: "$4" ("(100-$5)"%)";}';
echo -ne "$tsalerton"; df -P -h | awk '0+$5 >= 90 {print "FS: "$1" ("$6") Size: "$2" Used: "$3" ("$5") Free: "$4" ("(100-$5)"%)";}'; echo -ne "$tsalertof";
echo; echo -n "Uptime: "; uptime
echo

# Users
echo "Last logged users:"; last -n 5
echo; echo "Currently logged users:"; who
echo; echo -n "Current user: "; id
echo

# Security
# Shellshock vulnerability check (reports to root only)
if [[ $EUID -eq 0 ]]; then
env x='() { :;}; echo Bash vulnerable to Shellshock' bash -c 'echo -n'
fi
---

You may want to replicate your bash profile script as an executable script (called for example status.sh ) containing information provided in the sections starting from Message and below (you'd better change echo -n "welcome to "; to echo -n "this is ";).

Customize the MOTD (Message Of The Day), message shown after logging in, before than starting the shell:
jed /etc/motd # edit the MOTD

Customize the message shown before than the log in request:
jed /etc/issue # edit the welcome message

Description of a command:
whatis COMMAND

Manual of a command:
man COMMAND
apropos KEYWORD # list all manual pages related to the KEYWORD

Return the path of a command:
which COMMAND
type COMMAND # specifies if COMMAND is a shell builtin (internal)

List internal commands:
help
help COMMAND # show information about an internal command

Identify the type of a file:
file PATH/FILE
file -z PATH/FILE.zip # Attempts to check the file type of files contained inside a compressed file

Create an alias:
alias l="ls -laF --color=auto" # makes "l" an alias for "ls -laF --color=auto"

List all existing aliases:
alias

Remove an alias:
unalias ALIAS

Repeat last command:
!!

Start a comment (ignore following text):
#

Do nothing, successfully:
true

Do nothing, unsuccessfully:
false

Conditional execution:
if [ 1 == 1 ]; then echo "true"; fi
if [ 1 == 1 ]; then echo "true"; else echo "false"; fi

Execute a random command between two:
((RANDOM%2 == 0)) && echo "Yes" || echo "No"
if [ $((RANDOM % 2)) == 0 ]; then echo "Yes"; else echo "No"; fi

Find the time required to perform a command line:
# It may useful to compare commands, or parameters of the same command, and see which one is faster.
# The "real" value is the actual elapsed time which is the data you most likely need, while The "user" value shows the time spent by the code in user mode, and the "sys" value shows the time spent by the code in kernel mode.
time (ls) # Shows the time needed to list the current directory

Process priority:
nice -n0 COMMAND # execute a command with default priority (0)
nice -n20 COMMAND # execute a command with the lowest priority (20)
nice -n-20 COMMAND # execute a command with the highest priority (-20). Only root can assign negative (high) priority
renice 10 PID # reassign priority to an existing process

List running processes and their PIDs:
echo $$ # show the PID of the current shell (or script, if invoked from a script)
ps # show all processes with a tty in current shell
ps -a # show all processes with a tty in all shells
ps -A # show all processes
ps -A|grep "STRING" # show all processes that matches STRING
ps auxf | sort -nr -k 4 # show processes sorted by memory usage
ps auxf | sort -nr -k 4 | head --lines=10 # show processes sorted by memory usage (first 10 only)
ps auxf | sort -nr -k 3 # show processes sorted by CPU usage
ps auxf | sort -nr -k 3 | head --lines=10 # show processes sorted by CPU usage (first 10 only)

Show running processes and CPU load:
top # press "q" to quit

Show running processes accessing the disk or doing other Input/Output activity:
apt-get install iotop
iotop --only # press "q" to quit

Show jobs running in the background:
jobs

Show which user launched a process:
fuser PID

Terminate a process:
kill -3 PID  # quit a process
kill -15 PID  # term a process
kill -9 PID  # kill a process (most effective to quit an unresponsive process)

Terminate all processes with a matching name:
killall NAME
killall -I NAME  # ignore cases
killall -i NAME  # ask for confirmation before to kill

/etc/init.d/COMMAND start  # start a service, system daemon
/etc/init.d/COMMAND stop  # stop a service, system daemon
/etc/init.d/COMMAND restart  # restart a service, system daemon

Show information about all loaded module:
lsmod

Show open files:
apt-get install lsof
lsof # Show all open files
lsof PATH # Show all open files within PATH
lsof / | awk '{ if($7 > 1048576) print $7/1048576 "MB" " " $9 " " $1 }' | sort -n -u | tail --lines=10 # show the 10 largest open files with their size in MB and the processes that keep them open

Test to check if your Bash shell is vulnerable to Shellshock (you should only see a dot if your system is not vulnerable)
env x='() { :;}; echo Bash vulnerable to Shellshock' bash -c 'echo .'

-----

Information about the vendor:
cat /sys/class/dmi/id/sys_vendor # System Vendor

Information about the machine (virtual or physical):
cat /sys/class/dmi/id/product_name # Product Name
cat /sys/class/dmi/id/product_uuid # Product UUID
cat /sys/class/dmi/id/product_serial # Product Serial
cat /sys/class/dmi/id/product_version # Product Version (if any)

cat /sys/class/dmi/id/chassis_vendor # Chassis Vendor (if there is an enclosure)
cat /sys/class/dmi/id/chassis_type # Chassis Type
cat /sys/class/dmi/id/chassis_version # Chassis Version (if any)
cat /sys/class/dmi/id/chassis_serial # Chassis Serial (if any)
cat /sys/class/dmi/id/chassis_asset_tag # Chassis Asset Tag (if any)

cat /sys/class/dmi/id/board_vendor # Board Vendor
cat /sys/class/dmi/id/board_name # Board Name
cat /sys/class/dmi/id/board_version # Board Version (if any)
cat /sys/class/dmi/id/board_serial # Board Serial (if any)
cat /sys/class/dmi/id/board_asset_tag # Board Asset Tag (if any)

cat /sys/class/dmi/id/bios_vendor # BIOS Vendor
cat /sys/class/dmi/id/bios_version # BIOS Version
cat /sys/class/dmi/id/bios_date # BIOS Date

Show Machine's Unique ID:
cat /etc/machine-id # show the unique machine ID of the local system that is set during installation. The machine ID is a single newline-terminated, hexadecimal, 32-character, lowercase machine ID string. When decoded from hexadecimal, this corresponds with a 16-byte/128-bit string (from the manual). Also in /var/lib/dbus/machine-id

Boot ID:
cat /proc/sys/kernel/random/boot_id # show the random ID regenerated at each boot

Session ID
cat /proc/self/sessionid # show the session ID, which is unique for each login

Random UUID:
cat /proc/sys/kernel/random/uuid # generate a random UUID, different every time is invoked

Show system name:
hostname
cat /proc/sys/kernel/hostname

Show system's IP address:
hostname -i

Show client's IP address in a SSH connection:
echo $SSH_CLIENT | awk '{print $1}'
echo $SSH_CONNECTION | awk '{print $1}' # Note that $SSH_CONNECTION (not processed with awk) also contains the server's IP address
echo "$(whoami) (`echo $SSH_CLIENT | awk '{print $1}'`) @ $(hostname) ($(hostname -i))"; # client (client IP) @ host (host IP)

Change system name as NEW_NAME:
hostname NEW_NAME
jed /etc/hostname # change it as NEW_NAME
jed /etc/hosts # you'd better also add NEW_NAME as an alias hostname for 127.0.0.1 (localhost)

Show system information:
uname -a
uname -r # show only kernel version
cat /proc/sys/kernel/osrelease # kernel version
cat /etc/issue # Welcome message: contains Linux version (unless otherwise edited by the user)
cat /etc/debian_version # Debian version

Show the kernel ring buffer (to print out the bootup messages):
dmesg

Show last 20 lines of the system messages file:
tail --lines=20 /var/log/messages

By default, on most Debian based distributions, every 20 minutes syslog marks the message file with "localhost -- MARK --" to log that the system is up and running.
You can modify this behavior by editing the syslog startup file:
jed /etc/init.d/sysklogd
Locate the line that defines the variable SYSLOGD (generally, SYSLOGD="" ). If it's missing you can add it at the beginning of the configuration file, where other variables are defined.
and use the switch -m to set the delay between marks in minutes, or set it to 0 to disable the notification.
Example: SYSLOGD="-m 60" marks the message file every hour, SYSLOGD="-m 0" disable the notification.
Restart syslog to make the change effective:
/etc/init.d/sysklogd restart

Start Debian base system configuration tool:
base-config

Show Hardware info:
dmidecode -t TYPE_OR_SUPERTYPE # Valid types are: 0: BIOS, 1: System, 2: Base Board, 3: Chassis, 4: Processor, 5: Memory Controller, 6: Memory Module, 7: Cache, 8: Port Connector, 9: System Slots, 10: On Board Devices, 11: OEM Strings, 12: System Configuration Options, 13: BIOS Language, 14: Group Associations, 15: System Event Log, 16: Physical Memory Array, 17: Memory Device, 18: 32-bit Memory Error, 19: Memory Array Mapped Address, 20: Memory Device Mapped Address, 21: Built-in Pointing Device, 22: Portable Battery, 23: System Reset, 24: Hardware Security, 25: System Power Controls, 26: Voltage Probe, 27: Cooling Device, 28: Temperature Probe, 29: Electrical Current Probe, 30: Out-of-band Remote Access, 31: Boot Integrity Services, 32: System Boot, 33: 64-bit Memory Error, 34: Management Device, 35: Management Device Component, 36: Management Device Threshold Data, 37: Memory Channel, 38: IPMI Device, 39: Power Supply; Valid supertypes are: bios (0, 13), system (1, 12, 15, 23, 32), baseboard (2, 10), chassis (3), processor (4), memory (5, 6, 16, 17), cache (7), connector (8), slot (9).
dmidecode -t 6|grep -i "Size" # Return installed memory banks
dmidecode -t 1|grep -i "Manufacturer\|Product\|Serial" # Return system's Manufacturer, Product Name, and Serial Number

Show CPU info:
lscpu
cat /proc/cpuinfo
grep 'model name' /proc/cpuinfo # return CPU model
grep 'cpu cores' /proc/cpuinfo # return number of CPU cores
grep -c 'processor' /proc/cpuinfo # count CPU cores
grep 'flags' /proc/cpuinfo # return available features of the CPU
cat /proc/loadavg # show load average: the first three fields in this file are load average figures giving the number of jobs in the run queue (state R) or waiting for disk I/O (state D) averaged over 1, 5, and 15 minutes. They are the same as the load average numbers given by uptime(1) and other programs. The fourth field consists of two numbers separated by a slash (/). The first of these is the number of currently executing kernel scheduling entities (processes, threads); this will be less than or equal to the number of CPUs. The value after the slash is the number of kernel scheduling entities that currently exist on the system. The fifth field is the PID of the process that was most recently created on the system.

Show RAM info:
cat /proc/meminfo
grep MemTotal /proc/meminfo  # return total RAM seen by the system
egrep 'Mem|Cache|Swap' /proc/meminfo # return information about RAM, Cache, Swap
free -mh | xargs | awk '{print "Used/Free/Total Memory: " $9 " / " $10 " / " $8}' # Return Used, Free, Total Memory
free -k # show total, used, free memory in KB
free -h # show total, used, free memory in human readable format
free -m | xargs | awk '{print "Free/Total Memory: " $17 " / " $8 " MB"}'
vmstat -s # show information about memory activity

List all PCI buses and devices:
lspci

Show system temperature (if system supports ACPI):
apt-get install acpi
acpi -t

Localization:
dpkg-reconfigure locales # edit locale / add new locales
locale  # show locale settings
locale -a  # show available locales
locale|cut -d= -f1|xargs locale -kc|less  # list fields in locale database
locale territory  # show the locale territory # echo "This server is configured for `locale territory`"
printf "%'d\n" 1234567  # print an integer number grouping thousands with the locale configuration

List of all commands entered in the shell (history):
jed ~/.bash_history # edit history file manually
cat ~/.bash_history # show all commands in history
less ~/.bash_history # navigate in history
cat ~/.bash_history | tr "\|\;" "\n" | sed -e "s/^ //g" | cut -f 1 -d " " | sort -n | uniq -c | sort -n | tail -n 15 | sort -nr # show the 15 most used commands in history
history # show all commands in history together with their offset
history -d OFFSET # delete the specified history entry
history -c # clear the whole history file
!!:p # show last executed command
!-1:p # show last executed command
!-2:p # show second last executed command
!! # repeat (execute again) last executed command
!-1 # repeat (execute again) last executed command
!-2 # repeat (execute again) second last executed command
export HISTTIMEFORMAT="%F %T  " # Set history date format (normally by default date and time of executed commands is not preserved in history)
echo $HISTTIMEFORMAT # See history date format

Edit user creation preferences:
jed /etc/adduser.conf

Create a new user:
# useradd is similar to adduser
adduser USER # add a new USER with a home directory in the default path ( /home/USER )
adduser --home DIR USER # add a new USER specifying a non standard path for the home directory
adduser --no-create-home USER # add a new USER without a home directory

Modify a user information:
usermod [OPTIONS] USER

Edit user deletion preferences:
jed /etc/deluser.conf

Remove a user:
# userdel is similar to deluser
deluser USER # remove the specified user
deluser --remove-home USER # remove the specified user and his home directory and mail spool
deluser --remove-all-files USER # remove the specified user and all files owned by the user (be careful)

Change user contact information:
chfn [OPTIONS] USER

Change a user's default shell:
chsh [OPTIONS] USER

Get a user hashed password:
getent shadow|grep "USER"|cut -f 2- -d ":"|cut -f 1 -d ":" # Get the hash type, salt and hashed password for given USER
getent shadow|grep "$(whoami)"|cut -f 2- -d ":"|cut -f 1 -d ":" # Get the hash type, salt and hashed password for current user
getent shadow|grep "$(whoami)"|cut -f 2- -d ":"|cut -f 1 -d ":"|cut -f 4- -d "$" # Get the hashed password for given USER
getent shadow|grep "$(whoami)"|cut -f 2- -d ":"|cut -f 1 -d ":"|cut -f 4- -d "$" # Get the hashed password for current user
getent shadow|grep "USER"|cut -f 2- -d ":"|cut -f 1 -d ":"|cut -f 2- -d "$"|cut -f 2- -d "$"|cut -f 1 -d "$" # Get the salt for given USER
getent shadow|grep "$(whoami)"|cut -f 2- -d ":"|cut -f 1 -d ":"|cut -f 2- -d "$"|cut -f 2- -d "$"|cut -f 1 -d "$" # Get the salt for current user
getent shadow|grep "USER"|cut -f 2- -d ":"|cut -f 1 -d ":"|cut -f 2- -d "$"|cut -f 1 -d "$" # Get the hash type for given USER (1: md5, 5: sha-256, 6: sha-512)
getent shadow|grep "$(whoami)"|cut -f 2- -d ":"|cut -f 1 -d ":"|cut -f 2- -d "$"|cut -f 1 -d "$" # Get the hash type for current user (1: md5, 5: sha-256, 6: sha-512)

Change a user password:
passwd USER

Set a user password to expire:
# note that having frequently changing passwords is usually not a good policy, as it becomes harder for users to remember them, and they generally end up with weak passwords
chage -m 3 -M 30 -w 2 USER # USER's password can last up to 30 days, can't be changed earlier than 3 days after last change, and will be warned 2 days before password expiration

Create a new user group:
# addgroup is a link to adduser ( adduser --group ). There's also a groupadd command.
addgroup GROUP

Modify a group information:
groupmod [OPTIONS] GROUP

Remove a group:
# There's also a delgroup link to deluser ( deluser --group )
groupdel GROUP

Add a user to a group:
adduser USER GROUP

Execute a command with root privileges (root password will be asked):
sudo COMMAND

Change current user's password:
passwd

Show all existing users:
getent passwd # format: username:password (usually shadowed, a 'x' is shown instead):user id:group id:real name:home path:console. If you are root you can access directly the file with cat /etc/passwd
getent shadow # Show users and their hashed password. If you are root you can access directly the file with cat /etc/shadow

Show all logged users:
users # list all logged users
who # show information about all logged users
w # show more information than "who"

Show last logged users:
last # Show last logins
last USER # show last times when USER logged in
last -n 10 # show last 10 logged users
last -n 5 USER # show last 5 times when USER logged in
touch /var/log/lastlog # Create the login log file to enable lastlog
lastlog # Show last time each user logged in
lastlog -u USER # Show last time the USER logged in
lastlog -t 30 # Show only users who logged in during the last 30 days
lastlog -b 365 # Show only users who last logged more than 365 days ago

Show failed login attempts:

To enable faillog:
-----
Create the faillog log file:
touch /var/log/faillog

jed /etc/pam.d/common-auth

Add these lines at the top of the file:

# Log failed logins to /var/log/faillog
auth required pam_tally.so per_user magic_root onerr=fail

jed /etc/pam.d/sshd

Add the following lines immediately before @include common-auth (generally at the beginning of the file):

# Log failed login attempts to /var/log/faillog
auth required pam_tally.so per_user onerr=fail

jed /etc/ssh/sshd_config

Enable PAM, search for UsePAM and set it to yes if not enabled, or add the whole line if it's missing:

UsePAM yes

Restart SSH:

service ssh restart
-----
faillog # Show all users who attempted to log in without success
faillog -a # Show all failed login attempts including the ones of users who eventually logged in
failed -u USER # Show all failed login attempts for the specified USER, even if he eventually logged in
faillog -t 30 # Show only failed logins occurred during the last 30 days
faillog -l 5 # Lock the account for 5 seconds after each failed attempt
faillog -m 5 # Disable the account after 5 failed attempts (0 means that infinite attempts are allowed. You'd better leave the value for root to 0 to prevent a DoS attack)
faillog -u USER -r # Reset the counter of failed logins for the given user, enabling his account again if it was locked because of too many failed attempts (as specified in faillog -m N)
faillog -r # Reset counters of failed logins for all users
cat /var/log/faillog # Show the actual failed attempts log file
grep "authentication failure" /var/log/messages # extract failed login attempts from the messages file

Show current user's name:
whoami

Show information about a user:
finger USER

Show groups to which current user belongs:
groups

Show user and group IDs:
id # show user and group IDs for the current user
id USER # show user and group IDs for the specified USER

Send a message to a logged user (to the output console of his terminal):
# Check the user device with a w or a who command first then redirect the output of an echo command to such device
echo -ne "Hello\n">/dev/pts/1 # Send "Hello" to the user logged with pts/1

# Send a message to a logged user (to the output console of his terminal) who's using a specific process (identified by its PID):
echo -ne "Hello\n" > /proc/PID/fd/0

Send keystrokes to a terminal:
# (use ps -A|grep "PROCESS_NAME" to find in which terminal is running a specific process, in case you need to send keystroke to a process waiting for an input)
perl -e '$TIOCSTI = 0x5412; $tty = "/dev/pts/TERMINAL"; $char = "\n"; open($fh, ">", $tty); ioctl($fh, $TIOCSTI, $char);'; # Send a single keystroke (ENTER in this example) to the given TERMINAL
perl -e '$TIOCSTI = 0x5412; $tty = "/dev/pts/TERMINAL"; $str = "TEXT_STRING\n"; for my $i (1..length($str)) { open($fh, ">", $tty); $char=substr($str, $i-1, 1); ioctl($fh, $TIOCSTI, $char); }'; # Send a TEXT_STRING to the given TERMINAL

Send a message to all logged users:
wall PATH/FILE # show the content of FILE to all logged users (max 20 lines)
wall # use standard input (normally the keyboard) to show a message to all logged users. Message must be terminated with an EOF (End Of File) character (usually CTRL+D)

Execute a command as another user (impersonated user password will be requested):
su USER COMMAND

Start a console as another user (impersonated user password will be requested):
su USER
su # if no user is specified, then root is assumed by default

-----

Search for a file:
find PATH -type f # find all files in PATH and its subdirectories
find PATH -type d # find all subdirectories in PATH and its subdirectories
find PATH -name FILE_MASK # find all files and directories with given mask inside PATH and its subdirectories. Example: find / -name a* # find all files and directories starting with a in the server
find PATH -type f -name FILE_MASK # find all files with given mask inside PATH and its subdirectories. Example: find / -type f -name *.php # find any PHP file in the server
find PATH -type d -name FILE_MASK # find all directories with given mask inside PATH and its subdirectories. Example: find / -type d -name test* # find all test directories in the server
find PATH -type f -regex ".*\.EXT$" # Find all files in PATH and its subdirectories with given EXTension
find PATH -type f -regex ".*\.[0-9]*$" # Find all files in PATH and its subdirectories with a numeric extension
whereis FILENAME # return the paths of files with matching FILENAME
locate NAME # return all paths and files with a matching NAME string
grep -r "STRING" PATH # Search for all files containing the given STRING, within the PATH and its subdirectories (-r)
grep -r -l "STRING" PATH # Search for all files containing the given STRING, within the PATH and its subdirectories (-r) and return just matching files name and not each occurrence of the string within every file (-l)
find PATH | xargs grep "STRING" -sl # Search for all files containing the given STRING, within the PATH and its subdirectories, returns only filenames with path

Get information about a file:
basename PATH/FILE # return the file name of the specified file
dirname PATH/FILE # return the path (from root) of the specified file
stat PATH/FILE # show file properties
file PATH/FILE # identify file type

s1='/path/to/file'; echo ${s1#$(dirname "$(dirname "$s1")")/}; # return the last directory and the file name of a given path/file

s1='/path/to/file'; echo ${s1#$(dirname "$s1")/}; # return the file name of a given path/file

find PATH/DIRECTORY -type f | perl -ne 'print $1 if m/\.([^.\/]+)$/' | sort -u # return all different file extensions in given DIRECTORY

Copy a file or a directory:
cp -rp SOURCE DESTINATION  # copy a file or a directory tree preserving owners and properties
mkdir -p PATH && cp -rp SOURCE $_ # copy a file or a directory tree (preserving owners and properties) and create destination path if it doesn't exist

Copy a file over a network:
apt-get install netcat
# On the Destination computer (listener). Make sure the PORT you are listening isn't blocked. Timeout (-w) is set to 3 seconds:
nc -vvn -l -p PORT -w 3 > DEST_PATH/FILE
# On the Source computer (sender). Make sure the PORT you are sending to isn't blocked. It will quit 2 seconds after receiving an EOF. If -q is not specified, then netcat must be killed manually:
nc -vvn DESTINATION_IP PORT -q 2 < SOURCE_PATH/FILE 

Copy a file securely (through SSH) over a network (it will ask for passwords if needed for authentication on remote hosts):
# Use blowfish encryption (-c blowfish) instead of the default TripleDES (3DES) because it's more secure and faster.
scp -c blowfish SOURCEPATH/NAME DESTUSER@DESTHOST:DESTPATH/NAME  # copies a file from the local system to a remote host
scp -c blowfish SOURCEUSER@SOURCEHOST:DSOURCEPATH/NAME DESTPATH/NAME  # copies a file from a remote host to the local system
scp -c blowfish SOURCEUSER@SOURCEHOST:DSOURCEPATH/NAME DESTUSER@DESTHOST:DESTPATH/NAME  # copies a file from a remote host to another remote host
scp -c blowfish -r SOURCEUSER@SOURCEHOST:DSOURCEPATH DESTUSER@DESTHOST:DESTPATH/  # copies a directory tree from a remote host to another remote host

Move a file or a directory tree (or rename):
mv SOURCEPATH/NAME DESTPATH/NAME
for a in *; do mv -v "$a" "${a//STRING_SEARCH/STRING_REPLACE}"; done  # Batch rename all files in the current directory
for a in *; do mv -v "$a" "${a}_`date --iso-8601`"; done  # Append date as YYYY-MM-DD to all file names in the current directory (note that it will be appended after any extension)
find PATH -type f -name "*" -exec mv "{}" "{}".txt \; # Add an extension (.txt in this example) to all files in given PATH
find PATH -type f -not -name "*.*" -exec mv "{}" "{}".txt \; # Add an extension (.txt in this example) to all files without an extension in given PATH
for i in `ls PATH -1`; do mv $i "${i,,}" ; done # rename all files in the given PATH to lowercase
for i in `ls PATH -1`; do mv $i "${i^^}" ; done # rename all files in the given PATH to uppercase

Synchronize two directories:
apt-get install rsync
rsync -a -v /SOURCE_PATH/* /DEST_PATH/ # copy all files from /SOURCE_PATH that doesn't exist in /DEST_PATH
rsync -a --delete -v /SOURCE_PATH/* /DEST_PATH/ # copy all files from /SOURCE_PATH that doesn't exist in /DEST_PATH, and delete all files in /DEST_PATH that doesn't (no longer) exist in /SOURCE_PATH
rsync -a --delete -v rsync://www.example.com/dir/ /DEST_PATH # synchronize from an external rsync directory

Create a link to a file:
ln -s PATH/ACTUAL_FILE PATH/LINK_TO_CREATE

Create an empty file:
touch PATH/NAME # Create an empty file. If the given file already exists, it updates its access date

Create a unique temporary file or directory with a pseudo random name:
mktemp "tmp_XXXXXX" # Create an empty file with a random name generated from the given mask, for example tmp_92pIKB or tmp_Z1Pdz1
mktemp -d "tmp_XXXXXX" # Create an empty directory with a random name generated from the given mask, for example tmp_olD9rA or tmp_5Fw0JI
Sample usage of mktemp (from mktemp man):
TMPFILE=`mktemp /tmp/example.XXXXXX` || exit 1
echo "program output" >> $TMPFILE

Split a file in chunks:
split -b=BYTES PATH/FILE PREFIX # split FILE in chunks of BYTES bytes, using PREFIX to name generated chunks
split -l LINES TEXT_FILE PREFIX # split TEXT_FILE in chunks of LINES lines, using PREFIX to name generated chunks

Rebuild a split file (concatenate files):
cat PREFIX* > PATH/FILE
sed -e '$a\' test?.txt > testx1b.txt # concatenate text files appending an empty line at the end of every line if there isn't already one
(for r in PREFIX*;do sed s/\$/\ $r/ < "$r";done) > PATH/FILE # concatenate text files appending the file name after each line
cat $(ls PREFIX* -t) > PATH/FILE # concatenate files sorted by date and time (newer first, older last)
cat $(ls PREFIX* -tr) > PATH/FILE # concatenate files sorted by date and time (older first, newer last)

Type a file:
cat PATH/FILE
cat -n PATH/FILE # show a text file with line numbers
cat PATH/FILE | tr [:upper:] [:lower:] # show a text all in lower case
cat PATH/FILE | tr [:lower:] [:upper:] # show a text all in Upper case
tac PATH/FILE # show a text from the last line to the first (thus listing lines backwards), useful to ordering elements in a last-in, first-out (LIFO) way
fold -w COLOUMN PATH/FILE # wrap a text file, cutting lines at the given coloumn. Useful to format data as input for a program
fmt -u -w80 file.txt # format a text file output: 1 space between words, 2 spaces between sentences, 80 coloumns
nl PATH/FILE  # show line numbers for every line (empty lines are not counted by default)
nl -ba PATH/FILE # show line numbers for every line and count all lines, including empty ones
nl -s: PATH/FILE # show line numbers for every line and separate line numbers from line text using the specified separator (":" in this case)
pr -l 50 test.txt # show a text as pages, each page is 50 lines long (minimum 11, default is 66)
pr -l 50 --column 2 test.txt # show a text as pages, each page is 50 lines long and divided into 2 columns
more PATH/FILE  # type a file page by page
less PATH/FILE  # show a file page by page, and let scroll through pages. It shows escaped characters as plain text (get it with apt-get install less)
less -F PATH/FILE  # show a file page by page, and let scroll through pages (get it with apt-get install less). If the file output fits a single screen it quits without having to press "q"
grep "TEXT" PATH/FILE  # show only lines containing "TEXT"
grep -v "TEXT" PATH/FILE  # show only lines NOT containing "TEXT"
grep -i "TEXT" PATH/FILE  # show only lines containing "TEXT" (case insensitive)
grep -vi "TEXT" PATH/FILE  # show only lines NOT containing "TEXT" (case insensitive)
grep -B 1 -A 1 "TEXT" PATH/FILE  # show lines containing "TEXT" and show 1 line Before and 1 line After each match
grep -B 2 -A 2 -n "TEXT" PATH/FILE  # show lines containing "TEXT" and show 2 lines Before and 2 lines After each match, together with line numbers
head PATH/FILE  # show the head (first 10 lines) of FILE
head --lines=20 PATH/FILE  # show the first 20 lines of FILE
head --bytes=1024 PATH/FILE  # show the first 1024 bytes of FILE
head --lines=-50 PATH/FILE  # show all the FILE but the last 20 lines
head --bytes=-2048 PATH/FILE  # show all the FILE but the first 2048  bytes
tail PATH/FILE  # show the tail (last 10 lines) of FILE
tail --lines=20 -F PATH/FILE  # show the last 20 lines of FILE, and new lines as the file grows. If the files doesn't exist yet, it waits for it (useful for log files)
tail --lines=+20 -F PATH/FILE  # show the last lines of FILE starting from line 20
tail --bytes=1024 PATH/FILE  # show the last 1024 bytes of FILE
tail -f PATH/FILE # keep showing the last line of FILE whenever there are added new ones (useful for log files) until stopped
tail -f PATH/FILE | sed '/^ENDSTRING$/ q' # keep showing the last line of FILE whenever there are added new ones (useful for log files) until a given string appears
sed -ne "3p" PATH/FILE  # show line number 3 of FILE
sed -ne "2,4p" PATH/FILE # show lines 2 to 4 of FILE
sed -ne "1,5p" PATH/FILE # show lines 1 (beginning) to 5 of FILE
sed -ne "5p,$" PATH/FILE # show lines 5 to the end of FILE
sed -n -e '3,5p' -e '9,15p' PATH/FILE # show lines 3 to 5 and 9 to 15 of FILE
sed -e "2,30d" PATH/FILE # show all lines of FILE excepts for the ones from 2 to 30 included

Hexadecimal dump of a file:
od -Ax -tx1z -v PATH/FILE # hex dump of FILE
od -Ax -tx1z -v PATH/FILE | less # navigate the hex dump of FILE

Extrapolate strings of text from binary files:
strings -a -n 4 PATH/FILE # extrapolate strings of text from binary files (a string of text is considered at least 4 bytes long, you can change this value with the option -n) (get it with apt-get install binutils )
strings -a -t x PATH/FILE # extrapolate strings of text from binary files and return the offset of the string (-t x specifies the offset as hex, -t d decimal, -t o octal)

Count lines in text files:
wc PATH/FILE # line/word/byte count (handy options: --lines or -l print only the newline counts, --words print only the word counts, --bytes print only the byte counts, --chars print the character counts)
wc -L PATH/FILE # print the lenght of the longest line of FILE
grep "STRING" PATH/FILE|wc --lines # count how many lines in FILE contains STRING
wc -l PATH/*.txt # count the lines of all files with .txt extension in a directory
find PATH -name "*.txt" | xargs wc -l # count the lines of all files with .txt extension in a directory and its subdirectories

Sort a text file:
sort PATH/FILE.txt # sort FILE.txt alphabetically, case sensitive
sort -i PATH/FILE.txt # sort FILE.txt alphabetically, cosindering only printable characters
sort -d PATH/FILE.txt # sort FILE.txt alphabetically, cosindering only dictionary characters (alphanumeric characters and blanks)
sort -f PATH/FILE.txt # sort FILE.txt alphabetically, case insensitive
sort -fs PATH/FILE.txt # sort FILE.txt alphabetically, case insensitive and stable (sort according which case comes first for each letter)
sort -b PATH/FILE.txt # sort FILE.txt ignoring leading blank spaces
sort -r PATH/FILE.txt # sort FILE.txt in reverse alphabetical order
sort -n PATH/FILE.txt # sort numbers in FILE.txt
sort -h PATH/FILE.txt # sort human readable numbers (like 5K, 2G) in FILE.txt
sort -M PATH/FILE.txt # sort months (like Jan, Feb, Mar) in FILE.txt
sort -R PATH/FILE.txt # sort lines randomly in FILE.txt
sort -u PATH/FILE.txt # sort lines in FILE.txt and keep only unique lines
sort -k 2 PATH/FILE.txt # sort lines in FILE.txt using characters from column 2 (any blank character separates columns)
sort -t : -k 2 PATH/FILE.txt # sort lines in FILE.txt using characters from column 2 using ":" as a separator for columns
sort PATH/FILE.txt -o "PATH/DEST_FILE.txt" # sort FILE.txt and save the output in DEST_FILE.txt

More file operations:
# All these commands should work also with "awk", the older version of "nawk"
nawk '1; { print "" }' SOURCE>DESTINATION # Add an empty line after each line
nawk 'NF { print $0 "\n" }' SOURCE>DESTINATION # Add an empty line after each line, except when an empty line is already present
nawk '{ print FNR ".\t" $0 }' SOURCE>DESTINATION # Add line numbers followed by dot and tab before than each line
nawk '{sub(/^[ \t]+/, "")};1' SOURCE>DESTINATION # Remove leading spaces from each line
nawk '{sub(/[ \t]+$/, "")};1' f1.txt SOURCE>DESTINATION # Remove trailing spaces from each line
nawk '{gsub(/^[ \t]+|[ \t]+$/,"")};1' SOURCE>DESTINATION # Remove leading and trailing spaces from each line
nawk '{sub(/A/,"B")}; 1' SOURCE>DESTINATION # Replace the first occurrence of A with B in each line
nawk '{gsub(/A/,"B")}; 1' SOURCE>DESTINATION # Replace every occurrence of A with B in each line
nawk '/X/{gsub(/A/, "B")}; 1' SOURCE>DESTINATION # Replace every occurrence of A with B in each line that contains X
nawk '!/X/{gsub(/A/, "B")}; 1' SOURCE>DESTINATION # Replace every occurrence of A with B in each line that does not contain X
nawk 'BEGIN{while (a++<100) s=s "."; print s}' >DESTINATION # Create a file containing 100 dots (.)
nawk 'NR==2 {print;exit}' SOURCE # Print line number 2 from SOURCE file
nawk 'NR==2,NR==4' SOURCE # Print line numbers 2 to 4 (inclusive) from SOURCE file
nawk 'END{print NR}' SOURCE # count lines (emulates "wc -l")

Return the average lenght in lines for all text files in the tree:
find PATH -name "*.txt" -exec wc -l {} \; | nawk 'BEGIN {x=0;y=0} {x+=1; y+=$1} END {print y/x}'

Count the occurrences of all the words in a text file: 
tr -c a-zA-Z '\n' < PATH/FILE.txt | sed '/^$/d' | sort -n | uniq -ci | sort -nr # case insensitive
tr -c a-zA-Z '\n' < PATH/FILE.txt | sed '/^$/d' | sort -n | uniq -c | sort -nr # case sensitive

Count the occurrences of a string within another string:
needle=","; haystack="one, two, three, four, five"; occurrences=$(grep -o "$needle" <<< "$haystack" | wc -l); echo $occurrences

Truncate a string:
# The marker (delimiter) to set the point where to truncate a string is ":" in the following examples
cut -d ":" -f1 <<< "this:is:an:example" # Truncate a string at the first occurrence of a marker (removing the marker). Example: before: "this:is:an:example", after: "this".
cut -d ":" -f2- <<< "this:is:an:example" # Truncate a string after the first occurrence of a marker (removing the marker). Example: before: "this:is:an:example", after: "is:an:example".
echo "this:is:an:example" | cut -d ":" -f1 # Truncate a string at the first occurrence of a marker (removing the marker). Example: before: "this:is:an:example", after: "this".
echo "this:is:an:example" | cut -d ":" -f2- # Truncate a string after the first occurrence of a marker (removing the marker). Example: before: "this:is:an:example", after: "is:an:example".
echo "this:is:an:example" | rev | cut -d ":" -f2- | rev # Truncate a string up to the last occurrence of a marker (removing the marker). Example: before: "this:is:an:example", after: "this:is:an".
echo "this:is:an:example" | rev | cut -d ":" -f1 | rev # Truncate a string after the last occurrence of a marker (removing the marker). Example: before: "this:is:an:example", after: "example".

Clear screen:
clear # You can still scroll back to see the text previously on screen

Return a text string:
echo "STRING" # print STRING
echo -n "STRING" # print STRING without new line
echo {A,B,C}{A,B,C} # return all possible combinations of given characters (this example returns AA AB AC BA BB BC CA CB CC )

Colored text:
# Note: Colors are not a POSIX (Portable Operating System Interface) feature in the Bash shell, so it might not be shown in some systems
echo -e "\e[1;37m WHITE  \e[0;37m LIGHT GRAY  \e[1;30m GRAY  \e[0;30m BLACK  \e[0;31m RED  \e[1;31m LIGHT RED  \e[0;32m GREEN  \e[1;32m LIGHT GREEN  \e[0;33m YELLOW  \e[1;33m LIGHT YELLOW  \e[0;34m BLUE  \e[1;34m LIGHT BLUE  \e[0;35m PURPLE  \e[1;35m PINK  \e[0;36m CYAN  \e[1;36m LIGHT CYAN  \e[0m DEFAULT";

Remove colors and other control characters from a text file:
cat -v PATH/FILE | sed "s/\^\[\[.\;..m//g" | sed "s/\^\[\[.m//g" | sed "s/\^M$//g"

Type a text string (y by default) until interrupted:
yes STRING

Convert character encoding:
apt-get install recode
recode iso-8859-15..utf8 < FILE.TXT > file.utf8 # recode FILE.TXT from Latin Western Europe to UTF8
recode utf8..iso-8859-15 < FILE.UTF8 > file.txt # recode FILE.UTF8 from UTF8 to Latin Western Europe
recode ../b64 < FILE.TXT > file.b64 # recode FILE.TXT to Base 64
recode /qp.. < FILE.TXT > file.qp # recode FILE.TXT to quoted printable

Convert New Line Format:
sed -i "s/\r//" PATH/SOURCE_FILE>PATH/DESTINATION_FILE # From \r\n (Windows) to \n (Unix/Linux/Mac)
sed -i "s/\n/\r\n/" PATH/SOURCE_FILE>PATH/DESTINATION_FILE # From \n (Unix/Linux/Mac) to \r\n (Windows)

Convert tabs to spaces in a text file:
expand PATH/FILE  # convert only tabs at the beginning of the line
expand -a PATH/FILE  # convert all tabs, not just the ones at the beginning of the lines

Convert spaces to tabs in a text file:
unexpand PATH/FILE  # convert all tabs, not just the ones at the beginning of the lines
unexpand -i PATH/FILE # convert only spaces at the beginning of the lines

Search and Replace:
sed "s/SOURCE_STRING/REPLACEMENT/g" PATH/SOURCE_FILE>PATH/DESTINATION_FILE # Replace all occurrences of a string inside a text file, case sensitive
sed "s/SOURCE_STRING/REPLACEMENT/gi" PATH/SOURCE_FILE>PATH/DESTINATION_FILE # Replace all occurrences of a string inside a text file, case insensitive
sed "s/[Bb]ig/small/g" PATH/SOURCE_FILE>PATH/DESTINATION_FILE # Replace the word Big or big to small
sed "10,20 s/SOURCE_STRING/REPLACEMENT/gi" PATH/SOURCE_FILE>PATH/DESTINATION_FILE # Replace all occurrences of a string inside a text file, case insensitive, and process only lines 10 to 20 included
sed "/MATCH_STRING/ s/SOURCE_STRING/REPLACEMENT/g" PATH/SOURCE_FILE>PATH/DESTINATION_FILE # Replace SOURCE_STRING with REPLACEMENT only in lines where MATCH_STRING is present

Replace multiple spaces with a single space in a text file:
sed "s/ */ /g" PATH/SOURCE_FILE>PATH/DESTINATION_FILE

Remove comments from a text file:
sed "/ *#/d;" PATH/SOURCE_FILE>PATH/DESTINATION_FILE

Remove blank lines from a text file:
sed "/^ *$/d" PATH/SOURCE_FILE>PATH/DESTINATION_FILE

Remove comments and blank lines from a text file:
sed "/ *#/d; /^ *$/d" PATH/SOURCE_FILE>PATH/DESTINATION_FILE

Remove trailing spaces and tabs from a text file:
sed "s/[ \t]*$//" PATH/SOURCE_FILE>PATH/DESTINATION_FILE

Add a text at the end of each line:
sed "s/$/\;/g" PATH/SOURCE_FILE>PATH/DESTINATION_FILE  # Append a semicolon (";") at the end of each line

Add a text at the beginning of each line:
sed "s/^/LINE : /g" PATH/SOURCE_FILE>PATH/DESTINATION_FILE  # Add "LINE : " at the beginning of each line

Add blank lines after every line in a text file:
sed "G" PATH/SOURCE_FILE>PATH/DESTINATION_FILE # Add 1 blank line after every line in a text file
sed "G;G" PATH/SOURCE_FILE>PATH/DESTINATION_FILE # Add 2 blank lines after every line in a text file

Append a line of text to a file:
echo "line 1" >> PATH/FILE.txt

Append multiple lines of text to a file (in scripts):
echo "line 1
line 2" >> PATH/FILE.txt

Append multiple lines of text to a file (in scripts):
cat <<EOT >> PATH/FILE.txt
line 1
line 2
EOT

Case conversion:
echo "Test"|tr '[:lower:]' '[:upper:]'  # Return the text in Upper case
echo "Test"|tr '[:upper:]' '[:lower:]'  # Return the text in lower case
cat PATH/SOURCE_FILE|tr '[:lower:]' '[:upper:]'>PATH/DESTINATION_FILE  # Convert a text file to Upper case
cat PATH/SOURCE_FILE|tr '[:upper:]' '[:lower:]'>PATH/DESTINATION_FILE  # Convert a text file to lower case

Extract items from a delimited text file:
cut -d: -f1 PATH/SOURCE_FILE>PATH/DESTINATION_FILE  # choose the first item of a series delimited by ":"

Remove consecutive duplicate lines from a text file:
uniq PATH/FILE
uniq -u PATH/FILE # return only successive duplicated lines

Reverse lines of a file
rev PATH/TEXTFILE > PATH/REVTEXTFILE # Reverse the lines of a given file(s)

Get the checksum of a file:
sum PATH/FILE # return the checksum and block count for FILE
sha1sum PATH/FILE # return the SHA1 (160-bit) checksum for FILE
cksum PATH/FILE # return the CRC checksum and byte count for FILE

Compare and show differences between two binary files:
cmp -b -l PATH/FILE1 PATH/FILE2 # compare two files and returns the position of the different bytes followed by the value and the corresponding ASCII character for each file
cmp -b -l --bytes=10 PATH/FILE1 PATH/FILE2 # limit comparison to the first 10 bytes

Compare and show differences between two text files:
diff -abB PATH/FILE1 PATH/FILE2 # Report differences between two text files
diff -abBi PATH/FILE1 PATH/FILE2 # Report differences between two text files ignoring case differences in their content

Compare and show differences between three text files:
diff3 -e FILE1 FILE2 FILE3 # output unmerged changes from FILE2 to FILE3 into FILE1

Truncate a file:
cp /dev/null PATH/FILE # empty FILE. If FILE doesn't exist, it will be created.
truncate -s 0 PATH/FILE # truncate FILE to 0 bytes (empty FILE). If FILE doesn't exist, it will be created.
truncate -s 0 -c PATH/FILE # truncate FILE to 0 bytes (empty FILE). If FILE doesn't exist, it will not be created.
truncate -s 100 PATH/FILE # truncate FILE to 100 bytes. If FILE doesn't exist, or it's shorter than the given size, it will be filled by zeroes (0).
truncate -r 100 PATH/REFFILE PATH/FILE # truncate FILE to the size of REFFILE.

Split or limit lines in a text file:
cut -c 80 PATH/FILE.txt # cut every line of FILE.txt to 80 characters (extra characters will be lost)
fold -w 80 PATH/FILE.txt # split every line of FILE.txt after 80 columns (extra characters will be wrapped to the next line)
fold -w 80 -s PATH/FILE.txt # split every line of FILE.txt after 80 columns, breaking at spaces (full words will be preserved)

Delete lines from a text file:
sed -e '10d' PATH/FILE.txt > PATH/DESTFILE.txt # remove line 10
sed -e '5,15d' PATH/FILE.txt > PATH/DESTFILE.txt # remove lines 5 to 15

Empty files:
# Delete the content of a file without removing the file. It can be useful to don't break programs which expect to find the file, or before to delete a very large file.
cp /dev/null > PATH/NAME # Empty a file. It will also work as "> PATH/NAME" (without quotes)
echo -n '' > PATH/NAME # Empty a file
find PATH -type f -exec cp /dev/null {} \; # Empty all files in PATH and its subdirectories
find PATH -type f -regex ".*\.EXT$" -exec cp /dev/null {} \; # Empty all files in PATH and its subdirectories with given EXTension
find PATH -type f -regex ".*\.[0-9]*$" -exec cp /dev/null {} \; # Empty all files in PATH and its subdirectories with a numeric extension

Delete files:
rm PATH/NAME
rm -i PATH/NAME # ask for confirmation before to delete
find PATH -maxdepth 1 -type f -empty -delete # Delete all empty files in the given PATH (also: find PATH -maxdepth 1 -type f -size 0 -delete )
find PATH -maxdepth 1 -type f ! -empty -delete # Delete all non empty files in the given PATH (also: find PATH -maxdepth 1 -type f -size +0 -delete )
find PATH -type f -empty -delete # Delete all empty files in the given PATH and its subdirectories (also: find PATH -type f -size 0 -delete )
find PATH -type f ! -empty -delete # Delete all non empty files in the given PATH and its subdirectories (also: find PATH -type f -size +0 -delete )
find /var/www/test/t5 -type f -delete # Delete all files in /var/www/test/t5 and its subdirectories
find /var/www/test/t5 -type f -regex ".*\.EXT$" -delete # Delete all files in /var/www/test/t5 and its subdirectories with given EXTension
find /var/www/test/t5 -type f -regex ".*\.[0-9]*$" -delete # Delete all files in /var/www/test/t5 and its subdirectories with a numeric extension

Wipe a file (secure file deletion):
# Important note: wiping overwrites a file one or more time before to delete it, to make it unrecoverable. This technique does not work with journaled file systems, like ReiserFS, Reiser4, or ext3 in journaled mode. It also doesn't work with solid state drives (such as SD cards).
apt-get install wipe # if it doesn't work, try apt-get install secure-delete
wipe PATH/FILE # wipe a file (asks for confirmation, write protected files will be spared)
wipe -c -f PATH/FILE # wipe a file
wipe -c -f -s PATH/FILE # wipe a file, quiet mode (no output returned except for fatal errors like "No such file or directory")
wipe -c -f -k PATH/FILE # wipes the content of a file, but keeps the file in place (with a lenght of 0 bytes)
wipe -c -f -e PATH/FILE # wipes the file up to its exact size, without wiping extra data in the last used block

Undelete a file:
apt-get install testdisk # PhotoRec
photorec MEDIA # attempt to recover deleted files from given MEDIA (for example /dev/sda2 ). You'll have then to confirm the media, choose partition, select recovery options, specify the file type of the files you want to recover, specify the filesystem type where the files were stored, choose if you want to search for deleted files in the empty space or the whole partition, choose the destination directory (in a separate disk) where you want to save the recovered files.

Create a directory:
mkdir PATH
mkdir -p PATH # Create the whole path (not just the final directory in the path) if it doesn't exist

Delete a directory (must be empty):
rmdir PATH
find PATH -depth -type d -empty -exec rmdir {} \; # remove empty subdirectories within given PATH (recursive)

Delete a directory tree (recursive):
rm -rf PATH

Delete a directory with a large number (thousands and more) of files:
mkdir /empty/
rsync -a --delete /empty/ /PATH/dir/ # Note: if you haven't just created it, make sure the /empty/ directory is actually empty
rmdir /PATH/dir/

Wipe a directory tree (recursive secure file deletion):
apt-get install wipe
wipe -c -r PATH # wipe a directory tree (asks for confirmation)
wipe -c -f -r PATH # wipe a directory tree
wipe -c -f -r -s PATH # wipe a directory tree, quiet mode (no output returned except for fatal errors like "No such file or directory")

Show current directory:
pwd

Change directory:
cd PATH
pushd PATH # change directory and remember the current directory in the dirs stack
popd # return to the previous directory in the dirs stack
dirs # (or dirs -l ) list directories in the dirs stack keeping the output in a single line
dirs -p # list directories in the dirs stack showing the output in multiple lines
dirs -v # list directories in the dirs stack showing the output in multiple lines and line numbers
dirs -c # clear the dirs stack

Show the content of a directory:
ls PATH
ls -laF --color=auto PATH # extended information and colored output
ls -laFXB PATH # sort by extension
ls -laFSr PATH # sort by size, small to large
ls -laFS PATH # sort by size, large to small
ls -laFtr PATH # sort by size, old to recent
ls -laFt PATH # sort by date, recent to old
ls -laFtur PATH # sort by access date, old to recent
ls -laFtu PATH # sort by access date, recent to old
ls -A # show hiddem files
ls --group-directories-first # show directories first
ls -d */. # list only directories within current directory
ls -d PATH/*/. # list only directories within given PATH
ls -R PATH | grep ":$" | sed -e 's/:$//' -e 's/[^-][^\/]*\//--/g' -e 's/^/ /' -e 's/-/|/' # show directory tree (add | less at the end of the line to navigate the tree up and down)
find PATH -ls # The -ls parameters make find return results as the output of ls . You may also add the -print0 parameter and pipe to xargs -0 ls -lt (that is: -print0 | xargs -0 ls -lt )
find PATH -type f -mtime -30 # list files modified within last 30 days
find PATH -depth -type d -empty -exec echo {} \; # list empty subdirectories within given PATH (recursive)
find PATH -maxdepth 1 -type f -empty # Find all empty files in the given PATH (also: find PATH -maxdepth 1 -type f -size 0 )
find PATH -maxdepth 1 -type f ! -empty # Find all non empty files in the given PATH (also: find PATH -maxdepth 1 -type f -size +0 )
find PATH -type f -empty # Find all empty files in the given PATH and its subdirectories (also: find PATH -type f -size 0 )
find PATH -type f ! -empty # Find all non empty files in the given PATH and its subdirectories (also: find PATH -type f -size +0 )

Compare and show differences between two directories:
diff PATH/DIR1 PATH/DIR2 # Report different files between two directories
diff -q PATH/DIR1 PATH/DIR2 # Report different files between two directories without showing the different content of files with identical file names
diff -s PATH/DIR1 PATH/DIR2 # Report different files and identical files between two directories
diff --ignore-file-name-case PATH/DIR1 PATH/DIR2 # Report different files between two directories ignoring case differences in file names
diff -i PATH/DIR1 PATH/DIR2 # Report different files between two directories ignoring case differences in the content of text files
diff -r PATH/DIR1 PATH/DIR2 # Report different files between two directories and all their subdirectories
dir1="PATH/DIR1"; dir2="PATH/DIR2"; diff -q "$dir1" "$dir2" | grep "$dir2" | sed "s/$dir2\///g" | awk '{print $4}' # Report different files and files that are only present in DIR2

Return the full path of a file
realpath PATH/FILE

Change access permissions:
# Users: uuugggooo (ugo = u: owner user, g: all the users of the group, o: all other users)
# Modes: 0: ---, 1: --x, 2: -w-, 3: -wx, 4: r--, 5: r-x, 6: rw- , 7: rwx
chmod MOD FILE # Change permissions for a file
chmod -v MOD FILE # Change permissions for a file, with verbose output
chmod -R MOD DIRECTORY # Change permissions for a directory tree (recursive)
chmod -vR MOD DIRECTORY # Change permissions for a directory tree (recursive), with verbose output
find PATH -type f -exec chmod MOD {} + # Change permissions for all files (only files, not directories) in a directory tree (recursive)
find PATH -type d -exec chmod MOD {} + # Change permissions for all directories (only directories, not files) in a directory tree (recursive)

Change owner:
chown OWNER FILE # Change the owner of a file
chown OWNER:GROUP FILE # Change the owner and the group of a file
chown -R OWNER DIRECTORY # Change the owner of a directory tree (recursive)
chown -R OWNER:GROUP DIRECTORY # Change the owner and a group of a directory tree (recursive)

Change group:
chgrp OWNER FILE # Change the group of a file
chgrp -R OWNER DIRECTORY # Change the group of a directory tree (recursive)

File Permissions Backup and Restore
getfacl -R / > PATH/fsperms.txt # Back up file system permissions to a text file
setfacl --restore=PATH/fsperms.txt # Restore file system permissions from a text file

Raw copy:
dd if=/dev/fd0 of=/dev/fd1 # copy a floppy disk (using two floppy disk drives)
dd if=/dev/hda0 of=/dev/hda1 # copy a hard disk (or partition) into another one
dd if=/dev/hda | gzip -9v | dd of=/mnt/hdb/hda.img # back up the whole hard disk as a gzipped file (you may need to mount a partition of another hard disk as hdb: mount /dev/hdb2 /mnt/hdb )
dd of=/mnt/hdb/hda.img | gzip | if=/dev/hda # restore the gzipped image of a hard disk
dd if=/dev/fd0 of=/tmp/disk.bak # copy the whole floppy disk as a file
dd if=/dev/hda of=/tmp/mbr.bin count=1 bs=512 # back up MBR (boot loader and partition table)

Create and edit disk partitions:
apt-get install parted
parted # starts the partition utility
parted commands:
print # show current partitions
mkpart primary START END # create a new primary partition
mkpart extended START END # create a new extended partition
mkpart logical START END # create a new logical partition (within the range of an extended one)
resize PARTITON_N NEW_START NEW_END # resize an existing partition
rm PARTITION_N # remove a partition
quit # quit parted

Show available devices:
fdisk -l

Show device partitions:
fdisk -l /dev/DEVICE

Make a file system (format) a disk (partition):
mkfs.ext3 /dev/hda1 # Format hda1 as an ext3 partition
mkfs -V -t ext3 /dev/hda1 # Format hda1 as an ext3 partition (Verbose output)
mkfs -t msdos  /dev/hda2 # Format hda1 as a MS-DOS partition
mkfs -t msdos /dev/fd0 # Format a floppy disk using the MS-DOS file system
mkswap /dev/hda2 # Format hda2 as the swap partition

dd if=/dev/zero of=/dev/foo1 bs=512 count=1 # If you created or changed a DOS partition, you can use dd to zero the first 512 bytes

Create a RAMDisk:
mkdir -p /media/ramdisk
mount -t tmpfs -o size=2048M tmpfs /media/ramdisk # 2GB RAMDisk, it can use swap partition if needed, and doesn't preallocate disk space.
mount -t ramfs -o size=1024M ramfs /media/ramdisk # 1GB RAMDisk, it only uses physical RAM, not the swap partition, and preallocates disk space. Make sure you have enough spare RAM.
umount /media/ramdisk # unmount RAMDisk

Access a XFS partition:
# You typically need this to read disks from NAS devices, via USB.
mkdir /ramdisk/xfs1 # If you are using a Linux boot disk, like Knoppix, you'll probably need to mount the XFS partition in the RAMDisk
fdisk -l # Use fdisk to locate the XFS partition
mount -t xfs /dev/XFS_PARTITION /ramdisk/xfs1 # use the path to XFS partition you've found using fdisk instead of /dev/XFS_PARTITION
cp -rp /ramdisk/xfs1/* /DESTINATION # copy all files from the XFS formatted partition to another device ( /DESTINATION )

Check and repair a file system:
fsck
fsck -f -y FILE_SYSTEM # Note: Generally the device containing the file system to check can be /dev/hda0 on a single hard drive system or /dev/md0 on a RAID system); -y automatically reply yes (y) to all questions. Note that if it asks to connect to lost+found more than once, you may have some damaged files.

List RAID arrays:
mdadm --examine --scan

Check if a RAID array is up and working properly:
mdadm --detail /dev/md0 # assuming your RAID volume is /dev/md0

Check if two devices are in a RAID array:
mdadm --examine --scan /dev/sda1
mdadm --examine --scan /dev/sdb1
# Must return same MD array and UUID. Path to actual devices in your system may differ from /dev/sda1 and /dev/sdb1 of this example

Copy partitions from a device to another one:
sfdisk -d /dev/sda | sfdisk /dev/sdb # copy partitions from sda to sdb

Choose what partitions should be automatically mounted at system startup:
jed /etc/fstab # edit static file system information

Show memory usage:
free
free -m # show values in Megabytes
free -mh | xargs | awk '{print "Memory: Size: " $8 " Used: " $9 " Free: " $10}';

Virtual memory statistics: 
vmstat # report information about processes, memory, paging, block IO, traps, and cpu activity.
vmstat -s # event counters and memory statistics
vmstat -d # disk statistics

Show free disk space:
df -T # show free disk space, returns also file system type 
df -Th # show human readable values (in Megabytes, Gigabytes...)
df -h | xargs | awk '{print "Disk ("$8"): Size: " $9 " Used: " $11 " Free: " $10}';
df -P -h | nawk '0+$5 >= 90 {print "FS: "$1" ("$6") Size: "$2" Used: "$3" (\033[1;31m"$5"\033[0m) Free: "$4" (\033[1;31m"(100-$5)"%\033[0m)";}' # Only show near full file systems

Summarize disk usage:
# du shows size in KBytes by default (switch: -k), if you want to see the size in bytes, use the switch -b
du PATH/FILE # disk usage for specified file
du -s PATH # disk usage for the whole directory (recursively), total size
du -s --exclude=*.mp3 PATH # disk usage for the whole directory (recursively), total size, excluding all files ending with .mp3
du PATH # disk usage for the whole directory (recursively)
du -a PATH # disk usage for the whole directory (recursively), showing disk usage for all files and directories, not just directories
du -S PATH # disk usage for the whole directory (recursively, but size is separated for each directory)
du -a PATH | sort -nr | less # show directories and files sorted by size for the given PATH and its subdirectories
du -a PATH | sort -nr | head -n 25 # show the 25 largest directories and files within the given PATH . Use root ( / ) as PATH to see the largest directories and files in the whole disk
find PATH -type f -exec file -b '{}' \; -printf '%s\n' | awk -F , 'NR%2 {i=$1} NR%2==0 {a[i]+=$1} END {for (i in a) printf("%12u %s\n",a[i],i)}' | sort -nr # show the size occupied by files within a given PATH sorted by file type (slow)

Show deleted files locked by a running process:
# If you can't see more free space in your disk after deleting a large file it's probably locked by a running process so that it has been unlinked but not actually deleted yet. In this case you'll have to kill, stop, or restart the process to unlock the file and free its disk space
lsof +L1

Count files and subdirectories within directories:
ls -1 /PATH/ | wc -l # count files and subdirectories within a directory
ls -1R /PATH/ | wc -l # count files and subdirectories within a directory including subdirectories (recursively)
ls -1 /PATH/ | grep -i .*.EXT | wc -l # count files and subdirectories with a given EXTension within a directory
ls -1R /PATH/ | grep -i .*.EXT | wc -l # count files and subdirectories with a given EXTension within a directory including subdirectories (recursively)

Count files within directories:
find PATH -maxdepth 1 -type f | wc -l # count files within a directory
find PATH -type f | wc -l # count files within a directory including subdirectories (recursively)
find PATH -maxdepth 1 -type f | grep -i .*.EXT | wc -l # count files with a given EXTension within a directory
find PATH -type f | grep -i .*.EXT | wc -l # count files with a given EXTension within a directory including subdirectories (recursively)

Show mounted devices:
mount

Mount a device:
mount DEVICE # example: mount /dev/cdrom
mount -t FILESYSTEM DEVICE MOUNTPOINT # Mount a DEVICE at MOUNTPOINT as a given FILESYSTEM; example: mount -t ext3 /dev/sda1 /test

Mount an ISO image of a CD:
mount -o loop cd.iso /mnt/DIR_NAME/

Unmount a device:
umount DEVICE # example: umount /dev/cdrom 

Unmount and eject a device (like a CD-ROM):
eject # eject default device
eject -d # show default device
eject -r # eject CD-ROM

Force changed blocks cached in RAM to disk:
sync

-----

Mount a remote directory tree:
apt-get install sshfs fuse-utils

To let other users to handle the remote directory tree:
# create a group called fuse containing users allowed to mount the remote directory tree
chgrp fuse /usr/bin/fusermount
chmod u+s /usr/bin/fusermount
adduser USER fuse

Create the local directory where the remote directory tree will be mounted:
mkdir /mnt/sshdir
chown USER /mnt/sshdir

Mount a remote directory tree:
sshfs USER@HOST:/PATH /mnt/sshdir

-----

Install VIM editor:
atp-get install vim

Install Jed editor:
apt-get install jed

-----

Configure boot devices and partitions:

# see GNU GRUB (GRand Unified Bootloader) configuration:
less /boot/grub/grub.cfg
# /boot/grub/grub.cfg replaces /boot/grub/menu.lst and it's not meant to be edited
# in distros other than Debian the path could be /boot/grub/grub.conf or /etc/grub.conf

# edit LILO (LInux LOader) configuration:
jed /etc/lilo.conf

-----

Print a file on a printer:
lpr -Pprinter PATH/FILE
lpr -Pprinter -#N PATH/FILE # print N copies of FILE on printer

List print queue:
lpq

Remove a job from the print queue:
lprm JOB_ID
lprm # remove all jobs from the print queue

-----

Indentation:

Indent C code (or similarly formatted code):
apt-get install indent

indent PATH/SOURCE -o PATH/DEST
indent -nprs PATH/SOURCE -o PATH/DEST # Doesn't add a space after an open bracket and before a closed one

sed -i 's/^[[:space:]]*//g' PATH/FILE # Remove indentation from any text file

Indent HTML code:
apt-get install hindent

hindent PATH/SOURCE.HTML>PATH/DEST.HTML
hindent -i 0 PATH/SOURCE.HTML>PATH/DEST.HTML # Remove indentation
hindent -c PATH/SOURCE.HTML>PATH/DEST.HTML # Force all tags to lowercase (otherwise all tags are forced to UPPERCASE by default)
hindent -c -i 2 PATH/SOURCE.HTML>PATH/DEST.HTML # Set indentation level to 2 and force all tags to lowercase
hindent -c PATH/SOURCE.HTML>PATH/DEST.HTML # Force all tags to lowercase
hindent -f PATH/SOURCE.HTML>PATH/DEST.HTML # Flow: dump only HTML tags discarding any data between
hindent -l PATH/SOURCE.HTML>PATH/DEST.HTML # List all HTML tags contained in the source document

-----

Scheduled jobs with CronTab:

# You can change the editor for Crontab (parameters in the following commands are case sensitive):
export EDITOR=jed
export VISUAL=jed

# Edit CronTab file (using the specified editor, or the default editor, tipically "vi", otherwise)
crontab -e

Format:
# Line of comments start with the dash character (#)
minute (0-59) hour (0-23) day of month (1-31) month (1-12) day of week (0-6, Sunday=0) COMMAND

# Use asterisks to specify any occurrence of the time parameter
* * * * * echo "test" # execute the command every minute (not recommended)
30 * * * 1 echo "test" # execute the command every 30 minutes, every monday

# separate multiple istances of the same time parameter with a comma (,)
15,30,45 * * * 1,3,5 echo "test" # execute the command every 15 minutes, every monday, wednesday, friday

# separate two istances with a dash (-) to create a time range
0 0,12 * 6-9 1-5 echo "test" # execute the command at midnight amd at twelve, every day from monday to friday, during every month from June to September

# Percent character (%) must be escaped (\)
30 * * * 1 touch /PATH/file_$(date +\%Y\%m\%d).txt

# Normally, crontab sends a mail to the user with the output of the command executed, including errors. Append ">/dev/null 2>&1" to the command line to prevent this behaviour
* * * * * echo "test">/dev/null

# list all scheduled jobs
crontab -l

# erase the crontab file (not recommended because remove all jobs, including ones that might be necessary for the system, you'd better edit the crontab files and delete entries manually)
crontab -r

-----

Networking:

arp -a # show DNS, IP, and MAC address for all network interfaces
ifconfig -a # show DNS, IP, and MAC address for all network interfaces
ifconfig eth0 # show DNS, IP, and MAC address for all network interfaces # show DNS, IP, and MAC address for the given interface (eth0)

Change MAC Address of an interface (eth0):
ifconfig eth0 down
ifconfig eth0 hw ether xx:xx:xx:xx:xx:xx
ifconfig eth0 up

ifconfig # show local network interfaces
ifconfig INTERFACE IP_ADDRESS # change the IP address for a network interface
ifconfig INTERFACE down # disable a network interface (WARNING: if you are connected from a remote system and disable the network interface to which you are connected you'll be disconnected, and if there isn't another interface available or a script that sets the interface up again automatically, you'll need physical access to the computer to configure it)
ifconfig INTERFACE up # enable a network interface

iwconfig # show local wireless network interfaces
iwconfig INTERFACE IP_ADDRESS # change the IP address for a wireless network interface
iwconfig INTERFACE down # disable a wireless network interface (WARNING: if you are connected from a remote system and disable the network interface to which you are connected you'll be disconnected, and if there isn't another interface available or a script that sets the interface up again automatically, you'll need physical access to the computer to configure it)
iwconfig INTERFACE up # enable a wireless network interface

Wake On LAN (WOL):
apt-get install ethtool
ethtool -s eth0 wol g # Set Wake On LAN on the specified interface (eth0). The system must support Wake On LAN and it has to be enabled from the BIOS
ethtool -s eth0 wol d # Unset Wake On LAN on the specified interface (eth0)
apt-get install etherwake
etherwake xx:xx:xx:xx:xx:xx # Wake On LAN the remote system with the specified MAC address

Set default Gateway:
route add default gw GATEWAY_IP # GATEWAY_IP in most cases is 192.168.0.254 or 192.168.1.254 

# Specify DNS Servers to resolve domain names
jed /etc/resolv.conf
# OpenDNS servers
nameserver 208.67.222.222
nameserver 208.67.220.200

jed /etc/hosts # assign hosts to specific IPs

jed /etc/network/options
jed /etc/network/interfaces

/etc/init.d/networking restart

Show active Internet connections:
netstat -atunlp
netstat -atun | awk "{print $5}" | cut -d: -f1 | sed -e "/^$/d" | sort | uniq -c | sort -n # show open connections sorted by IP address
netstat -plntu # show applications listening TCP/UDP sockets
netstat -A inet -lnp # only servers
netstat -A inet # without servers

Show network sockets in use:
lsof -i tcp:80 # show all processes that are using port 80 TCP

Show bandwidth usage:
apt-get install iftop
iftop # show connections and their bandwidth usage

Ping host:
ping HOST # ping a host until CTRL-C is pressed
ping -c N HOST # ping a host N times

Trace route to host:
traceroute HOST

Network Mapping:
apt-get install nmap

nmap -sS 192.0.2.0 # search for open ports in the given IP
nmap -sS 192.0.2.0/24 # search for connected IPs (and related open ports) in the given IP mask
nmap -sS -O -v 192.0.2.0 # return Operating System and uptime for the given IP

Port scanning:
apt-get install netcat

# scan TCP ports:
nc -vvn -z 192.0.2.2 1-80 # scan TCP ports from 1 upto 80 in the given IP address

# scan UDP ports:
nc -u -vvn -z 192.0.2.8 1-255 # scan UDP ports from 1 upto 255 in the given IP address

Calculate broadcast, network, Cisco wildcard mask, and host range
apt-get install ipcalc
ipcalc 192.0.2.0/24
ipcalc 192.0.2.0/255.255.255.0
ipcalc 192.0.2.0-192.0.2.100 # deaggregate address range

DNS lookup:
apt-get install host # If you want to use the host command

# Resolve Hostname to IP address
dig +short www.example.com | awk '{ print ; }'
getent hosts www.example.com | awk '{ print $1 ; }'
host www.example.com | awk '/has address/ { print $4 ; }'
nslookup www.example.com | awk '/^Address: / { print $2 ; }'

# Resolve Hostname to IP address (only get the first result if there are more then one IP)
dig +short www.example.com | awk '{ print ; exit }'
getent hosts www.example.com | awk '{ print $1 ; exit }'
host www.example.com | awk '/has address/ { print $4 ; exit }'
nslookup www.example.com | awk '/^Address: / { print $2 ; exit }'

# Resolve Hostname to IPv4 or IPv6 address
host www.example.com # return both IPv4 and IPv6 addresses
host -t A www.example.com # return the IPv4 addresses
host -t AAAA www.example.com # return the IPv6 addresses

# Reverse DNS lookup
host 192.0.2.0
nslookup 192.0.2.0

DNS queries:
dig www.example.com A # get the IP address of the given domain
dig www.example.com TXT # get text annotations about the given domain
dig www.example.com MX # get the mail servers of the given domain
dig www.example.com NS # get the authoritative DNS servers for the given domain
dig www.example.com ANY # get all DNS information for the given domain
whois example.com # Information about a domain name

# IP Tables:
iptables -I INPUT -s 192.0.2.100 -j DROP # Block a specific IP
iptables -I INPUT -s 192.0.2.0/255.255.255.0 -j DROP # Block a range of IPs using a netmask
iptables -I INPUT -s 192.0.2.0/24 -j DROP # Block a range of IPs using CIDR

iptables -S # Show rules
iptables -L # List rules
iptables -L INPUT -n --line-numbers # List only INPUT rules together with their line numbers

iptables -D INPUT LINE_NUMBER # Delete the INPUT rule with specified LINE_NUMBER

Script to clear IPTables:
---
#!/bin/bash

# Clear iptables
iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -t raw -F
iptables -t raw -X
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT
---

Download files from the Internet:
# note: default ports: HTTP: 80, FTP: 21
wget "http://www.example.com/file.gz"  # download the document at the given URL to the local directory
wget "http://www.example.com/file.gz" -O PATH/downloaded_file.gz  # download the document at the given URL to the specified directory and with the specified file name
wget -S "http://www.example.com/file.gz"  # download the document at the given URL in the local directory, and save the HTTP header at the beginning of the file
wget -c --wait=2 --limit-rate=20k "http://www.example.com/file.gz" # continue an interrupted download, and pause 2 secods after getting every file and limit download speed to 20KBps
wget -t 50 "http://www.example.com/file.gz"  # retries up to 50 times if the download fails or it's incomplete
wget -t 0 "http://www.example.com/file.gz"  # retries forever if the download fails or it's incomplete, until it can get the requested file
echo 'wget "http://www.example.com/file.gz"'|at 02:00  # start a download at 2:00am
wget -O- "http://www.example.com/path/page.html" # dump the output of the page on the console (normally on the screen)
wget -q -O- "http://www.example.com/path/page.html"|grep 'a href'  # show all links within given document
wget -r -l1 "http://www.example.com/path/page.html"  # download the first layer of links from given document
wget -r -l1 --no-parent "http://www.example.com/path/page.html"  # download the first layer of links from given document, ignoring links that are not within the given URL path
wget -r -l0 "http://www.example.com/path/page.html"  # download a whole website to the local directory
wget --mirror "http://www.example.com/"  # download a whole website to the local directory
wget -r -l1 --no-parent -A "*.gif" "http://www.example.com/dir/"  # download all files matching a certain pattern (all GIF files in this case) from the specified URL
wget -U USER_AGENT_STRING "http://www.example.com/file.gz" # you may set the User Agent to pretend to get the file from the server with a normal browser

Dump the content of a webpage on the screen:
lynx -dump http://www.example.com/

Run a script on the web server:
wget -O- "http://localhost/script.php" # run a script on the local server
wget -q -O- "http://localhost/script.php" # run a script on the local server hiding the output
wget -q -T 0 -O- "http://localhost/script.php" # By default wget drops connection after 900 seconds (15 minutes), in this case you'll get a "Read error (Connection timed out) in headers.  Retrying." and if the script always takes more than the given time, it will retry uselessly until the maximum allowed retries (by default 20, but it can be changed with -t). To avoid time outs you can set a longer time using -T followed by the number of seconds (or 0 for infinite time, in this case make sure the script is working or it will run forever and if it uses resources incrementally it may crash the server). Also, if you experienced a time out with wget while running a script on localhost you should better restart the webserver as the script may continue running despite of wget timing out and dropping the connection.
lynx -dump http://localhost/script.php >/dev/null # run a script on the local server (less reliable than wget for this purpose) hiding the output
php PATH/FILE.php # run a PHP script
php -r "echo 'Hello World';"; # Execute a PHP line in the shell
php -r 'echo "Hello World\n";'; # Execute a PHP line in the shell. Use quotes to print character sequences.
perl -e "print 'Hello World';"; # Execute a Perl line in the shell
python -c "print 'Hello World'"; # Execute a Python line in the shell
ruby -e "puts 'Hello World'"; # Execute a Ruby line in the shell


# If you receive this error message from the PPP Daemon (PPPD)
# pppd: The remote system is required to authenticate itself but I couldn't find any secret (password) which would let it use an IP address.
# You can fix it adding the string "noauth" (or changing the existing string "auth" to "noauth") in /etc/ppp/options

-----

Mirror a disk through the network:

If you have two computers with identical hardware you can mirror a system into the other machine through the network
This is especially useful with clusters / virtual servers

Start both machines from a live CD (like Knoppix - http://www.knoppix.org/ )
Connect both machine with a cross cable through their eth1 interfaces and set them this way:
root@source_pc# ifconfig eth1 192.168.0.1
root@destin_pc# ifconfig eth1 192.168.0.2

Then execute these commands on the machines to start transfers:
root@source_pc# dd if=/dev/sda bs=128M | nc _w2 192.168.0.2 9000
root@destin_pc# nc _l _p 9000 | dd of=/dev/sda

Depending from the size of the disk it may need many hours

-----

Maths:

Return the result of an expression:
# operators must be escaped and separated by a space
# operators: + - * / %  < > = >= <= !=
expr \( 1 \+ 2 \) \* 3 # Result = 20
expr 5 \> 3 # Returns 1 (true)
expr 3 \> 5 # Returns 0 (false)

Show Multiplication Table:
for i in {1..9}; do for j in $(seq 1 $i); do echo -ne $i\x$j=$((i*j))\\t; done; echo; done;

Return the prime factors of a number:
factor NUMBER
factor NUMBER1 NUMBER2 NUMBER3... # return factors for the given numbers

Return a sequence of numbers:
echo {FIRST..LAST} # return a sequence of integer numbers from FIRST to LAST, numbers can be negative, and LAST can be less than FIRST
for ((i=FIRST; i<=LAST; i+=INCREMENT)); do echo $i; done # create a sequence from FIRST up to LAST number with given INCREMENT
seq LAST # create a sequence up to LAST number (default FIRST is 1, INCREMENT is 1)
seq FIRST LAST # create a sequence from FIRST up to LAST number (default INCREMENT is 1)
seq FIRST INCREMENT LAST # create a sequence from FIRST up to LAST number with given INCREMENT. Numbers can be negative, and END can be less than START, if the INCREMENT is negative
seq -s "," FIRST INCREMENT LAST # create a sequence from FIRST up to LAST number with given INCREMENT. Numbers are separated by a comma (",") instead of a line break
seq -s "," -w FIRST INCREMENT LAST # create a sequence from FIRST up to LAST number with given INCREMENT. Numbers are separated by a comma (",") instead of a line break. Numbers with less digit than the greater number are padded with 0s (or the sign for negative numbers)
seq FIRST INCREMENT LAST | shuf # shuffle a sequence of numbers

Random numbers:

echo $RANDOM  # return a random number between 0 and 32767 (16-bit integer)
echo $((RANDOM%(MAX+1-MIN)+MIN)) # return a random number between MIN and MAX
expr \( $RANDOM \% \( MAX \+ 1 \- MIN \) \+ MIN \) # return a random number between MIN and MAX


Base conversion:

apt-get install bc

echo 'ibase=10; obase=16; 155'|bc  # base conversion (155 from decimal to hexadecimal)
echo 'ibase=16; obase=2; F'|bc  # base conversion (F from hexadecimal to binary)
echo '2*(3+4)'|bc -l  # perform math operations


Measure conversion:

apt-get install units

units  # interactive
units "1 in" "cm"  # convert 1 inch to centimeters

-----

Time:

Set date and time:
date MMDDhhmmYYYY.ss

Set date from NTP (SNTP) server:
apt-get install ntpdate
ntpdate SERVER  # set the clock
ntpdate -q SERVER  # query only for the time, don't set the clock

SNTP servers:
ntp.ubuntu.com # Ubuntu
ntp1.inrim.it # INRiM, Torino, Italy
ntp.univ-lyon1.fr # University of Lyon, Lyon, France
vega.cbk.poznan.pl # Astrogeodynamic Observatory, Borowiec, Poland
bitsy.mit.edu # MIT, Cambridge, Massachusetts, USA
bonehed.lcs.mit.edu # MIT, Cambridge, Massachusetts, USA
clock.isc.org # Internet Systems Consortium, Palo Alto, California, USA

Start a NTP server:
apt-get install ntp-refclock (or another ntp* package)
jed /etc/ntp.conf
/etc/init.d/ntp-server restart

Date tokens:
%% : %
%a : Weekday's abbreviated name (for example, Mon)
%A : Weekday's full name (for example, Monday)
%b : Month's abbreviated name (for example, Jan; same as %h)
%B : Month's full name (for example, January)
%C : Century (for example, 21)
%d : Day of month (for example, 01)
%e : day of month, space padded; same as %_d
%F : Full date formatted as full year - month - day (same as %Y-%m-%d )
%g : last two digits of year of ISO week number (see %G)
%G : year of ISO week number (see %V); normally useful only with %V
%H : Hour in 24 hours format with padded 0 (00-23)
%I : Hour in 12 hours format with padded 0 (01-12)
%j : Day of year with padded 0 (001-366)
%k : Hour in 24 hours format with padded space ( 0-23)
%l : Hour in 12 hours format with padded space ( 1-12)
%m : Month with padded 0 (01-12)
%M : Minute with padded 0 (00-59)
%n : Newline
%N : Nanoseconds (000000000-999999999)
%p : AM or PM, uppercase
%P : AM or PM, lowercase
%R : 24-hour hour and minute (same as %H:%M )
%s : Seconds since Linux Epoch (1970-01-01 00:00:00 UTC)
%S : Second with padded 0 (00-60)
%t : Tab
%T : Time (same as %H:%M:%S )
%u : ISO Day of week (1-7: 1 is Monday, 7 is Sunday)
%U : Week number of year, with Sunday as first day of week and padded 0 (00-53)
%V : ISO Week number of the year, with Monday as first day of week and padded 0 (01-53)
%w : Day of week (0-6: 0 is Sunday, 6 is Saturday)
%W : ISO Week number of year, with Monday as first day of week and padded 0 (00-53)
%y : Last two digits of year (00-99)
%Y : Full Year
%z : Timezone as hhmm (e.g., -0200)
%:z : Timezone as hh:mm (e.g., -02:00)
%Z : Timezone's abbreviated name (e.g., CET)

Show current date and time:
date
date '+%a %d %b %Y %H:%M:%S %Z (UTC%:z)' # show day of the week, day of the month, month (short format), year (full digits), hour, minutes, seconds, timezone
echo "`date '+%a %d %b %Y %H:%M:%S %Z (UTC%:z)'`"; # echo day of the week, day of the month, month (short format), year (full digits), hour, minutes, seconds, timezone

Show information about your local time:
timedatectl status

Show System Time Zone
cat /etc/timezone

Linux Epoch (1 January 1970 00:00:00)
date +"%s" # Timestamp: seconds from Epoch to current time
date -d "2000-01-01" +"%s" # Seconds from Epoch to the specified date (from 1 January 1970 to 1 January 2000 in this example)
date -d @1474303375 # Converts seconds from Epoch to date

Dates:
date +"%F" -d now # Return current day in the specified format
date +"%F" -d yesterday # Return yesterday's day in the specified format
date +"%F" -d tomorrow # Return tomorrow's day in the specified format
date +"%F" -d last-week # Return last week's date on current day in the specified format
date +"%F" -d next-week # Return next week's date on current day in the specified format
date +"%F" -d last-month # Return last month's date on current day in the specified format
date +"%F" -d next-month # Return next month's date on current day in the specified format
date +"%F" -d last-year # Return last year's date on current day in the specified format
date +"%F" -d next-year # Return next year's date on current day in the specified format
date +"%F" -d last-WEEKDAY # Return the date of the last given weekday (for example last-monday) in the specified format
date +"%F" -d next-WEEKDAY # Return the date of the next given weekday (for example next-monday) in the specified format

Calculate dates:
date +"%F" -d "now +15 days" # Show the resulting date adding the specified time to the current date in the specified format (date only in this example)
date +"%F" -d "2000-01-01 +15 days" # Show the resulting date adding the specified time to the given date in the specified format (date only in this example)
date +"%F %H:%M:%S" -d "now +10 seconds" # Show the resulting date adding the specified time to the current date in the specified format (date and time in this example)
date +"%F %H:%M:%S" -d "2000-01-01 10:10:00 UTC +30 seconds" # Show the resulting date adding the specified time to the given date in the specified format (date and time in this example)

Get the day of the week for a specified date:
date -d "2000-01-01" +"%A"

Calculate how long takes a command to be executed:
time COMMAND COMMAND_PARAMETERS
time cat # works like a stopwatch: counts time until you send an EOF to stop it (which generally means until you press CTRL+D that closes the cat command)

Create filenames containing current date and time:
touch data_`date +%F`.bak # create a file with the name data_YYYY-MM-DD.bak
touch log_`date +%F_%H-%M-%S`.txt # create a file with the name log_YYYY-MM-DD_HH-MM-SS.txt

User and system times:
times

Show a calendar:
# cal  # show dayweeks in a row, Sunday first (use -m to have Monday first)
# ncal # show dayweeks in a coloumn, Monday first
cal # show a calendar for the current month
cal -y # show a calendar for the current year
cal -y YEAR # show a calendar for the given year
cal MONTH YEAR # show a calendar for the given MONTH and YEAR
ncal -e YEAR # show date of Easter (for Western Churches)
ncal -o YEAR # show date of Orthodox Easter (Greek and Russian Orthodox Churches)

Pause:
sleep S # pause for S seconds

Memorable dates:
The directory /usr/share/calendar/ contains some collections of remarkable dates.
For example:
cat /usr/share/calendar/calendar.history # Important historical dates
cat /usr/share/calendar/calendar.music # Important dates in music history
cat /usr/share/calendar/calendar.lotr # Lord of the Rings dates

-----

Archiving and Compression:

Install archivers:
apt-get install bzip2
apt-get install zip unzip
apt-get install zoo
apt-get install lha
apt-get install rar
apt-get install unp

Archiving with ISO:
dd if=/dev/dvd of=dvd.iso # Create an ISO file from a DVD
dd if=/dev/cdrom of=cd.iso # Create an ISO file from a CD-ROM
dd if=/dev/scd0 of=cd.iso # Create an ISO file from a SCSI CD-ROM
mkisofs -o /PATH_DEST/cd.iso /PATH_SOURCE/ # Create an ISO file from a directory tree

Archiving with TAR:
tar -cvf file.tar text.txt # archive text.txt as file.tar
tar -cvf file.tar *.txt # archive all files ending with ".txt" in the current directory as file.tar
tar -xvf file.tar # extract all files from file.tar in the current directory

Compressed TAR archives (tarball):
# Use tar to compress a directory tree into a single file, and gzip or bzip2 to compress each file into a directory tree
tar -cjvf file.tbz text.txt # TAR+BZIP2 compression (canonical extension .tbz or .tar.bz2)
tar -xjvf file.tbz # extract all files from a TAR+BZIP2 compressed archive
tar -czvf file.tgz text.txt # TAR+GZIP compression (canonical extension .tgz or .tar.gz)
tar -xzvf file.tgz # extract all files from a TAR+GZIP compressed archive

Compression with GZIP:
gzip -9vk text.txt # compress text.txt as text.txt.gz using highest compression and verbose output, keeping the original uncompressed file
gzip -9v text.txt > file.gz # compress text.txt as file.gz using highest compression and verbose output, keeping the original uncompressed file
gzip -9v text.txt # compress text.txt as text.txt.gz using highest compression and verbose output, deleting the original uncompressed file 
gzip -r9v file.txt file2 file3 /home/docs # compress file.txt file2 file3 and every file in /home/docs (if such directory exists), deleting the original uncompressed files
gzip -l file.gz # show the content of the compressed file
gzip -dk file.gz # extract the content of file.gz in the current directory and keep the compressed file (file.gz)
gzip -d file.gz # extract the content of file.gz in the current directory and delete the compressed file (file.gz)

Compression with BZIP2:
bzip2 text.txt # compress text.txt as text.txt.bz2
bunzip2 file.bz2 # extract the content of file.bz2 into the current directory

Compression with ZIP:
zip file.zip file.txt # add file.txt to the compressed file file.zip (if it doesn't exist, it will created)
zip -r file.zip /home/docs # compress the whole content of the directory /home/docs into file.zip
zip -r filename.zip file.txt file2 file3 /home/docs # compress file.txt file2 file3 and the content of /home/docs (if such directory exists) into file.zip
zip -F brokenfile.zip # try to fix a broken zipped file (use -FF to try even harder)
unzip file.zip # extract the content of file.zip into the current directory
unzip file.zip -d PATH_DEST # extract the content of file.zip into the specified directory

Encrypted Compression with ZIP:
# Note that Zip encryption is not very strong
zip -e file.zip file.txt # encrypt file.txt into file.zip prompting for a password
zip -P PASSWORD FILE.zip file.txt # encrypt file.txt into file.zip with the given PASSWORD (beware of shoulder surfing)
unzip file.zip # extract the content of file.zip and prompt for password if encrypted
unzip -P PASSWORD file.zip # unencrypt and extract the content of file.zip with given PASSWORD (beware of shoulder surfing)

Compression with LHa (LHarc) / LZH:
lha a file.lha text.txt # compress text.txt as file.lha (or add text.txt to an already existing file.lha compressed archive)
lha e file.lha # extract the content of file.lha into the current directory (will prompt for existing files)

Compression with ZOO:
zoo a file.zoo text.txt # compress text.txt as file.zoo (or add text.txt to an already existing file.zoo compressed archive)
zoo e file.zoo # extract the content of file.zoo into the current directory (will ignore existing files)

Compression with RAR (RAR is shareware):
rar a file.rar text.txt # compress text.txt as file.rar (or add text.txt to an already existing file.rar compressed archive)
rar e file.rar # extract the content of file.rar into the current directory (will prompt for existing files)

Unpack most common formats:
unp FILE # Supported extensions and formats: tar[.gz,.bz2], gz, bz2, Z, ar/deb, rpm, shar, rar, arj, zip, LHa, cab, ace, tnef, uu (mail, news), mime, hqx, sea, zoo, pmd, cpio, afio, lzop

-----

Encoding:

Install encoders/decoders:
apt-get install mpack
apt-get install uudeview
apt-get install yencode

MIME/Base64:
mpack -s SUBJECT -o PATH/DEST_FILE PATH/SOURCE_FILE # encode (B64encode) within a MIME message with header
munpack PATH/FILE # decode (B64decode) to the original file name
uudeview -i PATH/FILE # decode (B64decode) to the original file name

Base64:
uuenview -b PATH/SOURCE_FILE>PATH/DEST_FILE # encode (B64encode). will not be decoded without MIME header.

UU (Unix-to-Unix):
uuenview -u PATH/SOURCE_FILE>PATH/DEST_FILE # encode (uuencode)
uudeview -i PATH/FILE # decode (uudecode) to the original file name

XX:
uuenview -x PATH/SOURCE_FILE>PATH/DEST_FILE # encode (xxencode)
uudeview -i PATH/FILE # decode (xxdecode) to the original file name

Yenc:
uuenview -y PATH/SOURCE_FILE>PATH/DEST_FILE # encode (yencode)
uudeview -i PATH/FILE # decode (ydecode) to the original file name # uudeview handy options: +o don't overwrite existing files, -c autoclear (delete) successfully decoded files, -d desperate mode (process incomplete files)
yencode PATH/SOURCE_FILE # encode (yencode) SOURCE_FILE as SOURCE_FILE.ync
ydecode PATH/FILE # decode (ydecode) to the original file name
ydecode -f PATH/FILE # decode (ydecode) to the original file name (force overwriting already existing files, without prompting)

-----

Hashing:

md5sum PATH/FILE # Return the MD5 hash for the content of the given FILE
md5sum PATH/FILE* # Return the MD5 hash for the content of the files matching the given mask
md5sum PATH/FILE1 PATH/FILE2 PATH/FILE3 # Return the MD5 hash for the given FILEs
echo -n "STRING" | md5sum | sed "s/  -//g" # Return the MD5 hash of the given STRING
VARIABLE=`echo -n "STRING" | md5sum | sed "s/  -//g"`; # Store the MD5 hash of the given STRING into a VARIABLE
VARIABLE=`echo -n "STRING" | md5sum | sed "s/  -//g"`; echo "[$VARIABLE]"; # Store the MD5 hash of the given STRING into a VARIABLE and output it between square brackets

-----

Obfuscation:


Text scrambling (anagrams):

apt-get install wordplay

wordplay "text string" # list all the possible anagrams of the given text string


Alphabet scrambling:

apt-get install rotix

rotix # accept a text from standard input and return its ROT-13 to the standard output
rotix -f PATH/SOURCE_FILE -o PATH/DEST_FILE # read SOURCE_FILE and return its ROT-13 to DEST_FILE (note: a ROT-13 can be deobfuscated performing the same operation again)
rotix -r 15 -L -f PATH/SOURCE_FILE -o PATH/DEST_FILE # read SOURCE_FILE and return its ROT-15 to the Left to DEST_FILE (note: this can be deobfuscated performing a ROT-15 to the Right)
rotix -r 12 -R -f PATH/SOURCE_FILE -o PATH/DEST_FILE # read SOURCE_FILE and return its ROT-12 to the Right to DEST_FILE (note: this can be deobfuscated performing a ROT-12 to the Left)


Steganography:

apt-get install steghide

steghide --embed -p PASSWORD -ef SOURCE_FILE_TO_HIDE -cf DEST_OBFUSCATION_FILE # hide a file into another (AES encryption and compression are applied by default)
steghide --embed -p PASSWORD -ef SOURCE_FILE_TO_HIDE -cf OBFUSCATION_FILE -sf DEST_FILE # hide a file into another (AES encryption and compression are applied by default), but save the result into a new file
steghide --info STEGOFILE -p PASSWORD # return information about the file obfuscated into STEGOFILE
steghide --extract -p PASSWORD -sf STEGOFILE -xf DESTFILE # extract the file obfuscated into STEGOFILE as DESTFILE
# steghide applies AES encryption and compression by default. It can hide information into JPEG, BMP, WAV, and AU files.

apt-get install stegdetect

stegdetect FILE.jpg # try to find steganographed information into a JPEG file

-----

Encryption with GnuPG (GPG) and Pretty Good Privacy (PGP) wrapper:

apt-get install pgpgpg

# PGP

pgp -h # Help
pgp -k # Key Manager functions help

pgp -kg # Generate key
pgp -kxa USER_ID PATH/DEST_FILE.ASC # Export User's public key to file
pgp -ka PATH/PUBLIC.KEY # Add public key to keyring

pgp -ea PATH/SOURCE_CLEARTEXT RECIPIENT_ID # Encrypt file for Recipient
pgp PATH/ENCRYPTED_FILE # Decrypt file (need password)

# Handy options:
# (Usage example: PGP -e = binary object file, PGP -ea = ASCII object file)
# -a   ASCII armour
# -w   Wipe (overwrite&delete) original file  (when crypting) 
# -m   No output file (on screen display, "eyes only")  (when decrypting)

# GPG

gpg -h # Help
gpg --version # Show GPG version and supported algorithms

gpg --gen-key # generate key
gpg --list-keys # list public keys in the public keyring
gpg --list-secret-keys # list secret (private) keys in the secret keyring
gpg --export -a > PATH/DEST_FILE.ASC # Export all public keys to a file
gpg --export -a "USER_ID" > PATH/DEST_FILE.ASC # Export User's public key to file
gpg --import PATH/PUBLIC.KEY # Add public key to keyring

gpg -s -r "USER_ID" PATH/SOURCE_CLEARTEXT # Sign file for Recipient, file name is automatically generated
gpg -s -a -r "USER_ID" PATH/SOURCE_CLEARTEXT # Create a signature for the given file for Recipient and output with an ASCII armour, file name is automatically generated
gpg -se -r "USER_ID" PATH/SOURCE_CLEARTEXT # Sign and encrypt file for Recipient, file name is automatically generated
gpg -se -a -r "USER_ID" PATH/SOURCE_CLEARTEXT # Sign and encrypt file for Recipient and output with an ASCII armour
gpg --verify PATH/SIGNATURE.SIG # Verify a signature (without showing the signed content)

gpg PATH/ENCRYPTED_FILE # Decrypt file (need password), file name is automatically generated
gpg -d PATH/ENCRYPTED_FILE > PATH/DEST_FILE # Decrypt file specifying the output file name (need password)
gpg -d PATH/ENCRYPTED_FILE # Decrypt file, on screen display, "eyes only" (need password)

-----

Passwords:

Password generation:

tr -dc 'a-z0-9' < /dev/urandom | head -c 10 # generate a random password of 10 characters with only lower case letters and numbers

apt-get install apg

apg -n 1 -M NL -a 1 -m 8 -x 12 # Generate a random password from 8 to 12 characters long made with only lower case letters and numbers
apg -n 1 -M NL -a 0 -m 10 -x 15 # Generate a random but human readable password from 10 to 15 characters long made with only lower case letters and numbers

apt-get install pwgen

pwgen -1 -A -n -B # Generate a password of 8 characters made with only lower case letters (-A) and at least one number (-n, default behavior) without ambiguous characters (-B)

-----

Check for brute force attack:

jed /var/log/auth.log # See attempted and successful log ins (which include possible break-in attempts)

-----
Audio:

Play a sound:
printf '\7' # Ring the Bell

Extract audio tracks from a CD:

apt-get install cdda2wav

cdda2wav -B # extract tracks as WAVs from the default CD player
cdda2wav -D  /dev/cdrom -B # extract tracks as WAVs from the specified CD player


Create an Audio CD writing WAVs as audio tracks:

apt-get install cdrecord

cdrecord dev=0,0,0 speed=4 -eject -pad -audio -raw -swab *.wav # write all WAVs from the current directory


Create an Audio CD writing MP3s/OGGs/flacs as audio tracks:

apt-get install mp3burn

mp3burn *.mp3 # create an audio CD from all MP3s in the current directory
mp3burn -p "PLAYLIST.m3u" # create an audio CD fetching files listed in a M3U playlist


Play MP3 files:

apt-get install mpg321

mpg321 FILE.mp3


Edit ID3v1 tags:

apt-get install id3ed

id3ed -i FILE.mp3 # return id3v1 tag
id3ed -r FILE.mp3 # remove id3v1 tag
id3ed -s SONGNAME -n ARTIST -a ALBUM -y YEAR -c COMMENT -k TRACKNUM -g GENRE_NUMBER FILE.mp3 # set id3v1 for FILE.mp3
id3ed -s SONGNAME -n ARTIST -a ALBUM -y YEAR -c COMMENT -k TRACKNUM -g GENRE_NUMBER *.mp3 # set id3v1 for all mp3 files in the current directory


Edit ID3v1 and ID3v2 tags:

apt-get install id3v2

id3v2 -l FILE.mp3 # return id3 tags
id3v2 -1 -l FILE.mp3 # return id3v1 tag
id3v2 -2 -l FILE.mp3 # return id3v2 tag
id3v2 --delete-v1 FILE.mp3 # remove id3v1 tag
id3v2 --delete-v2 FILE.mp3 # remove id3v2 tag
id3v2 --delete-all FILE.mp3 # remove all tags (id3 v1 and v2)
id3v2 -C FILE.mp3 # convert id3v1 to id3v2 tag
id3v2 -a "ARTIST" -A "ALBUM" -t "SONG" -c "COMMENT" -g GENRE_NUMBER -y YEAR -T TRACK FILE.mp3 # set id3 for FILE.mp3
id3v2 -a "ARTIST" -A "ALBUM" -t "SONG" -c "COMMENT" -g GENRE_NUMBER -y YEAR -T TRACK *.mp3 # set id3 for all mp3 files in the current directory
id3v2 -1 -a "ARTIST" -A "ALBUM" -t "SONG" -c "COMMENT" -g GENRE_NUMBER -y YEAR -T TRACK FILE.mp3 # set id3v1 for FILE.mp3
id3v2 -2 -a "ARTIST" -A "ALBUM" -t "SONG" -c "COMMENT" -g GENRE_NUMBER -y YEAR -T TRACK FILE.mp3 # set id3v2 for FILE.mp3


Check MP3 files:

apt-get install mp3check

mp3check FILE.mp3 # check crc and headers of mp3 files for consistency
mp3check -a FILE.mp3 # check mp3 file for differences from layer 3, 44.1kHz, 128kB, joint stereo, no emphasis, has crc


Lossless MP3 normalization (peak and loudness normalization):

apt-get install mp3gain

mp3gain FILE1.mp3 FILE2.mp3 FILE3.mp3 # normalize specified mp3 files (warning: overwrites source files)
mp3gain -r FILE1.mp3 FILE2.mp3 FILE3.mp3 # normalize specified mp3 files all to equal loudness (warning: overwrites source files)
mp3gain -a FILE1.mp3 FILE2.mp3 FILE3.mp3 # normalize specified mp3 files to the average loudness, intended for all audio files from a same album (warning: overwrites source files)
mp3gain -u FILE.mp3 # undo changes made by mp3gain (based on info stored on tag)


Convert WAV to MP3:

apt-get install lame

lame FILE.wav FILE.mp3


Convert MP3 to WAV:

apt-get install mpg321

mpg321 -w FILE.wav FILE.mp3 # convert FILE.mp3 to FILE.wav


Convert MP3 to OGG:

apt-get install mp32ogg

mp32ogg FILE.mp3 # convert FILE.mp3 to FILE.ogg
mp32ogg --rename=DEST.ogg SOURCE.mp3 # convert SOURCE.mp3 to DEST.ogg
mp32ogg --rename=%a_%l_%t.ogg *.mp3 # convert all mp3 files in the current directory renaming generated OGG files as ARTIST_ALBUM_TRACKTITLE.ogg (make sure that id3 tag contains all such information, otherwise files will be called as SOURCEFILE.ogg)


Convert an audio format into another audio format

apt-get install sox

sox SOURCE_FILE.SOURCE_EXT DEST_FILE.DEST_EXT # convert SOURCE_FILE.SOURCE_EXT to DEST_FILE.DEST_EXT. File format conversion is based on standard extensions
sox FILE.ogg FILE.mp3 # convert FILE.ogg into FILE.mp3 (note that MP3 support is optional. check sox --help for supported formats)

Reverse audio:
sox -V SOURCE_FILE.wav DEST_FILE.wav reverse

-----

Manipulate video files

apt-get install ffmpeg

Convert a video format into another video format:

ffmpeg -i PATH/SOURCE_VIDEO.avi PATH/DEST_VIDEO.mpg # convert from AVI to MPEG

ffmpeg -i PATH/SOURCE_VIDEO.avi -sameq FILE.avi PATH/DEST_VIDEO.mpg # convert from AVI to MPEG, keeping same quality of the source file

ffmpeg -i PATH/SOURCE_VIDEO.avi -ss 00:00:00 -t 00:45:00 PATH/DEST_VIDEO.mpg # convert from AVI to MPEG, taking only the first 45 minutes of the source video (from 00:00:00 to 00:45:00)

ffmpeg -i PATH/SOURCE_VIDEO.avi -i PATH/SOURCE_AUDIO.mp3 PATH/DEST_FILE.avi # create a video with audio taking video and audio from two different sources

ffmpeg -loop_input -i PATH/SOURCE_IMAGE.jpg -i PATH/SOURCE_AUDIO.mp3 -shortest -acodec copy PATH/DEST_FILE.mp4 # create a video with a static image and audio

Dump all frames from a video:
ffmpeg -i PATH/SOURCE_VIDEO -an -r 1/1 PATH/FILE_%06d.jpg # Dump all frames as JPG, you can dump them in .bmp or .png by changing the extension of the destination file. In this example the generated file name will have 6 digits

Speed up or slow down a video:
ffmpeg -i PATH/SOURCE_VIDEO -filter:v "setpts=SPEED*PTS" PATH/DEST_VIDEO # SPEED must be set to 1/(wanted speed x), for example set SPEED: 0.5 : double speed (2x), 0.25 : 4x speed, 2.0 = half speed (1/2x). If you want the destination video to have a specific duration, you can calculate the value of SPEED with this formula: 1 / (source video duration in seconds / destination video wanted duration in seconds)

Extract the audio track from a video:
ffmpeg -i PATH/SOURCE_VIDEO.mpg -vn -f mp3 AUDIO_TRACK.mp3 # extract the audio track as MP3 from a video
ffmpeg -i PATH/SOURCE_VIDEO.mpg -vn -ac 1 -f mp3 AUDIO_TRACK.mp3 # extract the audio track as MP3 from a video as mono. Use -ac 2 to force stereo even if the input file is mono.

-----

Play a video:

apt-get install vlc

vlc MOVIE.AVI # play a video (even in the text console)
vlc http://www.example.com:8081/file.mpg # play a video from the web

Streaming video server:

vlc -vvv file.mpg --sout '#standard{access=http,mux=mpeg1,url=www.example.com:8081/file.mpg}' --ttl 12 --loop # Stream a MPEG file (loop) from the url http://www.example.com:8081/file.mpg

# Note that you must have a streaming format like ASF to create a real stream of data
vlc -vvv file.asf --sout '#standard{access=mmsh,mux=asf,url=www.example.com:8082}' --ttl 12 --loop # Stream an ASF file (loop) from the url mms://www.example.com:8082/file.asf

-----

Manipulate and convert images:

apt-get install imagemagick

convert SOURCE.jpg DEST.png # Convert a JPEG image into PNG 
convert -resize 50% SOURCE.jpg DEST.jpg # resize SOURCE.jpg at 50% of the original size, and save it as DEST.jpg
convert -resize 600x300 SOURCE.jpg DEST.jpg # resize SOURCE.jpg to 600x300 (width x height) pixels, and save it as DEST.jpg
convert -resize 600 SOURCE.jpg DEST.jpg # resize SOURCE.jpg to a width of 600 pixels and relative height, and save it as DEST.jpg
convert -resize x300 SOURCE.jpg DEST.jpg # resize SOURCE.jpg to a height of 300 pixels and relative width, and save it as DEST.jpg
convert -thumbnail 150 SOURCE.jpg DEST.jpg # create a thumbnail of SOURCE.jpg with a width of 150 pixels and relative height, and save it as DEST.jpg # the difference between -resize and -thumbnail is that -thumbnail strips all meta data (like EXIF information) from the original image
convert -size 1x1 xc:none IMAGE.png # create a transparent .png image sized 1 x 1 pixels
convert -thumbnail x100 PATH/FILE.pdf[0] THUMBNAIL.png # create a thumbnail with a height of 100 pixels for the first page of a PDF document
convert -thumbnail x100 PATH/FILE.pdf THUMBNAIL.png # create a thumbnail with a height of 100 pixels for every page of a PDF document

-----

EXIF information:

apt-get install exiftool

exiftool FILE.jpg # show EXIF tags inside a JPEG image
exiftool -geotag= FILE.jpg # remove geographical coordinates from the given image
exiftool -all= FILE.jpg # remove all EXIF information from the given image
for i in PATH/*.jpg; do echo "Processing: $i"; exiftool -all= "$i"; done # remove all EXIF information from all the JPEG images in the given PATH


apt-get install exif

exif FILE.jpg # show EXIF tags inside a JPEG image
exif -im FILE.jpg # return EXIF tags inside a JPEG image in a machine readable format (-i ID instead of tags, -m tabs delimited output)
exif --ifd=0 -t TAG --set-value=VALUE SOURCE.jpg -o=DEST.jpg # set TAG as VALUE in SOURCE.jpg and save the resulting manipulated image as DEST.jpg
exif --remove --ifd=0 SOURCE.jpg -o=DEST.jpg # remove all EXIF information from SOURCE.jpg and save the resulting cleaned image as DEST.jpg
exif -r SOURCE.jpg -o=DEST.jpg # remove the thumbnail from SOURCE.jpg and save the resulting cleaned image as DEST.jpg
exif -e SOURCE.jpg -o=THUMBNAIL.jpg # extract the thumbnail from SOURCE.jpg and save it as THUMBNAIL.jpg 
exif FILE.jpg -n=THUMBNAIL.jpg # add THUMBNAIL.jpg to FILE.jpg and save the resulting manipulated image as FILE.jpg.modified.jpeg

-----

GPS data:

apt-get install gpsbabel

gpsbabel -i gdb -f PATH/SOURCE_FILE.gdb -o gpx -F PATH/DEST_FILE.gpx # convert a Garmin .gdb file into .gpx (GPS eXchange Format, XML)

-----

Disk quotas:

apt-get install quota quotatool

quota # display disk usage and limits
quotacheck # scan a file system for disk usage
quotactl # set disk quotas

-----

Quitting a session, turning off (halting) or rebooting system:

# Note that when a process receives a signal may behave differently from the default behavior
CTRL+C # terminate an application (sends a SIGINT signal to the process)
CTRL+\ # terminate the process and dump to a core file (sends a SIGQUIT signal to the process)
CTRL+D # send an EOT (End Of Tranmission) signal to the process
CTRL+Z # suspend an application (sends a SIGSUSP signal to the process)
exit # exit from a non login shell
logout # logout from current session
shutdown -h now # halt the system
shutdown -h -P now # halt the system and turn off the machine (power off)
shutdown -r now # reboot the system
shutdown -r 22:00 # reboot the system at 22:00 (10:00 pm)
shutdown -r +5 # reboot the system after 5 minutes
reboot # reboot the system (like shutdown -r), might not work on older systems
reboot -f # force reboot without using shutdown

Show system reboot history:
last reboot
last -x|grep shutdown
for wtmp in `ls -t /var/log/wtmp*`; do last reboot -f $wtmp; done|grep reboot
last reboot|head -1 # Show only last reboot

Show uptime:
uptime # a sysadmin will be judged by his uptime and by how many reboots he performed

# Edit system startup script
# Commands that will be executed when the system boots. Different runlevels are allowed.
jed /etc/inittab

Choose what command must be performed when CTRl+ALT+DEL is pressed:
jed /etc/inittab
# What to do when CTRL-ALT-DEL is pressed.
ca:12345:ctrlaltdel:/sbin/shutdown -t1 -a -r now

-----

Install a LAMP webserver:

Install servers and related software:
apt-get install apache2
apt-get install lynx
apt-get install openssl
apt-get install php4 libapache2-mod-php4 php4-pear php4-gd php4-mcrypt # PHP 4
apt-get install php4-mysql # MySQL support for PHP 4
apt-get install php5 libapache2-mod-php5 php-pear php5-gd php5-mcrypt php5-sqlite # PHP 5
apt-get install php5-mysql # MySQL support for PHP 5
apt-get install php7.0 libapache2-mod-php7.0 php-pear php7.0-gd php7.0-mcrypt php7.0-sqlite3 # PHP 7
apt-get install php7.0-mysql # MySQL support for PHP 7
apt-get install mysql-server
apt-get install phpmyadmin


Apache 2 configuration:

/usr/sbin/apache2 -v  # return Apache 2 version
openssl version -v # Get OpenSSL version

apache2 -l # list compiled in Apache 2 modules
apache2ctl -M # list loaded Apache 2 modules

a2enmod # Add a module choosing it from a list. You'll have to restart Apache 2 then.

jed /etc/apache2/apache2.conf

If your webserver is too slow, you may have to adjust the value for MaxClients: if it's too low it will allow too few simultaneous accesses, if it's too high it will require too much resources to handle them. Try with MaxClients 50 or MaxClients 100 .
<IfModule prefork.c>
MaxClients 100
</IfModule>

You can also add the expires module and set expiry periods for the cache of file types you don't change often on the server (like images):
<IfModule expires_module>
ExpiresActive On
ExpiresDefault "access plus 2 days"
ExpiresByType text/php "access plus 1 second"
ExpiresByType text/x-php "access plus 1 second"
ExpiresByType application/php "access plus 1 second"
ExpiresByType application/x-php "access plus 1 second"
ExpiresByType application/x-httpd-php "access plus 1 second"
ExpiresByType text/html "access plus 2 days"
ExpiresByType text/plain "access plus 2 days"
ExpiresByType image/jpg "access plus 1 month"
ExpiresByType image/jpeg "access plus 1 month"
ExpiresByType image/gif "access plus 1 month"
ExpiresByType image/png "access plus 1 month"
ExpiresByType text/css "access plus 1 month"
ExpiresByType application/pdf "access plus 1 month"
ExpiresByType text/x-javascript "access plus 1 week"
ExpiresByType application/x-shockwave-flash "access plus 1 month"
ExpiresByType image/x-icon "access plus 1 year"
</IfModule>

If you use .htaccess files in your web directories to have them to be executed by the web server, you must be sure that AllowOverride is set to All for such directories:
<Directory /var/www/>
        Options Indexes FollowSymLinks
        AllowOverride All
        Require all granted
</Directory>

jed /etc/apache2/ports.conf # Port listened by the webserver (normally 80)
jed /etc/apache2/sites-enabled/default.conf # set DocumentRoot and create Virtual Hosts (vhosts). Older versions of Apache2 stores sites by default in /etc/apache2/sites-available/default

This is an example of a VirtualHost:
<VirtualHost www.example.com:80>
ServerName www.example.com
ServerAdmin webmaster@example.com
DocumentRoot "/var/www/example.com/www"
ServerSignature On
LogLevel Warn
ErrorLog /var/log/apache2/example-com_error.log
CustomLog /var/log/apache2/example-com_access.log combined
</VirtualHost>

Check Apache configuration for errors:
apache2ctl configtest

Restart Apache after modifying its configuration:
/etc/init.d/apache2 restart

See if Apache is running:
netstat -plntu | grep apache # if you get a line containing the socket listened by the Apache webserver, then it's running

See Apache error log:
cat /var/log/apache2/error.log

Delete Apache access and error logs
rm /var/log/apache2/*.gz # Delete archived logs
rm /var/log/apache2/* # In case you want to delete access and error logs for all websites. IMPORTANT: Note that you'll have to restart Apache to set up logs and have them running again

# If you can't see the phpmyadmin directory into /var/www/ then the package likely installed it into /usr/share/ , in this case you have to link it manually to access it from the web
ln -s /usr/share/phpmyadmin /var/www/


PHP configuration on Apache 2:
jed /etc/php4/apache2/php.ini # PHP 4 configuration
jed /etc/php5/apache2/php.ini # PHP 5 configuration
jed /etc/php/7.0/apache2/php.ini # PHP 7.0 configuration

Make sure engine = On in the PHP configuration file or PHP will not work at all.

You may want to set short_open_tag = On if your PHP code uses short tags, which means < ? ... ? > instead of < ?php ... ? >

You may specify the charset (character encoding) used by PHP generated pages. Note that it overrides charsets specified in the Apache configuration or in the HTML document (such as <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1" /> in HTML 4 or <meta charset="ISO-8859-1" /> in HTML 5), so you'd probably better leave it empty
default_charset = ""
however in case you need to set it, most popular settings are:
default_charset = "UTF-8"
or
default_charset = "iso-8859-1"

Show PHP version:
php -v

List compiled-in PHP modules:
php -m

To send e-mails from PHP scripts, you need to install sendmail
apt-get install sendmail sendmail-bin sendmail-doc rmail

Then edit the hosts file ( jed /etc/hosts ) and set the alias for the IP 127.0.0.1 as
127.0.0.1 DOMAIN localhost localhost.localdomain HOSTNAME
where DOMAIN is a domain name (for example: example.com ) addressed to your server's IP and HOSTNAME is the hostname of your server (that you can get with the command hostname ). The domain name is not necessary but strongly recommended because some antispam services blacklist servers which send e-mails identifying themselves (EHLO) as "localhost.localdomain".

After setting the hosts file you have to reconfigure and restart sendmail:
sendmailconfig # Normally you have to just reply with Y to all questions
service sendmail restart

To send a test e-mail from the console:
echo "Subject: test"|/usr/lib/sendmail -v -F SENDER@MAILBOX.EXAMPLE.COM -t RECEIVER@MAILBOX.EXAMPLE.COM

You should check sendmail_path in PHP configuration (php.ini) and make sure it's either disabled or set to the default value of sendmail -t -i

To send an e-mail from a PHP page you have to use the mail command, like in this example:
mail($emailto, $emailsubject, $emailbody, "From: ".$emailfrom."\nReply-To: ".$emailreplyto."\nX-Mailer: ".$xmailer);
Of course all sample variables used must be set properly. This is a test without using variables, you need to replace email addresses in this example with actual working ones:
mail('user2@mailbox.example.com', 'Test', "This is a test\nfrom PHP", "From: ".'user1@mailbox.example.com'."\nReply-To: ".'user1@mailbox.example.com'."\nX-Mailer: ".'PHP');

Check the log of last sent e-mails:
tail --lines=20 /var/log/mail.log


Show MySQL version:
mysql -V

See if Apache is running:
netstat -plntu | grep mysql # if you get a line containing the socket listened by the MySQL server, then it's running

Set MySQL root password:
$ mysql
 mysql> SET PASSWORD FOR root@localhost = PASSWORD('newpassword');

Create an alternate user with root-like access:
$ mysql -u root -pROOT_PASSWORD mysql

 mysql> CREATE USER 'USER_NAME'@'localhost' IDENTIFIED BY 'SET_PASSWORD_HERE'; GRANT ALL PRIVILEGES ON *.* TO 'USER_NAME'@'localhost'; FLUSH PRIVILEGES;

Create a new mysql user and assign a database to him
$ mysql -u root -pROOT_PASSWORD mysql

 mysql> insert into user (Host, User, Password, Select_priv) values ('localhost', 'USERNAME', password('USER_PASSWORD'), 'N');
 Query OK, 1 row affected (0.00 sec)

 mysql> insert into db (Host, Db, User, Select_priv, Insert_priv, Update_priv, Delete_priv, Create_priv, Drop_priv, Grant_priv, References_priv, Index_priv, Alter_priv, Create_tmp_table_priv, Lock_tables_priv) values ('localhost', 'USER_DATABASE', 'USER_NAME', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y');
 Query OK, 1 row affected (0.00 sec)

 mysql> quit

$ mysqladmin -u root -pROOT_PASSWORD reload

The user can access the assigned database:
$ mysql -u USERNAME -pUSER_PASSWORD USER_DATABASE

Remove an user:
mysql> drop user USERNAME@HOSTNAME;
Or
mysql> delete from mysql.user where User='USERNAME' and Host='HOSTNAME';
mysql> revoke all privileges on *.* from USERNAME@HOSTNAME;
mysql> revoke grant option on *.* from USERNAME@HOSTNAME;
mysql> flush privileges;

Show all MySQL users:
mysql> select user, host from mysql.user; # show only user name and related host name for every user
mysql> select * from mysql.user; # show detailed information for every user

Show all databases:
mysql> show databases;

Show all tables in the current database:
mysql> show tables;
mysql> show table status; # provide information about every table

Show all fields (coloumns) of a table:
mysql> show fields from TABLE; # same as describe TABLE;
mysql> show fields from TABLE from DB; # show fields of a table from another database
mysql> show full fields from TABLE; # also lists privileges available to the current user

Show all indexes (keys) of a table:
mysql> show index from TABLE;
mysql> show index from TABLE from DB; # show indexes of a table from another database

Create an index (key) for a field (coloumn):
mysql> alter table TABLE add index (`FIELD`) ;

Count all records within a table:
mysql> select count(*) from TABLE;

Backup a database to a MySQL dump file (structure only):
mysqldump -d -h localhost -u MYSQLUSER -pPASSWORD DATABASE > MYSQLDUMP.sql

Backup a database to a MySQL dump file (structure and data):
mysqldump -h localhost -u MYSQLUSER -pPASSWORD -c --add-drop-table --add-locks --all --quick --lock-tables DATABASE > MYSQLDUMP.sql

Back up a MySQL database into a GZIP file:
mysqldump -h localhost -u MYSQLUSER -pPASSWORD -c --add-drop-table --add-locks --all --quick --lock-tables DATABASE | gzip -9v > MYSQLDUMP_`date +%Y%m%d`.sql.gz

Restore a database from a MySQL dump file:
mysql -h localhost -u MYSQLUSER -pPASSWORD DATABASE < MYSQLDUMP.sql

Delete a database:
mysql> drop database if exists DATABASE;

Delete a table:
mysql> drop table if exists TABLE;

Delete a field:
mysql> alter table TABLE drop FIELD;

Delete an index:
mysql> drop index INDEX on TABLE;

Empty a table without deleting it:
mysql> truncate table TABLE;

Change engine type:
mysql> alter table TABLE type = MYISAM; # change the engine tipe of TABLE to MyISAM

Analyzes indexes of a table:
# for MyISAM and BDB tables only
mysql> analyze table TABLE;

Check a table for errors:
mysql> check table TABLE;

Attempt to repair a corrupted table
mysql> repair table TABLE;

Optimize a table (defragment and rebuild indexes):
mysql> optimize table TABLE;

Show MySQL server status:
mysql> show status;

Check running MySQL processes (queries):
$ mysqladmin -h localhost -u root -pROOT_PASSWORD processlist
Or, from the MySQL console:
mysql> show processlist;

Kill a MySQL process:
$ mysqladmin -u root -pROOT_PASSWORD kill ID
Or, from the MySQL console:
mysql> kill ID;

Quit MySQL console:
mysql> quit

Show MySQL version:
mysql -V

To restart MySQL:
/etc/init.d/mysql restart
# If MySQL fails to start make sure the directories /var/log/mysql and /var/log/mysqld are owned by mysql:adm

Run a MySQL file and store output into another file:
mysql -u USER --password=PASSWORD DATABASE_NAME < SOURCE.sql > DESTINATION.txt


This may fix these errors that may happen after an upgrade:
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock'
ERROR 1577 (HY000) at line 1: Cannot proceed because system tables used by Event Scheduler were found damaged at server start

apt-get install php5-mysql mysql-server mysql-server-5.1
mv /etc/mysql/my.cnf /etc/mysql/my.cnf.bak1
mv /etc/mysql/my.cnf.dpkg-dist /etc/mysql/my.cnf
/etc/init.d/mysql start
apt-get install php5-mysql mysql-server mysql-server-5.1


Upgrading from PHP 4 to PHP 5:

# Note: this will also upgrade from MySQL 4 to MySQL 5, if MySQL 4 is installed
# Warning: newer versions of MySQL often use different formats for tables than earlier versions. In some cases you can attempt to make restore such tables compatible using  repair table TABLE;  and  optimize table TABLE; You'd better dump all your tables before to upgrade, in case you'll need to restore them after. In fact you may need to DROP tables which are no longer working as expected and rebuild them. A common issue are tables containing floating point fields: queries containing comparisons based on those fields will no longer work as they used to.

# stop the servers
/etc/init.d/apache2 stop
/etc/init.d/mysql stop

# remove PHP 4
apt-get remove libapache2-mod-php4 php4 php4-gd php4-pear php4-mcrypt php4-mysql

# You have to remove MySQL as well
apt-get remove mysql-server

# Install PHP 5
apt-get install php5 libapache2-mod-php5 php5-sqlite php-pear php5-gd php5-mcrypt # PHP 5
apt-get install php5-mysql # MySQL support for PHP 5

# Install MySQL again
apt-get install mysql-server

# Reinstall phpmyadmin (if you need it)
apt-get install phpmyadmin # Removing PHP 4 will remove phpmyadmin as well, you'll have to reinstall it if you need it

# Make sure that there are no references to PHP 4 left in the Apache configuration
jed /etc/apache2/apache2.conf
# Change <IfModule mod_php4.c> as <IfModule mod_php5.c>
# Add .php5 to AddType application/x-httpd-php
# Comment out <IfDefine HAVE_PHP4> ... </IfDefine> and everything between

# restart the servers
/etc/init.d/mysql restart
/etc/init.d/apache2 restart


Switch from PHP 5 to PHP 7:

If you have two different versions of PHP, for example PHP 5.6 and PHP 7.0, as it can happen after upgrading Debian from jessie to stretch, you can switch between them with these commands:

update-alternatives --config php # Set the default PHP version for the command php (this will not influence the version used on the Apache webserver)

a2dismod php5 # Dismount the Apache2 Module for PHP5
a2enmod php7.0 # Mount the Apache2 Module for PHP7
systemctl restart apache2 # Restart Apache


If you want to have https websites (so that the client can access the server securely through an encrypted connection to the server) hosted in your webserver, follow this procedure:

Certificates and keys should be stored in these directories:
/etc/ssl/certs # for certificates
/etc/ssl/private # for keys

Generate a Key:
openssl genrsa -out /etc/ssl/private/DOMAIN.key 2048
Enter a valid domain name 

Generate a request certificate:
openssl req -new -sha256 -key /etc/ssl/private/DOMAIN.key -out www.example.com.csr

The only fields you must fill are Organization Name (eg, company) and Common Name (e.g. server FQDN or YOUR name), you can leave empty all other fields. Note that fields with a given default value must be filled with a dot (.) to make them empty, if you simply press RETURN the default value will be used instead:

----------
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:.
State or Province Name (full name) [Some-State]:.
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:COMPANY
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:DOMAIN
Email Address []:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
----------

If you want to validate your domain (to claim its possession) via e-mail, enable the e-mail address admin@DOMAIN (that is, make sure you can receive e-mails to admin) for validation

Buy a certificate from a Certificate Authority (CA)
https://www.digicert.com/
https://www.ssl.com/ (via comodo)
https://www.comodo.com/

Validate the domain (claim it as yours) according to the procedure requested by the Certificate Authority

Download the certificates ( ca-bundle-client.crt , DOMAIN.crt )
Upload the certificates into /etc/ssl/certs (the private key you've generated should already be into /etc/ssl/private/ )

Edit the Apache configuration file:
jed /etc/apache2/sites-enabled/default.conf

Create a VirtualHost for the DOMAIN for port 443 with the following directives:

<VirtualHost DOMAIN:443>
SSLEngine on
SSLCertificateFile /etc/ssl/certs/DOMAIN.crt
SSLCertificateKeyFile /etc/ssl/private/DOMAIN.key
SSLCertificateChainFile /etc/ssl/certs/ca-bundle-client.crt
# HSTS (mod_headers is required) (15768000 seconds = 6 months)
# Header always set Strict-Transport-Security "max-age=15768000"
# ...
</VirtualHost>

For example:

<VirtualHost www.example.com:443>
SSLEngine on
SSLCertificateFile /etc/ssl/certs/example_com.crt
SSLCertificateKeyFile /etc/ssl/private/example_com.key
SSLCertificateChainFile /etc/ssl/certs/ca-bundle-client.crt
# HSTS (mod_headers is required) (15768000 seconds = 6 months)
# Header always set Strict-Transport-Security "max-age=15768000"
ServerName www.example.com
ServerAdmin webmaster@example.com
DocumentRoot "/var/www/example.com/www"
ServerSignature On
LogLevel Warn
ErrorLog /var/log/apache2/example-com_error.log
CustomLog /var/log/apache2/example-com_access.log combined
</VirtualHost>

If you still want the website to be available via http (without encryption), you also have to configure it without SSL (normally for port 80). Note that it can be configured to have a different document directory () and thus provide different content from the https (encrypted) website.

<VirtualHost www.example.com:80>
ServerName www.example.com
ServerAdmin webmaster@example.com
DocumentRoot "/var/www/example.com/www"
ServerSignature On
LogLevel Warn
ErrorLog /var/log/apache2/example-com_error.log
CustomLog /var/log/apache2/example-com_access.log combined
</VirtualHost>

If you want to redirect all http traffic to https for a portion of the website, add this line inside the VirtualHost for http (port :80)
Redirect /ALWAYS_HTTPS_DIRECTORY_NAME https://www.example.com/ALWAYS_HTTPS_DIRECTORY_NAME
If you want to redirect all http traffic to https for the whole website, add this line inside the VirtualHost for http (port :80)
Redirect / https://www.example.com/

Restart the webserver when you're done with editing the configuration file:
/etc/init.d/apache2 restart


Password protected directories in Apache:

# Create a password file, it should be placed in a directory not served by the webserver (for example, /var/ , which normally is out of the htdocs directory)
touch PATH/.htpasswd
chmod 644 PATH/.htpasswd

# Create users and assign passwords to them
htpasswd -b PATH/.htpasswd USER PASSWORD # Create a password for every user. The same statement can be used to change a password of an already existing user

htpasswd -D PATH/.htpasswd USER # Delete a user from the password file

# Create an access control file in the directory containing the files with restricted access
# Use Require valid-user to allow access to any user in the password file, or list specific users allowed to access the restricted files
jed PATH_TO_PROTECT/.htaccess
---
AuthUserFile PATH/.htpasswd
AuthName "Enter Password"
AuthType Basic
<Limit GET POST>
# Require valid-user
Require user USER1
Require user USER2
</Limit>
---

Web site statistics:

grep " 404 " /var/log/apache2/ACCESS.log | awk -F\" '$9=404{print $9" "$2}' | sort | uniq -c | sort -rg | head --lines=25 # Show top 404 error ("Not Found") requested pages
grep -i "GET /PATH/DOCUMENT" /var/log/apache2/ACCESS.log | awk '{print $1}' | sort -n | uniq -c | sort -rn | head --lines=25 # Show top IPs accessing the given DOCUMENT (web page)


Webalizer:

apt-get install webalizer
jed /etc/webalizer.conf # or jed /etc/webalizer/webalizer.conf # or another custom .conf file
webalizer /var/log/apache2/access.log.1 # create a webalizer report for access.log.1
webalizer /var/log/apache2/example-com_access.log # create a webalizer report for example-com_access.log

If you get the following error message:
Error Opening file /usr/share/GeoIP/GeoIP.dat
you can fix it installing the GeoIP database:
apt-get install geoip-database

To have statistics available on the web for many websites, create a different access log, webalizer configuration file and output directory, for every website and a configuration file, and call webalizer from a script.

Example:

# 1. Set Apache log files
jed /etc/apache2/sites-available/default

<VirtualHost www.example.com>
[...]
CustomLog /var/log/apache2/example-com_www_access.log combined
[...]
</VirtualHost>

<VirtualHost my.example.net>
[...]
CustomLog /var/log/apache2/example-net_my_access.log combined
[...]
</VirtualHost>

/etc/init.d/apache2 restart

# 2. Create output directories
mkdir /var/www/webalizer
mkdir /var/www/webalizer/www.example.com
mkdir /var/www/webalizer/my.example.net

# 3. Assign ownership of the directories to www-data
chown www-data:www-data /var/www/webalizer/www.example.com
chown www-data:www-data /var/www/webalizer/my.example.net

# 4. Create custom Webalizer configuration files
jed /etc/webalizer/webalizer_www.example.com.conf
LogFile         /var/log/apache2/example-com_www_access.log
OutputDir       /var/www/webalizer/www.example.com
Incremental     yes
HostName        www.example.com
HideSite        *example.com
HideReferrer    example.com/
[...]

jed /etc/webalizer/webalizer_my.example.net.conf
LogFile         /var/log/apache2/example-net_my_access.log
OutputDir       /var/www/webalizer/my.example.net
Incremental     yes
HostName        my.example.net
HideSite        *example.com
HideReferrer    example.com/
[...]

# 5. Create a script that invokes webalizer for every website

mkdir /etc/scripts

# You can have a verbose version to be invoked from a shell window, and a quiet version to be launched from CronTab:

jed /etc/scripts/webalizev.sh
#!/bin/bash
# Webalize (verbose)
echo -e "\nWebalizer for www.example.com"
webalizer -c /etc/webalizer/webalizer_www.example.com.conf
echo -e "\nWebalizer for my.example.net"
webalizer -c /etc/webalizer/webalizer_my.example.net.conf

jed /etc/scripts/webalize.sh
#!/bin/bash
# Webalize (quiet)
webalizer -c /etc/webalizer/webalizer_www.example.com.conf>/dev/null
webalizer -c /etc/webalizer/webalizer_my.example.net.conf>/dev/null

Sample lines to add to Crontab:
1 0,12 * * * /var/www/scripts/webalize.sh>/dev/null
21 6 * * * /var/www/scripts/webalize.sh>/dev/null
The first line tells Crontab to invoke Webalizer every day at 00:01 and 12:01, the second line makes Crontab invoke Webalizer every day at 6:21 . This is because logs start and end at a certain hour of the day (for example, 06:26), and you'd miss all the hours between last call of Webalizer, every day. To know the exact time when your logs starts and stops check the head and tail of the previous log, for example: head --lines=10 /var/log/apache2/www.example.net_access.log.1 , tail --lines=10 /var/log/apache2/www.example.net_access.log.1

# If you want to reset information cached from previous runs of Webalizer (especially if you've set Incremental yes in the webalizer configuration file), delete the files webalizer.current and webalizer.hist in the Webalizer OutputDir.
# Example:
# rm /var/www/webalizer/webalizer.current
# rm /var/www/webalizer/webalizer.hist

-----

SQLite:

apt-get install sqlite sqlite3 libsqlite3-dev

# To fix these errors: sqlite_exec(): attempt to write a readonly database; sqlite_exec(): unable to open database file
chmod 666 *.sdb
chmod 777 the directory containing the .sdb file

-----

FTP server:

PROFTP:

apt-get install proftpd
(run as standalone)

Create an user called "www" to access the www directory:

adduser --home /var/www www

Allow FTP access to the user "www":

jed /etc/proftpd.conf

Add the following lines in the user section of the proftpd configuration file:

<Anonymous ~www>
User www
Group www
DirFakeUser on ftp
DirFakeGroup on ftp
RequireValidShell off
MaxClients 2
DisplayLogin welcome.msg
DisplayFirstChdir .message
 <Directory *>
  <Limit READ WRITE STOR>
   AllowAll
  </Limit>
 </Directory>
</Anonymous>

/etc/init.d/proftpd restart


VSFTP:
# VSFTP allows virtual users, besides real shell users and anonymous users (you can simulate a chroot'ed FTP)

apt-get install vsftpd
apt-get install libpam-pwdfile

Create a password file for virtual users:
# Note: htpasswd only allows passwords up to 8 characters long
htpasswd -bc /etc/vsftpd_passwd USERNAME PASSWORD

Additional users can be created without the -c parameter. The same statement can be used to change the password for an existing user:
htpasswd -b /etc/vsftpd_passwd USERNAME PASSWORD

If you want to enter the password manually, omit the -b parameter:
htpasswd /etc/vsftpd_passwd USERNAME

Use this command to remove a virtual user:
htpasswd -D /etc/vsftpd_passwd USERNAME

You'll need to create a directory for each virtual user (unless you configure vsftp otherwise):

First, create the main directory for all virtual users:
mkdir /var/www/ftp

Then create a directory for each virtual user with proper read/write permissions:
mkdir /var/www/ftp/USERNAME
chmod 755 /var/www/ftp/USERNAME
chown ftp /var/www/ftp/USERNAME


Configure PAM (Pluggable Authentication Modules aka X/Open Single Sign-on) for vsftp:
jed /etc/pam.d/vsftpd
# Log in using htpasswd password file
auth    required pam_pwdfile.so pwdfile /etc/vsftpd_passwd
account required pam_permit.so


jed /etc/vsftpd.conf
# Config file /etc/vsftpd.conf
listen=YES
#listen_ipv6=YES
anonymous_enable=NO
local_enable=YES
write_enable=YES
local_umask=022
anon_upload_enable=NO
anon_mkdir_write_enable=NO
dirmessage_enable=YES
xferlog_enable=YES
connect_from_port_20=YES
#chown_uploads=YES
#chown_username=ftp
xferlog_file=/var/log/vsftpd.log
#xferlog_std_format=YES
idle_session_timeout=600
data_connection_timeout=120
#nopriv_user=ftp
#async_abor_enable=YES
#ascii_upload_enable=YES
#ascii_download_enable=YES
ftpd_banner=Welcome
#deny_email_enable=YES
#banned_email_file=/etc/vsftpd.banned_emails
chroot_local_user=YES
#chroot_list_file=/etc/vsftpd.chroot_list
#ls_recurse_enable=YES
secure_chroot_dir=/var/run/vsftpd
pam_service_name=vsftpd
rsa_cert_file=/etc/ssl/certs/vsftpd.pem
guest_enable=YES
virtual_use_local_privs=YES
#user_config_dir=/etc/vsftpd_user_conf
#guest_username=ftp
user_sub_token=$USER
local_root=/var/www/ftp/$USER
hide_ids=YES
#cmds_allowed=PASV,BYE,LIST,ABOR,CWD,NLST,PORT,PWD,QUIT,RETR,SIZE,TYPE

# Note that the option user_config_dir lets you create custom vsftp configuration files for each virtual users
# You can create a config file with the name of the virtual user within the specified path
jed /etc/vsftpd_user_conf/USERNAME
# VSFTP extra config for user USERNAME
local_root=/var/www/example.com/www/USERNAME


Restart VSFTP to make it work with the new users or configuration:
/etc/init.d/vsftpd restart

-----

Samba:

apt-get install samba smbfs smbclient samba-doc

mkdir /public
mkdir /public/shared
chmod -v 0777 /public
chmod -v 0777 /public/shared

jed /etc/samba/smb.conf

---
# samba configuration file

[global]
allow hosts = 192.168.0.0/255.255.255.0
workgroup = network
server string = %h server (Samba %v)
log file = /var/log/samba/log.%m
max log size = 1024

; security = user
encrypt passwords = true
passdb backend = tdbsam guest
guest account = nobody
invaluid users = root

preserve case = yes
short preserve case = yes

; include /home/samba/etc/smb.conf.%m
socket options = TCP_NODELAY

[homes]
comment = Home directories
browseable = no
writable = no
create mask = 0700
directory mask = 0700

[shared]
comment = Public shared directory
path = /public/shared
public = yes
browseable = yes
; read only = no
writable = yes
; printable = no
create mask = 0766
directory mask = 0766
guest ok = yes
; valid users = user_name
---

testparm /etc/samba/smb.conf

/etc/init.d/samba restart

smbpasswd -a nobody
New SMB password:
Retype new SMB password:

log file (default):
/var/log/samba/log.%m

smbtree # find Windows machines

-----

Linux Shell scripts:

First line headers:

Bourne Shell (sh)
#!/bin/sh

Bourne Again Shell (bash)
#!/bin/bash

C-Shell (csh)
#!/bin/csh

Korn Shell (ksh)
#!/bin/ksh

Perl
#!/usr/bin/perl

PHP
#!/usr/bin/php

Python
#!/usr/bin/python

Turbo Shell (tcsh)
#!/usr/bin/tcsh

TCL/Tk
#!/usr/bin/wish -f

Set the executable flag:
chmod +x FILE # declares the file as executable

-----

Invoke Linux shell commands from PHP:

PHP can perform calls to the Linux shell using the shell_exec() function:

 $output = shell_exec('ls -laF');
 echo '<pre>'.$output.'</pre>';

or enclosing the Linux shell command between backticks:

 $output = `ls -laF`;
 echo '<pre>'.$output.'</pre>';

PHP also has a system() function that returns the value returned by the Linux shell command and the last line of output of such command:

 $last_line = system('ls -laF', $returned_value);

-----

GUI commands
# Commands for the Graphical User Interface

xwinifo # Enter this command and then click on any window to get information about it such as size in pixels and color depth. You may also click on the desktop.

sudo rm -rf  ~/.local/share/Trash/files/* # Empty trashcan

-----

Fortune cookies:
apt-get install fortunes
/usr/games/fortune  # return a fortune cookie (fetches a random sentence (% separated lines) from a random file in /usr/share/games/fortunes/ )
/usr/games/fortune PATH/FILE  # return a fortune cookie fetching a random sentence (% separated lines) from the given FILE

To create your own fortune files:
- Create a text file separating each fortune cookie with lines containing only a percent (%) sign.
- Create a database file for that text file, with the same file name followed by a .dat extension, and place it in the same directory: strfile PATH/FILENAME PATH/FILENAME.dat

-----

ASCII Art:

Labyrinth:
c=(\/ \\); n=${#c[@]}; clear; printf "%s" "${c[RANDOM%n]"{1..1920}"}"
c=(\/ \\); n=${#c[@]}; clear; while :; do printf -- "${c[RANDOM%n]}"; done # Endless labyrinth (break with CTRL+C)

apt-get install figlet

figlet "Hello" # Type Hello using ASCII Art

apt-get install cowsay

/usr/games/cowsay "Hello" # Show a cow saying Hello
/usr/games/cowthink "Hello" # Show a cow thinking Hello
/usr/games/cowsay -f CHARACTER "Hello" # Use alternate CHARACTERs, for example: tux, moose, sheep, milk
/usr/games/cowthink -f CHARACTER "Hello" # Use alternate CHARACTERs, for example: tux, moose, sheep, milk
ls /usr/share/cowsay/cows # Show available cows (characters)
/usr/games/fortune|/usr/games/cowsay # Fortune teller cow (requires fortune cookies to be installed)
/usr/games/cowsay -f "$(ls /usr/share/cowsay/cows | sort -R | head -1)" "$(/usr/games/fortune -s)" # Random character tells fortune

apt-get install sl

sl # show a running steam locomotive in ASCII Art
sl -c # show a small steam locomotive
sl -F # show a flying steam locomotive
sl -a # show people crying for help into a steam locomotive
sl -a # allow the animation to be interrupted with CTRL+C

-----

Text editors:

Jed

apt-get install jed

jed FILE.txt # Create or edit FILE.txt
jed FILE.txt -g LINE # Open FILE.txt for editing, set cursor position at the given LINE
jed FILE.txt -s "TEXT" # Open FILE.txt for editing, set cursor position at the firt occurrence of the given TEXT


Vim

# While Jed is an easy and intuitive text editor, it might happen that it's not available in the system on which you are working.
# In this case you'll likely have to use vim instead which is powerful in trained hands but much less intuitive.
# This is a list of some basic and useful commands

Cursor keys / h, l, j, k : Cursor movement (left, right, down, up)
b : Jump to the word on the left
w : Jump to the word on the right
( : Jump to the sentence on the left
) : Jump to the sentence on the right
^ / 0 : Jump to the beginning of the line
$ / gm : Jump to the end of the line
- : Jump to the first non blank character of the previous line
+ : Jump to the first non blank character of the next line
{ : Jump to the previous paragraph
} : Jump to the next paragraph
[[ : Jump to the beginning of the file
]] : Jump to the end of the file
[LINE_NUMBER]G / [LINE_NUMBER]gg : Jump to the given LINE NUMBER
[VALUE]% : Jump to the specified position of the document
[COLOUMN]| : Jump to the specified COLOUMN in the current line
[LINE]H : Jump to the given LINE visible in the current window, from the top
[LINE]L : Jump to the given LINE visible in the current window, from the bottom
M : Jump to the line in the middle of the current window

zt : Place current line at the top of the window
zz : Place current line at the middle of the window
zb : Place current line at the bottom of the window

i : Enter Insert mode
I : Jump at the beginning of the line and enter Insert mode
s : Delete the character under the cursor and enter Insert mode
R : Enter Replace mode
ESC : Exit Edit mode (either Insert or Replace (Overwrite))

Commands in Edit mode (^ means CONTROL):
^w : Delete characters from cursor position to the beginning of the current word
^u : Delete characters from cursor position to the beginning of the current line
^t : Add a Tab at the beginning of the current line (Add indentation)
^r : Remove a Tab from the beginning of the current line (Remove indentation)

r : Replace a single character at the cursor position
o : Create a new empty line below the current one and enter Insert mode
O : Create a new empty line above the current one and enter Insert mode
gu[CURSOR_KEY] : Change a single character to lowercase in the direction specified with the cursor key
gU[CURSOR_KEY] : Change a single character to uppercase in the direction specified with the cursor key

x : Remove the character at the cursor (Delete)
X : Remove the character before than the cursor (Backspace)
D : Delete until the end of the current line
dd : Delete the current line
J : Join the current line with the next one adding a space inbetween

Y / yy : Copy the current line
P / p : Paste the line in the clipboard 

ga : Show the ASCII value of the character at the cursor position

/[TEXT] : Search for the given TEXT
n : Repeat last search, forward
N : Repeat last search, backwards

:s/[SEARCH]/[REPLACE]/ : Search and Replace the first occurrence in the current line only, case sensitive
:s/[SEARCH]/[REPLACE]/g : Search and Replace every occurrence in the current line only, case sensitive
:%s/[SEARCH]/[REPLACE]/g : Search and Replace every occurrence in the whole file, case sensitive
:%s/[SEARCH]/[REPLACE]/gI : Search and Replace every occurrence in the whole file, case sensitive
:%s/[SEARCH]/[REPLACE]/gi : Search and Replace every occurrence in the whole file, case insensitive
:%s/[SEARCH]/[REPLACE]/gic : Search and Replace every occurrence in the whole file, case insensitive, and ask for confirmation

:%s/\s\+$// : Remove trailing spaces in the whole file
:%s/^\s\+// : Remove leading spaces in the whole file

:set number / :set nu : Display line numbers
:set nonumber / :set nonu : Turn off line numbers
:set number! / :set nu! : Toggle line numbers
:set relativenumber : Display line numbers relative to the cursor line
:set norelativenumber : Remove line numbers relative to the cursor line
:set relativenumber! : Toggle line numbers relative to the cursor line
# If number and relativenumber are both enabled the absolute line number will be shown in the cursor line (instead of 0)

:set list : Display special characters
:set nolist : Hide special characters
:set list! : Toggle special characters
:set listchars=eol:$,tab:>-,trail:~,nbsp=! : Configure how to show special characters
:help listchars : Display help for listchars (quit with :q)

:syntax on : Enable syntax
:syntax off : Disable syntax
:set syntax=[LANGUAGE] : Display syntax for the specified language (example: html, php, whitespace)

:w : Save current file
:wq / :x / ZZ : Save current file and quit
:q : Quit (it only quits if the current file is saved, otherwise returns an error message)
:q! : Quit without saving (doesn't ask for confirmation)

-----

Easter eggs:

$ apt-get moo
         (__)
         (oo)
   /------\/
  / |    ||
 *  /\---/\
    ~~   ~~
...."Have you mooed today?"...

$ apt-get -h
[...]
                       This APT has Super Cow Powers.

$ aptitude -h
[...]
                  This aptitude does not have Super Cow Powers.

$ aptitude moo
There are no Easter Eggs in this program.
$ aptitude -v moo
There really are no Easter Eggs in this program.
$ aptitude -vv moo
Didn't I already tell you that there are no Easter Eggs in this program?
$ aptitude -vvv moo
Stop it!
$ aptitude -vvvv moo
Okay, okay, if I give you an Easter Egg, will you go away?
$ aptitude -vvvvv moo
All right, you win.

                               /----\
                       -------/      \
                      /               \
                     /                |
   -----------------/                  --------\
   ----------------------------------------------

Happy?
$ aptitude -vvvvvv moo
What is it?  It's an elephant being eaten by a snake, of course.

# Note: aptitude's easter egg is a reference to "The Little Prince" written in 1943 by Antoine de Saint-Exupéry.

-----






Page issued on 23-Apr-2018 05:27 GMT
Copyright (c) 2018 Geody - Legal notices: copyright, privacy policy, disclaimer