# The Debian Linux Magic Spells File Cheat Sheet
# by Elf Qrin - https://labs.geody.com/
v1.89 r2021-10-11 fr2005-09-xx

# "One man cast a lingering spell of awe and wonder, of magical innocence overcoming evil, of simple courage conquering fear" -- J.R.R. Tolkien's The Lord of the Rings (1978 Ralph Bakshi animated film)
# 'someone who thinks of code as magical incantations and asks only "what do I need to type to make this happen?"' -- from the definition of Script Kiddies in the Jargon File

-----

Notes:

- In this file, jed is used as the default editor for it's more intuitive than others. You'll have to install it ( apt-get install jed ). Alternatively, you might want to use vi or vim ( apt-get install vim ) instead.
- Most commands work also on Ubuntu / Kubuntu / Xubuntu / Lubuntu / Edubuntu, Mint, Xandros, Raspbian, and other Debian-based distributions (Debian derivatives). Some commands work on all Linux distributions.
- You may send contributions and suggestions to (see: https://www.geody.com/contacts.php )

-----

Terminal:

- You can access the terminal in most Debian based distributions (including Debian itself and Ubuntu) by pressing CTRL+ALT+F1 (actually you can switch between terminals with CTRL+ALT+function keys. With CTRL+ALT+F7 you normally return to the graphic interface in Ubuntu). In Xandros, you have to press CTRL+ALT+T.
- To access a remote terminal via SSH from Windows you may use the free and open source software PuTTY.
- To access a remote terminal via SSH from Android you may use the free and open source software ConnectBot.

Keyboard shortcuts for the terminal:

TAB # Autocomplete the command, directory or file name you are typing. If it can't autocomplete your text from the characters you've typed, hit it again to show all matching possibilities

CTRL+A # Move the cursor to the beginning of the line
CTRL+E # Move the cursor to the end of the line

CTRL+L # Clear the Screen (scrolls all the all the text above the visible area, same as printf "\033c" )

CTRL+C # Break (Halt) the current running program (signal 2)
CTRL+D # Exit / Logout

CTRL+U # Delete all characters to the left of the cursor up to the beginning of the line
CTRL+K # Delete all characters to the right of the cursor up to the end of the line
CTRL+W # Delete all characters to the left up to the beginning of the current word
CTRL+Y # Paste the text erased using CTRL+U, CTRL+K, CTRL+W. Note that this uses a separate clipboard.

CTRL+P or CURSOR UP # Show the previous command in History
CTRL+N or CURSOR DOWN # Show the next command in History

-----

Linux Filesystem Hierarchy
Meaning of Linux directories:

/ # Root directory. The operating system kernel must be located in either / or /boot.
/bin # Essential user command binaries (for use by all users). It contains commands that may be used by both the system administrator and by users, but which are required when no other filesystems are mounted (e.g. in single user mode). It may also contain commands which are used indirectly by scripts.
/boot # Static files of the boot loader. Files required for booting the system. This directory contains everything required for the boot process except configuration files not needed at boot time and the map installer. Thus /boot stores data that is used before the kernel begins executing user-mode programs. This may include saved master boot sectors and sector map files.
/cdrom # Non standard directory for CD Roms. It should be inside /media instead.
/dev # Device files.
/etc # Et cetera. Host-specific system configuration. The /etc hierarchy contains configuration files. A "configuration file" is a local file used to control the operation of a program; it must be static and cannot be an executable binary. It is recommended that files be stored in subdirectories of /etc rather than directly in /etc. 
/etc/opt # Configuration files for /opt .
/etc/X11 # Configuration for the X Window System (optional).
/etc/sgml # Configuration files for SGML (optional).
/etc/xml # Configuration files for XML (optional).
/home # User home directories. No program should assume any specific location for a home directory, rather it should query for it. (optional).
/lib # Essential shared libraries and kernel modules needed to boot the system and run the commands in the root filesystem, ie. by binaries in /bin and /sbin.
/lost+found # Files recovered after a system crash.
/media # Mount point for removable media. This directory contains subdirectories which are used as mount points for removable media such as floppy disks, cdroms, zip disks.
/mnt # Mount point for a temporarily mounted filesystem. This directory is provided so that the system administrator may temporarily mount a filesystem as needed. The content of this directory is a local issue and should not affect the manner in which any program is run. This directory must not be used by installation programs: a suitable temporary directory not in use by the system must be used instead.
/opt # Add-on application software packages. A package to be installed in /opt must locate its static files in a separate /opt/[package] or /opt/[provider] directory tree, where [package] is a name that describes the software package and [provider] is the provider's LANANA (Linux Assigned Names And Numbers Authority) registered name.
/proc # Kernel and process information virtual filesystem.
/root # Home directory for the root user (optional). The root account's home directory may be determined by developer or local preference, but this is the recommended default location.
/run # Run-time variable data. This directory contains system information data describing the system since it was booted. Files under this directory must be cleared (removed or truncated as appropriate) at the beginning of the boot process. The purposes of this directory were once served by /var/run. In general, programs may continue to use /var/run to fulfill the requirements set out for /run for the purposes of backwards compatibility. Programs which have migrated to use /run should cease their usage of /var/run, except as noted in the section on /var/run. Programs may have a subdirectory of /run; this is encouraged for programs that use more than one run-time file. Users may also have a subdirectory of /run, although care must be taken to appropriately limit access rights to prevent unauthorized use of /run itself and other subdirectories.
/sbin # Essential system binaries. Utilities used for system administration (and other root-only commands) are stored in /sbin, /usr/sbin, and /usr/local/sbin. /sbin contains binaries essential for booting, restoring, recovering, and/or repairing the system in addition to the binaries in /bin. Programs executed after /usr is known to be mounted (when there are no problems) are generally placed into /usr/sbin. Locally-installed system administration programs should be placed into /usr/local/sbin.
/srv # Data for services provided by this system.
/sys # Kernel and system information virtual filesystem.
/tmp # Temporary files. Programs must not assume that any files or directories in /tmp are preserved between invocations of the program. It is recommended that files and directories located in /tmp be deleted whenever the system is booted. Temporary files that can survive a reboot should be stored in /var/tmp .
/usr # "usr" stands for "UNIX System Resources", not "user". It contains shareable, read-only data.
/usr/bin # Most user commands. This is the primary directory of executable commands on the system.
/usr/include # Directory for standard include files. This is where all of the system's general-use include files for the C programming language should be placed.
/usr/lib # Libraries for programming and packages.
/usr/libexec # Binaries run by other programs (optional). It includes internal binaries that are not intended to be executed directly by users or shell scripts. Applications may use a single subdirectory under /usr/libexec. Applications which use /usr/libexec in this way must not also use /usr/lib to store internal binaries, though they may use /usr/lib for the other purposes documented here. /usr/lib can be also used if /usr/libexec is missing.
/usr/local # Local hierarchy, for use by the system administrator when installing software locally. It needs to be safe from being overwritten when the system software is updated. It may be used for programs and data that are shareable amongst a group of hosts, but not found in /usr. Locally installed software must be placed within /usr/local rather than /usr unless it is being installed to replace or upgrade software in /usr.
/usr/sbin # Non-essential standard system binaries. This directory contains any non-essential binaries used exclusively by the system administrator. System administration programs that are required for system repair, system recovery, mounting /usr, or other essential functions must be placed in /sbin instead. There must be no subdirectories in /usr/sbin.
/usr/share # Architecture-independent data. This hierarchy is intended to be shareable among all architecture platforms of a given OS; thus, for example, a site with i386, Alpha, and PPC platforms might maintain a single /usr/share directory that is centrally-mounted. Note, however, that /usr/share is generally not intended to be shared by different OSes or by different releases of the same OS. Any program or package which contains or requires data that doesn't need to be modified should store that data in /usr/share (or /usr/local/share, if installed locally). It is recommended that a subdirectory be used in /usr/share for this purpose. Applications using a single file may use /usr/share/misc. Game data stored in /usr/share/games must be purely static data. Any modifiable files, such as score files, game play logs, and so forth, should be placed in /var/games.
/usr/share/color # Color management information (optional).
/usr/share/dict # Word lists (optional).
/usr/share/man # Manual pages.
/usr/share/misc # Miscellaneous architecture-independent data.
/usr/share/ppd # Printer definitions (optional).
/usr/share/sgml # SGML data (optional).
/usr/share/xml # XML data (optional).
/usr/src # Source code (optional). The only source code that should be placed in a specific location is the Linux kernel source code. It is located in /usr/src/linux.
/var # variable data files. This includes spool directories and files, administrative and logging data, and transient and temporary files. Some portions of /var are not shareable between different systems. For instance, /var/log, /var/lock, and /var/run. Other portions may be shared, notably /var/mail, /var/cache/man, /var/cache/fonts, and /var/spool/news. /var exists in order to make it possible to mount /usr read-only. Everything that once went into /usr that is written to during system operation (as opposed to installation and software maintenance) must be in /var.
/var/account # Process accounting logs (optional). This directory holds the current active process accounting log and the composite process usage data (as used in some UNIX-like systems by lastcomm and sa).
/var/cache # Application cache data. /var/cache is intended for cached data from applications. Such data is locally generated as a result of time-consuming I/O or calculation. The application must be able to regenerate or restore the data. Unlike /var/spool, the cached files can be deleted without data loss. The data must remain valid between invocations of the application and rebooting the system. Files located under /var/cache may be expired in an application specific manner, by the system administrator, or both. The application must always be able to recover from manual deletion of these files (generally because of a disk space shortage). No other requirements are made on the data format of the cache directories.
/var/cache/fonts # Locally-generated fonts (optional). It should be used to store any dynamically-created fonts. In particular, all of the fonts which are automatically generated by mktexpk must be located in appropriately-named subdirectories of /var/cache/fonts. An application (or a group of inter-related applications) must use a subdirectory of /var/lib for its data. There is one required subdirectory, /var/lib/misc, which is intended for state files that don't need a subdirectory; the other subdirectories should only be present if the application in question is included in the distribution.
/var/cache/man # Locally-formatted manual pages (optional).
/var/crash # System crash dumps (optional).
/var/games # Variable game data (optional).
/var/lib # Variable state information.  State information is data that programs modify while they run, and that pertains to one specific host. Users must never need to modify files in /var/lib to configure a package's operation, and the specific file hierarchy used to store the data must not be exposed to regular users.
/var/lib/[editor] # Editor backup files and state (optional).
/var/lib/color # Color management information (optional).
/var/lib/hwclock # State directory for hwclock (optional).
/var/lib/misc # Miscellaneous variable data.
/var/lock # Lock files. Lock files for devices and other resources shared by multiple applications, such as the serial device lock files that were originally found in either /usr/spool/locks or /usr/spool/uucp, must now be stored in /var/lock. The naming convention which must be used is "LCK.." followed by the base name of the device. For example, to lock /dev/ttyS0 the file "LCK..ttyS0" would be created. The format used for the contents of such lock files must be the HDB UUCP lock file format. The HDB format is to store the process identifier (PID) as a ten byte ASCII decimal number, with a trailing newline. For example, if process 1230 holds a lock file, it would contain the eleven characters: space, space, space, space, space, space, one, two, three, zero, and newline.
/var/log # Log files and directories. This directory contains miscellaneous log files. Most logs must be written to this directory or an appropriate subdirectory. The following files, or symbolic links to files, must be in /var/log, if the corresponding subsystem is installed: lastlo: record of last login of each user; messages: system messages from syslogd; wtmp: record of all logins and logouts.
/var/mail # User mailbox files (optional). User mailbox files in this location must be stored in the standard UNIX mailbox format.
/var/opt # Variable data for /opt . This is where the static data from an add-on software package is stored, except where superseded by another file in /etc .
/var/run # Run-time variable data. This directory was once intended for system information data describing the system since it was booted. These functions have been moved to /run; this directory exists to ensure compatibility with systems and software using an older version of this specification.
/var/spool # Application spool data. It contains data which is awaiting some kind of later processing. Data in /var/spool represents work to be done in the future (by a program, user, or administrator); often data is deleted after it has been processed.
/var/spool/cron # cron and at jobs. This directory contains the variable data for the cron and at programs.
/var/spool/lpd # Line-printer daemon print queues (optional). The lock file for lpd, lpd.lock, must be placed in /var/spool/lpd. It is suggested that the lock file for each printer be placed in the spool directory for that specific printer and named lock.
/var/spool/rwho # Rwhod files (optional).
/var/tmp # Temporary files preserved between system reboots.
/var/yp # Network Information Service (NIS) database files (optional). Variable data for the Network Information Service (NIS), formerly known as the Sun Yellow Pages (YP), must be placed in this directory.

-----

System and Software installation:

Install a software package using APT (Advanced Package Tool):

apt-get install base-config
apt-setup

Set APT sources (repositories) manually:
jed /etc/apt/sources.list # use vi /etc/apt/sources.list if jed (or other editor) hasn't been installed yet. vi should always be present on Debian Linux
You can also use edit the sources file with
apt edit-sources # Edit APT sources file using the default editor or the one set in export EDITOR=jed ; export VISUAL=jed ; # (or another editor. Remember to install it before to set it as a system default)

deb http://http.us.debian.org/debian stable main contrib non-free
deb-src http://http.us.debian.org/debian stable main contrib non-free

deb http://security.debian.org stable/updates main contrib non-free
deb-src http://security.debian.org stable/updates main contrib non-free

You can use a local repository instead of the US Debian one, for example debian.fastweb.it instead of http.us.debian.org
Use unstable instead of stable if you want bleeding edge technology. Not very recommended in production environments.

To keep an old version up to date:
deb http://archive.debian.org/debian/ DEBIAN_VERSION main non-free contrib
deb-src http://archive.debian.org/debian/ DEBIAN_VERSION main non-free contrib

deb http://archive.debian.org/debian-security/ DEBIAN_VERSION/updates main non-free contrib
deb-src http://archive.debian.org/debian-security/ DEBIAN_VERSION/updates main non-free contrib

Update the list of packages:
# After adding a new apt-get source, or before installing new packages, enter this command
apt-get update
apt update
apt update ; apt list --upgradable ; # check for upgradable packages and list them if found
apt update 2>/dev/null | grep packages # check for upgradable packages and show the count of them if found (output is redirected to /dev/null to remove the message: WARNING: apt does not have a stable CLI interface. Use with caution in scripts.)
apt update 2>/dev/null | tail --lines=1 # check for upgradable packages and show the count of them if found (output is redirected to /dev/null to remove the message: WARNING: apt does not have a stable CLI interface. Use with caution in scripts.)

Should a "Segmentation fault" error occur when updating the list of packages, you can fix it with these commands:
rm -r /var/lib/apt/lists/*
rm /var/cache/apt/*.bin # will delete pkgcache.bin and srcpkgcache.bin

Show upgradable packages without upgrading them (after an update)
apt-get -u upgrade --assume-no
apt list --upgradable

Upgrade installed packages:
apt-get upgrade # upgrade already installed packages but don't remove anything or install anything new
apt upgrade

Upgrade a distribution:
# This is just a basic procedure to upgrade a Debian release. Check https://www.debian.org/releases/stable/amd64/release-notes/ch-upgrading.html for mode detailed information.
apt-get update
apt-get upgrade
aptitude search '~i(!~ODebian)' # List third-party packages that may be lost after a dist-upgrade
dpkg --audit # List Half-Installed or Failed-Config packages
apt-get autoremove # remove packages installed as dependency of already uninstalled packages
jed /etc/apt/sources.list # Set APT sources (repositories) for the new Debian release
apt-get update
apt-get upgrade
apt-get dist-upgrade # upgrade everything removing/adding packages as required to resolve dependencies
apt-get autoremove
reboot

Make sure the system is up to date after a fresh installation:
dpkg --audit
apt-get update
apt-get upgrade
apt-get dist-upgrade
dpkg --audit

If the system is installed on a VMWare machine, you'd better install VMware Tools or Open VM Tools. The package for VMware Tools can be downloaded from VMWare website, while its open counterpart (Open VM Tools) can be installed via apt-get:
apt-get install open-vm-tools

Prevent a package to be upgraded:
apt-get install wajig
wajig hold PACKAGE # the specified package will no longer be upgraded (for example with an apt-get upgrade)
wajig unhold PACKAGE # the package will be upgraded again
wajig hold mysql-server-5.0 # a common problem with Kernel 2.4 is that latest releases of MySQL server 5 are not compatible with such kernel version, interfering with the upgrade process

Search for a package:
apt-cache search TEXT # search for a package containing the specified text or keyword in its name or description

List installed packages:
dpkg-query -l # List all installed packages
dpkg --list # List all installed packages
dpkg --get-selections # List all installed packages
dpkg -l # List all installed packages with version and description
dpkg --get-selections | wc --lines # Count installed packages
grep install /var/log/dpkg.log | tail -10 # Last 10 installed packages
dpkg --listfiles PACKAGE # List all files "owned" by PACKAGE
cat /var/log/dpkg.log | grep "status installed " # Show last installed packages
cat /var/log/apt/history.log # Show last apt commands (installed and upgraded packages)
aptitude search '~i(!~ODebian)' # List third-party packages only

Check if a package is installed:
dpkg -s PACKAGE
dpkg-query -l PACKAGE
dpkg-query -W -f='${Status} ${Version}\n' PACKAGE # If the package is installed, show status and version 
dpkg-query -l "TEXT*" # Show all installed packages starting with TEXT
dpkg-query -l "*TEXT" # Show all installed packages ending with TEXT
dpkg-query -l "*TEXT*" # Show all installed packages containing TEXT
cat /var/log/dpkg.log | grep "status installed PACKAGE" # Search for a specific package among the last installed ones

Install a package:
apt-get install PACKAGE # get and install a PACKAGE from a repository specified in /etc/apt/sources.list
dpkg -i PACKAGE.deb # manually install a package available locally (already downloaded)
apt-get install --reinstall PACKAGE # reinstall a PACKAGE

Show information about a package and all its dependencies
apt-cache show PACKAGE
aptitude show PACKAGE # similar to apt-cache show PACKAGE but shows extra information
apt-cache showpkg PACKAGE # show dependencies and reverse dependencies

Verify if all packages are correctly installed and configured:
dpkg --audit # List Half-Installed or Failed-Config packages. If everything is fine, it doesn't return anything.

Check for broken dependencies:
apt-get check

Uninstall a package:
apt-get remove PACKAGE
aptitude purge PACKAGE # remove PACKAGE and its configuration files

Show a random non installed package:
apt-cache search ~g | while read ; do echo "$RANDOM $REPLY" ; done | sort -n | head -1 | cut -d' ' -f2-

Store/Restore Packages:
dpkg --get-selections > installedpackages.txt # Store the list of installed packages into the file installedpackages.txt
dpkg --set-selections < installedpackages.txt # Restore the list of installed packages from the file installedpackages.txt
apt-get install `cat installedpackages.txt` # Restore the list of installed packages from the file installedpackages.txt

Clean up packages:
apt-get autoremove # remove packages installed as dependency of already uninstalled packages
apt-get clean # delete all downloaded packages (that is, all files stored in /var/cache/apt/archives/ and /var/cache/apt/archives/partial/ , except for lock files). If you want to reinstall one of such packages, apt-get will have to download them again.
apt-get autoclean # only delete packages which is no longer possible to download (obsolete).

If you get one of the following error messages:

E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)
E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?

dpkg: status database area is locked by another process

or anyway if something goes wrong when upgrading packages, you may fix it this way:

rm /var/lib/apt/lists/lock
rm /var/lib/dpkg/lock
rm /var/cache/apt/archives/lock
dpkg --configure -a # to fix interrupted configure (may take a while)
apt-get -f install # to fix broken dependencies

If you still get error messages there are probably running processes working with installation files, you either have to find and kill them all or reboot before than attempting to recover the upgrade process again.

After upgrading the Linux kernel, it's better to perform a update-grub and then reboot the system.

Debian Linux (Sarge, Kernel 2.6): workaround for
E: This installation run will require temporarily removing the essential package e2fsprogs due to a Conflicts/Pre-Depends loop. This is often bad, but if you really want to do it, activate the APT::Force-LoopBreak option.
apt-get remove e2fsprogs
apt-get install e2fsprogs sysvinit initscripts
upgrade glibc when requested.
# note that this is a workaround, it should be fixed with apt-get dist-upgrade

Note: when the Debian stable changes, if you don't upgrade your system accordingly, you'll have to modify the /etc/apt/sources.list file to specify you are using an old version of Debian (that is, you are no longer using the "current stable" version of Debian). For example, if you are using Sarge, and Debian Etch is released as stable, and you are not upgrading to Etch, you have to change all references to stable as sarge in your source.list file.
Which means, you have to change "deb http://debian.example.com/debian stable main contrib non-free" as "deb http://debian.example.com/debian sarge main contrib non-free".
If you don't make this change, when you'll try to install new packages, you'll receive the following message:
The following packages will be REMOVED:
  base-config initrd-tools kernel-image-2.4.27-2-386
This is an example, the kernel image can be different in your case, however you'll not be able to install the package and it may damage your system. In fact, if you procede, you will be warned by this message:
You are running a kernel (version 2.4.27-2-386) and attempting to remove the same version.
This is a potentially disastrous action. Not only will /boot/vmlinuz-2.4.27-2-386 be removed, making it impossible to boot it, (you will have to take action to change your boot loader to boot a new kernel), it will also remove all modules under the directory /lib/modules/2.4.27-2-386.
Just having a copy of the kernel image is not enough, you will have to replace the modules too.
I repeat, this is very dangerous. If at all in doubt, answer no. If you know exactly what you are doing, and are prepared to hose your system, then answer Yes.

If you get an error like
'Template parse error near `Description-sr@latin.UTF-8', in stanza #X of /PATH/FILE.templates
then edit the specified file removing the lines containing the given text (and, if present, the following line with the description) from /PATH/FILE.templates

If you get an error with locales, like
locale: Cannot set LC_CTYPE to default locale
you might be able to fix it with
apt-get install locales

If you get a message ending with the following request after a power outage:

Give root password for maintenance
(or type Control-D to continue):

enter the root password
then check the file system of the faulty device (read the whole message to identify the faulty driver, generally is /dev/hda0 on a single hard drive system or /dev/md0 on a RAID system):
fsck -f FAULTY_DEVICE
reply yes (y) to all questions. Note that if it asks to connect to lost+found more than once some of your files might be seriously damaged. However you can also use fsck -f -y FAULTY_DEVICE to reply automatically "yes" to all questions.
then
reboot

If you can't perform a simple operation that always used to work (for example, copying a file) make sure the disk is not full (check with df).

Set keyboard layout
# Enter this command to localize the keyboard for your country
apt-get install console-common

Set system default editor used for apt edit-sources and crontab -e (parameters in the following commands are case sensitive). If not set the system will use its default editor, tipically "vi". Remember to install the editor first, before to set it as a system default.
export EDITOR=jed
export VISUAL=jed

You can also use the apt command instead of apt-get for most common tasks:
apt install # apt-get install
apt remove # apt-get remove
apt purge # apt-get purge
apt update # apt-get update
apt upgrade # apt-get upgrade
apt autoremove # apt-get autoremove
apt full-upgrade # apt-get dist-upgrade
apt search # apt-cache search
apt show # apt-cache show

-----

Install a software package from the source (compile a package source):

To compile packages you need to install a C compiler and other tools:
apt-get install build-essential
If a script uses OpenSSL to handle encryption, you may also need to install OpenSSL headers:
apt-get install libssl-dev

Packages in form of source code are generally stored into .tar.gz (.tgz) or .tar.bz2 (.tbz) files.

You may need to download them from the Internet:
wget http://www.example.com/source/package.tgz

Once you have the package (from the Internet, a CD, or otherwise) in your work directory, you'll have to decompress it:

tar -xjvf package.tbz # extract all files from a TAR+BZIP2 compressed archive
or
tar -xzvf package.tgz # extract all files from a TAR+GZIP compressed archive

it should decompress inside a new directory. Move inside this new directory:
cd PACKAGE_SOURCE_DIRECTORY_NAME

and execute these commands:
./configure
make
su ROOT_PASSWORD # (if you're not logged as root)
make install

The executable file should be created inside /usr/local/bin ,  and you should be able to invoke it typing its name in the console and then hitting ENTER.

To uninstall a package, you have to move inside the directory containing the source (if you kept it):
cd PACKAGE_SOURCE_DIRECTORY_NAME
and uninstall it:
make uninstall
If you haven't kept the source, you'll have to locate all files (try locate PACKAGE_NAME) and delete them manually.

-----

Change root password without knowing the existing one
You need physical access to the system.

Add init=/bin/bash to boot parameters
mount -o remount,rw # mount file system
passwd # set new password. WARNING: sudo passwd will change root password, regardless of the current account.

Recover a deleted password file:
# Debian makes regular backups of the password file in /var/backups/
cp /var/backups/passwd.bak /etc/passwd
chmod 644 /etc/passwd

-----

Writing text files for Linux using an editor running on another Operating System:
When you edit a system file, a shell script, a fortune cookie text file, or another file meant to be processed by a Linux shell command or application, make sure that the lines are divided UNIX style (LF only, x0A; not CR/LF, x0D0A as in Windows), otherwise the line breaks will not be recognized as such and the whole file will be treated as a single line.

-----

Install SSH:
apt-get install ssh

Get SSH version:
ssh -V

Start a SSH connection (port 22):
ssh HOST

Configure SSH:
jed /etc/ssh/sshd_config
# If your client can't authenticate to your server, it might be sending the password as a cleartext and your server is not accepting it. In this case you have to enable tunnelled clear text passwords setting PasswordAuthentication yes

# Make sure SSH server is not using the outdated Protocol 1. This line should be present in the configuration file instead:
Protocol 2

# Set SSH timeout (in seconds). A logged user will be logged out after idling past specified time.
ClientAliveInterval 600
ClientAliveCountMax 0

/etc/init.d/ssh restart # Restart SSH to make changes take effect. However note that they will apply to next SSH session.


Start a Telnet (unencrypted) connection:
telnet HOST # start a telnet connection on HOST at default port 23
telnet HOST:PORT # start a telnet connection on HOST at the specified PORT

-----

Bash shells:

Show full name of the terminal type:
tput longname

Show BASH version:
echo $BASH_VERSION

Command Line Navigation:
CURSOR LEFT / CURSOR RIGHT # move cursor one character left or right on command line
LEFT ALT+[B/F] # move cursor to previous / next word on command line
CURSOR UP / CURSOR DOWN # move to previous / next command in history
CTRL+R # search for a command in History: hit CTRL+R, type the string you want to search, hit CTRL+R again to find further matches

Switch bash shells:
LEFT ALT+[F1-F6]

Cycle between bash shells:
LEFT ALT+[CURSOR LEFT / CURSOR RIGHT]

Show current terminal:
tty

Terminal line settings:
stty -a # Print all terminal line settings in human readable format
stty -g # Print all terminal line settings in stty readable format

Reset a "corrupted" shell:
reset # It will also reset the terminal terminal window size to 80 x 36 characters

Clear shell terminal window:
clear # Clear screen (if your terminal has a buffer, you can still scroll back to see the text previously on screen)
printf "\033c" # clear the screen by scrolling all the text above the visible area (same as CTRL+L)

Set Terminal Window Size:
echo -e "\e[8;LINES;COLUMNSt" # resize the terminal window to LINES x COLUMNS . For example, tlins=40; tcols=160; echo -e "\e[8;$tlins;${tcols}t"; (or echo -e "\e[8;40;160t" ) resizes the window to 160 x 40 characters.

Get Terminal Window Size:
tput lines # return the height in characters (number of lines) of the terminal window
tput cols # return the width in characters (number of columns) of the terminal window
echo "$(tput cols) x $(tput lines)"; # show terminal window size
# The environmental variables $LINES and $COLUMNS contain the values of lines and colums of the terminal window, but their value is not preserved within scripts, so that you'll have to add this line to your script assign them the expected values: COLUMNS="$(tput cols)"; LINES="$(tput lines)";

Open a new Bash shell:
bash

Duplicate a terminal:
apt-get install conspy
conspy N # N can range from 1 to 6, then execute another conspy N with the same number on another bash shell

Show current command line:
cat /proc/self/cmdline # typical output: cat/proc/self/cmdline

Exit from a non login shell:
exit

Show last exit code (exit status):
echo $? # Return the exit status of last command (0: Success (no errors, if the last performed command was executed properly with a clean exit), 1 (or different than 0): Failure). Note that if you execute the command twice, the second time you'll get the exit status of the first execution of echo #?

Exit Codes with special meanings (Reserved Exit Codes):
Exit Code Number: Meaning / Example / Comments:
0: No errors / echo "Hello World" / Command executed successfully. Note that quitting a command with CTRL+D will also return 0.
1: Catchall for general errors / let "var1 = 1/0" / Miscellaneous errors, such as "divide by zero"
2: Misuse of shell builtins / empty_function() {} / Missing keyword or command, or permission problem (and diff return code on a failed binary file comparison).
126: Command invoked cannot execute / /dev/null / Permission problem or command is not an executable
127: "command not found" / illegal_command / Possible problem with $PATH or a typo
128: Invalid argument to exit / exit 3.14159 / exit takes only integer args in the range 0 - 255 (see first footnote)
128+n: Fatal error signal "n" / kill -9 $PPID of script / kill -9 returns 137 (128 + 9) ; CTRL+C returns 130 (128 + 2), CTRL+Z returns 148 (128 + 20).
255*: Exit status out of range / exit -1 / exit takes only integer args in the range 0 - 255

Show environment variables:
printenv # List all environment variables
export -p # List all exported (global scope) variables

Change prompt:
export PS1="[\t] \u@\h:\w\\\$ " # show time in 24 hours format between square brackets, the logged user name, the host name, the current path, and the root status ("#" if root, "$" otherwise)
export PS1="\[\e[1;32m\][\t] \u@\h:\w\\\$ \[\e[0m\] " # same prompt, but colored in light green
export PS1="\[\e[1;32m\][\$(date +%a\ %d%b%Y\ %H:%M:%S)] \u@\h:\w\\\$\[\e[0m\] " # full date and time between square brackets, the logged user name, the host name, the current path, and the root status ("#" if root, "$" otherwise), colored in light green. Note that the shell command date is used, invoked with \$() .  If $() (without the leading backslash) were used, the date would be generated only the first time and then stored into the variable and wouldn't be updated every time the prompt is shown. Note: if you want to set Windows Command Prompt to match your Linux Shell Prompt you have to set the %prompt% environment variable. For example: set prompt=[$d$s$t]$s%username%@%computername%:$p$+$g$s
export PS1="\[\e[1;32m\][\$(date +%Y-%m-%d\ %H:%M:%S\ %a)] \u@\h:\w\\\$\[\e[0m\] " # full date and time (in a sortable format) between square brackets, the logged user name, the host name, the current path, and the root status ("#" if root, "$" otherwise), colored in light green.
echo $PS1 # See current prompt

Edit the script executed at the shell start-up (user login shell):
# Custom Prompt and Aliases can be set here
jed ~/.bash_profile

Sample bash profile script:
(see https://labs.geody.com/systatus/ )
---
#!/bin/bash
# ~/.bash_profile

# Systatus BASH Shell Start-up
# r2020-12-27 fr2016-10-18
# by Valerio Capello - https://labs.geody.com/ - License: GPL v3.0

# Config
tshilon="\e[0;33m"; tshilof="\e[0m";
tsalerton="\e[0;31m"; tsalertof="\e[0m";

# Set Terminal Window Size
# tlins=40; tcols=160; echo -e "\e[8;$tlins;${tcols}t";

# Get Terminal Window Size
COLUMNS="$(tput cols)"; LINES="$(tput lines)";

# Prompt
export PS1="\[\e[1;32m\][\$(date +%Y-%m-%d\ %H:%M:%S\ %a)] \u@\h:\w\\\$\[\e[0m\] "

# History Date Format
export HISTTIMEFORMAT="%F %T  "

# Aliases
alias l="ls -laF --group-directories-first --color=auto"
alias d="ls -aF --group-directories-first --color=auto"

# Message
echo; clear
date "+%a %d %b %Y %H:%M:%S %Z (UTC%:z)"
echo -n "Hello "; echo -ne "$tshilon"; echo -n "$(whoami)"; echo -ne "$tshilof";
if [[ "$SSH_CONNECTION" ]]; then
echo -n " ("; echo -ne "$tshilon"; echo -n "`echo $SSH_CLIENT | awk '{print $1}'`"; echo -ne "$tshilof)";
fi
echo -n ", ";
echo -n "welcome to "; echo -ne "$tshilon"; echo -n "$(hostname)"; echo -ne "$tshilof";
echo -n " ("; echo -ne "$tshilon"; echo -n "$(hostname -i)"; echo -ne "$tshilof)";
echo ".";
echo -n "Machine ID: "; echo -n "$(cat /etc/machine-id) ";
echo -n "Boot ID: "; echo -n "$(cat /proc/sys/kernel/random/boot_id) ";
echo -n "Session ID: "; echo "$(cat /proc/self/sessionid)";
echo -n "You are ";
if [[ -n "$SSH_CONNECTION" ]]; then
echo -n "connected remotely via SSH";
elif [[ "${DISPLAY%%:0*}" != "" ]]; then
echo -n "connected remotely "; echo -ne "$tsalerton"; echo -n "NOT"; echo -ne "$tsalertof"; echo " via SSH (which is Bad)";
else
echo -n "connected locally";
fi
utty="$(tty)";
if [[ $utty == "/dev/"* ]]; then
utty=$(cut -c 6- <<< $utty);
fi
echo -n " on $utty";
echo ". Your Terminal Window Size is $COLUMNS x $LINES"
if [[ $EUID -eq 0 ]]; then
echo -ne "$tsalerton"; echo -n "You have ROOT superpowers!"; echo -e "$tsalertof";
fi
echo

# Machine
# echo -n "Vendor: "; echo "$(cat /sys/class/dmi/id/sys_vendor)";
echo -n "Machine: "; echo "$(cat /sys/class/dmi/id/product_name)";
# echo -n "Machine Type: "; echo "$MACHTYPE";
# echo -n "Board: "; echo "$(cat /sys/class/dmi/id/board_vendor) $(cat /sys/class/dmi/id/board_name)";
# echo -n "BIOS: "; echo "$(cat /sys/class/dmi/id/bios_vendor) $(cat /sys/class/dmi/id/bios_version) $(cat /sys/class/dmi/id/bios_date)";
echo -n "CPU: "; echo -n "$(grep 'model name' /proc/cpuinfo | head -1). ";
echo -n "Cores: "; grep -c 'processor' /proc/cpuinfo

echo -n "CPU average load: "; uptime | awk -F'[a-z]:' '{print $2}' | xargs | awk '{print "1 m: "$1" 5 m: "$2" 15 m: "$3}';
free -h | xargs | awk '{print "Memory: Size: "$8" Used: "$9" Free: "$10" Avail: "$13}';
df / -Th | xargs | awk '{print "FS: "$9" Type: "$10" Size: "$11" Used: "$12" ("$14") Avail: "$13" ("(100-$14)"%)"}';
echo "Last Boot / Uptime: `uptime -s` (`uptime -p`)";
echo

# Software version
lsb_release -ds
uname -a
echo "Bash version: $BASH_VERSION"
# Webserver version
echo -n "$(/usr/sbin/apache2 -v|head --lines=1) "; echo "$(/usr/sbin/apache2 -v|tail --lines=1)";
openssl version -v
php -v|head --lines=1
mysql -V
# echo
echo -n "Installed Packages: "; dpkg --get-selections | wc -l;
# echo "Last installed packages:"; grep install /var/log/dpkg.log | tail --lines=5;
echo

# Users
echo "Last logged users:"; last -n 5 | sed '/^$/d'
echo; echo "Currently logged users:"; who
echo; echo -n "Current user: "; id
echo

# Security
# Shellshock vulnerability check (reports to root only)
if [[ $EUID -eq 0 ]]; then
env x='() { :;}; echo Bash vulnerable to Shellshock' bash -c 'echo -n'
fi
---

You may want to replicate your bash profile script as an executable script (called for example status.sh ) containing information provided in the sections starting from Message and below (you'd better change echo -n "welcome to "; to echo -n "this is ";).

Customize the MOTD (Message Of The Day), message shown after logging in, before than starting the shell:
jed /etc/motd # edit the MOTD

Customize the message shown before than the log in request:
jed /etc/issue # edit the welcome message

Customize the command line:

jed ~/.inputrc

Sample .inputrc script:
---
# ~/.inputrc

# .inputrc
# r2020-11-12 fr2020-11-12
# by Valerio Capello - https://labs.geody.com/ - License: GPL v3.0

# Enable autocomplete
set disable-completion off

# Show autocomplete results in pages
set page-completions on

# Show file types for completed results (like ls -F )
set visible-stats on

# If there are too many completion results, ask to show them all
set completion-query-items 100

# Bell
# set bell-style audible

# Go to next autocomplete result with Shift-Tab
"\e[Z": menu-complete

# History search (Arrows Up and Down)
"\e[A": history-search-backward
"\e[B": history-search-forward
---

Description of a command:
whatis COMMAND

Manual of a command:
man COMMAND
apropos KEYWORD # list all manual pages related to the KEYWORD
apropos . | shuf -n 1 | awk '{print$1}' | xargs man # show a random manual page

Return the path of a command:
which COMMAND

Return the type of a command:
# Bash builtins (internal commands): alias, bg, bind, break, builtin, case, cd, command, compgen, complete, continue, declare, dirs, disown, echo, enable, eval, exec, exit, export, fc, fg, getopts, hash, help, history, if, jobs, kill, let, local, logout, popd, printf, pushd, pwd, read, readonly, return, set, shift, shopt, source, suspend, test, times, trap, type, typeset, ulimit, umask, unalias, unset, until, wait, while.
type COMMAND # Get the command type. If it's a file or an alias, it also returns the path to COMMAND.
type -t COMMAND # Get the command type in a single keyword: alias, keyword, function, builtin, file.


List internal commands:
help
help COMMAND # show information about an internal command

Identify the type of a file:
file PATH/FILE
file -z PATH/FILE.zip # Attempts to check the file type of files contained inside a compressed file

Alias
alias l="ls -laF --color=auto" # create an alias: makes "l" an alias for "ls -laF --color=auto"
alias {helloworld,hellow}="echo 'Hello World'" # define multiple aliases
alias # list all existing aliases
type ALIAS # show the commands associated to the ALIAS
unalias ALIAS # remove an alias (if the alias was defined together with other multiple aliases, the other aliases will be unaffected and remain active)

Symlinks
# Symlinks / Sym links / Symbolic links / Soft links
ln -s /PATH/TARGET_FILE /PATH/SYMLINK # Create a SYMLINK to TARGET_FILE . Permissions for the file will be the ones for TARGET_FILE, regardless of the ones for SYMLINK. Use rm to remove the SYMLINK (TARGET_FILE will not be affected)
find . -type l ! -exec test -e {} \; -print # Find broken symlinks

Repeat last command:
!!

Start a comment (ignore following text):
#

Do nothing, successfully:
true

Do nothing, unsuccessfully:
false

Execute a command multiple times at a given time interval
watch -n SECONDS COMMAND
watch -n 1 date # Execute the command date every second

Conditional execution:
if [[ 1 == 1 ]]; then echo "true"; fi
if [[ 1 == 1 ]]; then echo "true"; else echo "false"; fi

Execute a random command between two:
((RANDOM%2 == 0)) && echo "Yes" || echo "No"
if [[ $((RANDOM % 2)) == 0 ]]; then echo "Yes"; else echo "No"; fi

Find the time required to perform a command line:
# It may useful to compare commands, or parameters of the same command, and see which one is faster.
# The "real" value is the actual elapsed time which is the data you most likely need, while The "user" value shows the time spent by the code in user mode, and the "sys" value shows the time spent by the code in kernel mode.
time (ls) # Shows the time needed to list the current directory

Process priority:
nice -n0 COMMAND # execute a command with default priority (0)
nice -n20 COMMAND # execute a command with the lowest priority (20)
nice -n-20 COMMAND # execute a command with the highest priority (-20). Only root can assign negative (high) priority
renice 10 PID # reassign priority to an existing process

List running processes and their PIDs:
echo $$ # show the PID of the current shell (or script, if invoked from a script)
ps # show all processes with a tty in current shell
ps -a # show all processes with a tty in all shells
ps -A # show all processes
ps -A|grep "STRING" # show all processes that matches STRING
ps -A|grep " pts/." # show all processes for remote terminal devices
pgrep -f "STRING" # show the PID for all processes that matches STRING
pgrep -fo "STRING" # show the PID for the least recent (oldest) process that matches STRING
pgrep -fn "STRING" # show the PID for the most recent (newest) process that matches STRING
ps auxf | sort -nr -k 4 # show processes sorted by memory usage
ps auxf | sort -nr -k 4 | head --lines=10 # show processes sorted by memory usage (first 10 only)
ps auxf | sort -nr -k 3 # show processes sorted by CPU usage
ps auxf | sort -nr -k 3 | head --lines=10 # show processes sorted by CPU usage (first 10 only)

Show running processes and system load:
top # Show running processes and system load. Press "q" to quit
top -o %CPU # Show running processes and system load, sorted by CPU usage. Press "q" to quit
top -o %MEM # Show running processes and system load, sorted by Memory usage. Press "q" to quit
top -b -n1 | grep COMMAND | awk '{ SUM += $10} END { print SUM }' # Show the sum of CPU usage grouped by COMMAND (for example: apache2 , mysqld , fail2ban-server )
top -b -n1 | grep COMMAND | awk '{ SUM += $9} END { print SUM }' # Show the sum of memory usage grouped by COMMAND (for example: apache2 , mysqld , fail2ban-server )

Show running processes accessing the disk or doing other Input/Output activity:
apt-get install iotop
iotop --only # press "q" to quit

Services:
service --status-all # Show the status of all services (+ for active, - for inactive)
service SERVICE status | grep -m 1 'Active:' | cut -d ":" -f2- | sed 's/^ *//g' # Show the status of a SERVICE, if it's active (running) or it's inactive. Some popular services are: ssh, apache2, tor, mysql, sendmail, fail2ban, ufw

Jobs:
CTRL+Z # Stop (Pause) current program and send it to the background. Returns Signal 20 (Exit Code 148)
jobs # List all jpbs (programs) in the background with their job id
fg JOB_ID # Restart the specified job (program) and restore it to the foreground

# If you get the message "There are stopped jobs." when logging out ('logout' from the console), run 'jobs' to see the list of jobs in the background. Then you have to options: the cleanest one is to bring the jobs to the foreground ('fg JOB_ID') and then close them as required by that specific job, or list all the running processes ('ps'), look for the one corresponding to the job stuck in the background and kill it ('kill -9 PID').  

Processes:
# LSBInitScripts require to support the following actions (as per Debian Policy, chapter 9.3.2 Writing the scripts): start, stop, restart, try-restart (optional), reload (optional), force-reload, status (optional but encouraged, see Debian Bug #291148 2005-01-19), version (not required, but useful).
/etc/init.d/COMMAND start # Start the service. Note: if the process is already running it should return success, not start another copy.
/etc/init.d/COMMAND stop # Stop the service.
/etc/init.d/COMMAND restart # Restart the service if it's already running, otherwise just report success (optional).
/etc/init.d/COMMAND try-restart # Restart the service if it's already running, otherwise just report success (optional).
/etc/init.d/COMMAND reload # Cause the configuration of the service to be reloaded without actually stopping and restarting the service (optional).
/etc/init.d/COMMAND force-reload # Cause the configuration to be reloaded if the service supports this, otherwise restart the service.
/etc/init.d/COMMAND status # Report the current status of the service (optional but encouraged, see Debian Bug #291148 2005-01-19)
/etc/init.d/COMMAND version # Return version number and / or release date (currently not required for LSB compliance)

Show which user launched a process:
fuser PID

Terminate a process:
kill -3 PID # quit a process
kill -15 PID # term a process
kill -9 PID # kill a process (most effective to quit an unresponsive process)

Terminate all processes with a matching name:
killall NAME
killall -I NAME # ignore cases
killall -i NAME # ask for confirmation before to kill

Show information about all loaded module:
lsmod

Show Boot parameters for last (current) boot:
cat /proc/cmdline

Show open files:
apt-get install lsof
lsof # Show all open files
lsof PATH # Show all open files within PATH
lsof / | awk '{ if($7 > 1048576) print $7/1048576 "MB" " " $9 " " $1 }' | sort -n -u | tail --lines=10 # show the 10 largest open files with their size in MB and the processes that keep them open

Test to check if your Bash shell is vulnerable to Shellshock (you should only see a dot if your system is not vulnerable)
env x='() { :;}; echo Bash vulnerable to Shellshock' bash -c 'echo .'

-----

Information about the vendor:
cat /sys/class/dmi/id/sys_vendor # System Vendor

Information about the machine (virtual or physical):
cat /sys/class/dmi/id/product_name # Product Name
cat /sys/class/dmi/id/product_uuid # Product UUID
cat /sys/class/dmi/id/product_serial # Product Serial
cat /sys/class/dmi/id/product_version # Product Version (if any)

cat /sys/class/dmi/id/chassis_vendor # Chassis Vendor (if there is an enclosure)
cat /sys/class/dmi/id/chassis_type # Chassis Type
cat /sys/class/dmi/id/chassis_version # Chassis Version (if any)
cat /sys/class/dmi/id/chassis_serial # Chassis Serial (if any)
cat /sys/class/dmi/id/chassis_asset_tag # Chassis Asset Tag (if any)

cat /sys/class/dmi/id/board_vendor # Board Vendor
cat /sys/class/dmi/id/board_name # Board Name
cat /sys/class/dmi/id/board_version # Board Version (if any)
cat /sys/class/dmi/id/board_serial # Board Serial (if any)
cat /sys/class/dmi/id/board_asset_tag # Board Asset Tag (if any)

cat /sys/class/dmi/id/bios_vendor # BIOS Vendor
cat /sys/class/dmi/id/bios_version # BIOS Version
cat /sys/class/dmi/id/bios_date # BIOS Date

Show Machine's Unique ID:
cat /etc/machine-id # show the unique machine ID of the local system that is set during installation. The machine ID is a single newline-terminated, hexadecimal, 32-character, lowercase machine ID string. When decoded from hexadecimal, this corresponds with a 16-byte/128-bit string (from the manual). Also in /var/lib/dbus/machine-id

Boot ID:
cat /proc/sys/kernel/random/boot_id # show the random ID regenerated at each boot

Session ID
cat /proc/self/sessionid # show the session ID, which is unique for each login

Random UUID:
cat /proc/sys/kernel/random/uuid # generate a random UUID, different every time is invoked

Show system name:
hostname
cat /proc/sys/kernel/hostname

Show system's IP address:
hostname -i # Local IP
ifconfig | grep -Eo 'inet (addr:)?([0-9]*\.){3}[0-9]*' | grep -Eo '([0-9]*\.){3}[0-9]*' | grep -v '127.0.0.1' # Global IPv4
ifconfig | sed -En 's/127.0.0.1//;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\2/p' # Global IPv4

Show client's IP address in a SSH connection:
echo $SSH_CLIENT | awk '{print $1}'
echo $SSH_CONNECTION | awk '{print $1}' # Note that $SSH_CONNECTION (not processed with awk) also contains the server's IP address
echo "$(whoami) (`echo $SSH_CLIENT | awk '{print $1}'`) @ $(hostname) ($(hostname -i))"; # client (client IP) @ host (host IP)

Change system name as NEW_NAME:
hostname NEW_NAME
jed /etc/hostname # change it as NEW_NAME
jed /etc/hosts # you'd better also add NEW_NAME as an alias hostname for 127.0.0.1 (localhost)

Show system information:
uname -a
uname -r # show only kernel version
cat /proc/sys/kernel/osrelease # kernel version
cat /etc/issue # Welcome message: contains Linux version (unless otherwise edited by the user)
cat /etc/os-release # Get Distribution Name and Version
awk -F'=' '/^ID=/ {print $2}' /etc/os-release # Get Distribution ID with quotes
awk -F'=' '/^ID=/ {print $2}' /etc/os-release | sed -e 's/"//g' # Get Distribution ID without quotes
awk -F'=' '/^ID=/ {print $2}' /etc/os-release | tr -d '"' # Get Distribution ID without quotes
awk -F'=' '/^NAME=/ {print $2}' /etc/os-release # Get Distribution Name with quotes
awk -F'=' '/^NAME=/ {print $2}' /etc/os-release | sed -e 's/"//g' # Get Distribution Name without quotes
awk -F'=' '/^NAME=/ {print $2}' /etc/os-release | tr -d '"' # Get Distribution Name without quotes
awk -F'=' '/^VERSION=/ {print $2}' /etc/os-release # Get Distribution Version with quotes
awk -F'=' '/^VERSION=/ {print $2}' /etc/os-release | sed -e 's/"//g' # Get Distribution Version without quotes
awk -F'=' '/^VERSION=/ {print $2}' /etc/os-release | tr -d '"' # Get Distribution Version without quotes
awk -F'=' '/^VERSION_ID=/ {print $2}' /etc/os-release # Get Distribution Version ID with quotes
awk -F'=' '/^VERSION_ID=/ {print $2}' /etc/os-release | sed -e 's/"//g' # Get Distribution Version ID without quotes
awk -F'=' '/^VERSION_ID=/ {print $2}' /etc/os-release | tr -d '"' # Get Distribution Version ID without quotes
awk -F'=' '/^PRETTY_NAME=/ {print $2}' /etc/os-release # Get Distribution Pretty Name (Name and Version) with quotes
awk -F'=' '/^PRETTY_NAME=/ {print $2}' /etc/os-release | sed -e 's/"//g' # Get Distribution Pretty Name (Name and Version) without quotes
awk -F'=' '/^PRETTY_NAME=/ {print $2}' /etc/os-release | tr -d '"' # Get Distribution Pretty Name (Name and Version) without quotes
lsb_release -a # Get Distribution Name and Version (more detailed than /etc/os-release )
lsb_release --description|awk '{$1 = ""; print $0}'|sed 's/^ //g' # Get Distribution Pretty Name (more detailed than /etc/os-release )
lsb_release -ds # Get Distribution Pretty Name (more detailed than /etc/os-release )
cat /etc/debian_version # Debian version

Show the kernel ring buffer (to print out the bootup messages):
dmesg

Show last 20 lines of the system messages file:
tail --lines=20 /var/log/messages

System Up Marker:
By default, on most Debian based distributions up to Debian Etch, every 20 minutes syslog marks the message file with "-- MARK --" to log that the system is up and running.

You can modify this behavior by editing the syslog startup file:
jed /etc/init.d/sysklogd

Locate the line that defines the variable SYSLOGD (generally, SYSLOGD="" ). If it's missing you can add it at the beginning of the configuration file, where other variables are defined.
and use the switch -m to set the delay between marks in minutes, or set it to 0 to disable the notification.
Example: SYSLOGD="-m 60" marks the message file every hour, SYSLOGD="-m 0" disable the notification.

Restart syslog to make the change effective:
/etc/init.d/sysklogd restart

After Etch, the time marker is disabled by default. You can enable it editing the rsyslog configuration file and enabling the immark module (which is commented out):
jed /etc/rsyslog.conf
module(load="immark")  # provides --MARK-- message capability

You should also be able to set the delay between marks setting the RSYSLOGD_OPTIONS variable in rsyslog, unfortunately RSYSLOGD_OPTIONS is ignored:
jed /etc/init.d/rsyslog
RSYSLOGD_OPTIONS="-m 60" marks the message file every hour

Restart syslog to make the change effective:
/etc/init.d/rsyslog restart

Check last marks in syslog and messages:
# Time marks should coincide in both syslog and messages
grep "MARK" /var/log/syslog | tail --lines=10 ; echo ; grep "MARK" /var/log/messages | tail --lines=10

# Check last times rsyslog was stopped
grep "] exiting" /var/log/messages | tail --lines=10 # You could also grep "] exiting" /var/log/syslog | tail --lines=10

# Check last times rsyslog was started
grep "] start" /var/log/messages | tail --lines=10 # You could also grep "] start" /var/log/syslog | tail --lines=10


Start Debian base system configuration tool:
base-config

Show Hardware info:
dmidecode -t TYPE_OR_SUPERTYPE # Valid types are: 0: BIOS, 1: System, 2: Base Board, 3: Chassis, 4: Processor, 5: Memory Controller, 6: Memory Module, 7: Cache, 8: Port Connector, 9: System Slots, 10: On Board Devices, 11: OEM Strings, 12: System Configuration Options, 13: BIOS Language, 14: Group Associations, 15: System Event Log, 16: Physical Memory Array, 17: Memory Device, 18: 32-bit Memory Error, 19: Memory Array Mapped Address, 20: Memory Device Mapped Address, 21: Built-in Pointing Device, 22: Portable Battery, 23: System Reset, 24: Hardware Security, 25: System Power Controls, 26: Voltage Probe, 27: Cooling Device, 28: Temperature Probe, 29: Electrical Current Probe, 30: Out-of-band Remote Access, 31: Boot Integrity Services, 32: System Boot, 33: 64-bit Memory Error, 34: Management Device, 35: Management Device Component, 36: Management Device Threshold Data, 37: Memory Channel, 38: IPMI Device, 39: Power Supply; Valid supertypes are: bios (0, 13), system (1, 12, 15, 23, 32), baseboard (2, 10), chassis (3), processor (4), memory (5, 6, 16, 17), cache (7), connector (8), slot (9).
dmidecode -t 6|grep -i "Size" # Return installed memory banks
dmidecode -t 1|grep -i "Manufacturer\|Product\|Serial" # Return system's Manufacturer, Product Name, and Serial Number

Show RAM info:
cat /proc/meminfo
grep MemTotal /proc/meminfo  # return total RAM seen by the system
egrep 'Mem|Cache|Swap' /proc/meminfo # return information about RAM, Cache, Swap
vmstat -s # show information about memory activity

Show memory usage:
free # show total, used, free, available memory in Bytes
free -w # show total, used, free, available memory in Bytes, in wide format (separating values for buffers and cache)
free -k # show total, used, free, available memory in KB
free -m # show total, used, free, available memory in MB
free -h # show total, used, free, available memory in human readable format
free -wh -c 3 -s 1 # show memory usage 3 times with 1 second interval
free -h | xargs | awk '{print "Memory: Size: "$8" Used: "$9" Free: "$10" Avail: "$13}' # Return Total, Used, Free, Available Memory

Swap memory:
cat /proc/sys/vm/swappiness # Show system Swappiness (Range: 0 - 100 ; 0: Never use Swap (disabled) , 100: Always use Swap ; Default: 60 )
sysctl vm.swappiness=10 # Set Swappiness to 10
swapoff -a # Disable Swap memory
swapon -a # Enable Swap memory (wait about 30 seconds after disabling Swap before to enable it again, to let your system clear Swap memory proprerly
swapon --show # Show summary of swap devices and their swap memory usage in human readable format
swapon --show --noheadings --raw --bytes # Show summary of swap devices and their swap memory usage in bytes without headers
swapon --show --noheadings --raw --bytes | xargs | awk '{print $1","$3","$4}' # Show swap memory device name and usage in bytes, comma separated
swapon --show --noheadings --raw | xargs | awk '{print "Swap File: Dev: "$1" Total: "$3" Used: "$4}' # Show swap memory device name and usage in verbose human readable format
for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done # List Swap space usage per process
for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 2 -n -r # List Swap space usage per process and sort results

Virtual memory statistics: 
vmstat # report information about processes, memory, paging, block IO, traps, and cpu activity.
vmstat -s # event counters and memory statistics
vmstat -d # disk statistics

Show CPU info:
lscpu
lscpu | grep "mode(s)" # return CPU op-modes (32-bit, 64-bit)
lscpu | grep "Byte Order" # return CPU Byte Order (Little Endian, Big Endian)
cat /proc/cpuinfo
grep 'model name' /proc/cpuinfo # return CPU model
grep -c 'processor' /proc/cpuinfo # count CPU cores
grep 'flags' /proc/cpuinfo # return available features of the CPU
cat /proc/loadavg # show load average of the system: the first three fields in this file are load average figures giving the number of jobs in the run queue (state R) or waiting for disk I/O (state D) averaged over 1, 5, and 15 minutes. They are the same as the load average numbers given by uptime(1) and other programs. The fourth field consists of two numbers separated by a slash (/). The first of these is the number of currently executing kernel scheduling entities (processes, threads); this will be less than or equal to the number of CPUs. The value after the slash is the number of kernel scheduling entities that currently exist on the system. The fifth field is the PID of the process that was most recently created on the system.
grep 'cpu ' /proc/stat | awk '{usage=($2+$4)*100/($2+$4+$5)} END {print usage "%"}' # show CPU load
uptime | awk -F'[a-z]:' '{print $2}' # Show load average of the system over the last 1, 5, and 15 minutes.
echo -n "CPU average load (Cores: "; grep 'cpu cores' /proc/cpuinfo | xargs | awk '{printf $4}'; echo -n "): "; uptime | awk -F'[a-z]:' '{print $2}' | xargs | awk '{print "1 m: "$1" 5 m: "$2" 15 m: "$3}'; # Show load average of the system over the last 1, 5, and 15 minutes (verbose).

Show Display info:
cat /sys/class/graphics/*/name # Show Display Adapter Name
cat /sys/class/drm/*/modes # Show available Video modes
cat /sys/class/graphics/*/modes # Show current Video mode
cat /sys/class/graphics/*/virtual_size # Show Virtual Display resolution
cat /sys/class/graphics/*/bits_per_pixel # Show color depth

List all PCI buses and devices:
lspci

Show system temperature (if system supports ACPI):
apt-get install acpi
acpi -t

-----

Localization:
dpkg-reconfigure locales # edit locale / add new locales
locale  # show locale settings
locale -a  # show available locales
locale|cut -d= -f1|xargs locale -kc|less  # list fields in locale database
locale territory  # show the locale territory # echo "This server is configured for `locale territory`"
printf "%'d\n" 1234567  # print an integer number grouping thousands with the locale configuration

l /usr/share/i18n/locales/ # List all available locales
l /usr/share/i18n/locales/en_* # List all available English locales

localectl # Show system locales
cat /etc/default/locale # Show system locale variables

jed /etc/default/locale # Set system locale variables

Set the following variables to the given values if you want to get the date in 12 or 24 hours:

# 24 hours
----------
LANG=en_IE.UTF-8
LANGUAGE=en_IE:en
----------

# 12 hours (AM/PM)
----------
LANG=en_US.UTF-8
LANGUAGE=en_US:en
----------

jed /etc/locale.gen # Locales to be generated. Uncomment (remove # ) the locales you want to generate

locale-gen # generate locales (the ones anabled in /etc/locale.gen )

-----

List of all commands entered in the shell (history):
jed ~/.bash_history # edit history file manually
cat ~/.bash_history # show all commands in history (up to last session)
less ~/.bash_history # navigate in history
history | grep "apt-get install \|apt install " # Search for last packages installed manually
cat ~/.bash_history | tr "\|\;" "\n" | sed -e "s/^ //g" | cut -f 1 -d " " | sort -n | uniq -c | sort -nr | head -n 10 # show the 10 most used commands in history (including piped commands)
history | awk '{CMD[$2]++;count++;}END { for (a in CMD)print CMD[a] " " CMD[a]/count*100 "% " a; }' | grep -v "./" | column -c3 -s " " -t | sort -nr | nl | head -n 10 # show the 10 most used commands in history (first command only). NOTE: if dates are enabled in history, it will show days when more commands were issued
history | awk '{CMD[$4]++;count++;}END { for (a in CMD)print CMD[a] " " CMD[a]/count*100 "% " a; }' | grep -v "./" | column -c3 -s " " -t | sort -nr | nl | head -n 10 # show the 10 most used commands in history (first command only) if dates are enabled
history # show all commands in history together with their offset
history -d OFFSET # delete the specified history entry
history -c # clear the whole history file
!!:p # show last executed command
!-1:p # show last executed command
!-2:p # show second last executed command
!TEXT:p # show last command starting with TEXT (for example, ls)
!! # repeat (execute again) last executed command
!-1 # repeat (execute again) last executed command
!-2 # repeat (execute again) second last executed command
!TEXT:p # repeat (execute again) last command starting with TEXT (for example, ls)
export HISTTIMEFORMAT="%F %T  " # Set history date format (normally by default date and time of executed commands is not preserved in history)
echo $HISTTIMEFORMAT # See history date format

Edit user creation preferences:
jed /etc/adduser.conf

Create a new user:
# useradd is similar to adduser
adduser USER # add a new USER with a home directory in the default path ( /home/USER )
adduser --home DIR USER # add a new USER specifying a non standard path for the home directory
adduser --no-create-home USER # add a new USER without a home directory

Modify a user information:
usermod [OPTIONS] USER

Edit user deletion preferences:
jed /etc/deluser.conf

Remove a user:
# userdel is similar to deluser
deluser USER # remove the specified user
deluser --remove-home USER # remove the specified user and his home directory and mail spool
deluser --remove-all-files USER # remove the specified user and all files owned by the user (be careful)

Change user contact information:
chfn [OPTIONS] USER

Change a user's default shell:
chsh [OPTIONS] USER

Get a user hashed password:
getent shadow|grep "USER"|cut -f 2- -d ":"|cut -f 1 -d ":" # Get the hash type, salt and hashed password for given USER
getent shadow|grep "$(whoami)"|cut -f 2- -d ":"|cut -f 1 -d ":" # Get the hash type, salt and hashed password for current user
getent shadow|grep "$(whoami)"|cut -f 2- -d ":"|cut -f 1 -d ":"|cut -f 4- -d "$" # Get the hashed password for given USER
getent shadow|grep "$(whoami)"|cut -f 2- -d ":"|cut -f 1 -d ":"|cut -f 4- -d "$" # Get the hashed password for current user
getent shadow|grep "USER"|cut -f 2- -d ":"|cut -f 1 -d ":"|cut -f 2- -d "$"|cut -f 2- -d "$"|cut -f 1 -d "$" # Get the salt for given USER
getent shadow|grep "$(whoami)"|cut -f 2- -d ":"|cut -f 1 -d ":"|cut -f 2- -d "$"|cut -f 2- -d "$"|cut -f 1 -d "$" # Get the salt for current user
getent shadow|grep "USER"|cut -f 2- -d ":"|cut -f 1 -d ":"|cut -f 2- -d "$"|cut -f 1 -d "$" # Get the hash type for given USER (1: md5, 5: sha-256, 6: sha-512)
getent shadow|grep "$(whoami)"|cut -f 2- -d ":"|cut -f 1 -d ":"|cut -f 2- -d "$"|cut -f 1 -d "$" # Get the hash type for current user (1: md5, 5: sha-256, 6: sha-512)

Change a user password:
passwd USER

Set a user password to expire:
# note that having frequently changing passwords is usually not a good policy, as it becomes harder for users to remember them, and they generally end up with weak passwords
chage -m 3 -M 30 -w 2 USER # USER's password can last up to 30 days, can't be changed earlier than 3 days after last change, and will be warned 2 days before password expiration

Create a new user group:
# addgroup is a link to adduser ( adduser --group ). There's also a groupadd command.
addgroup GROUP

Modify a group information:
groupmod [OPTIONS] GROUP

Remove a group:
# There's also a delgroup link to deluser ( deluser --group )
groupdel GROUP

Add a user to a group:
adduser USER GROUP

Execute a command with root privileges (root password will be asked):
sudo COMMAND

Change current user's password:
passwd

Show all existing users:
getent passwd # format: username:password (usually shadowed, a 'x' is shown instead):user id:group id:real name:home path:console. If you are root you can access directly the file with cat /etc/passwd
getent shadow # show users and their hashed password. If you are root you can access directly the file with cat /etc/shadow
getent passwd | grep -c . # count all existing users

Show all logged users:
users # list all logged users
who # show information about all logged users
w # show more information than "who"
who | grep -c . # count all logged users

Show last logged users:
last # Show last logins
last USER # show last times when USER logged in
last -n 10 # show last 10 logged users
last -n 5 USER # show last 5 times when USER logged in
touch /var/log/lastlog # Create the login log file to enable lastlog
lastlog # Show last time each user logged in
lastlog -u USER # Show last time the USER logged in
lastlog -t 30 # Show only users who logged in during the last 30 days
lastlog -b 365 # Show only users who last logged more than 365 days ago
lastb -n 10 # show last 10 bad login attempts (which are likely break-in attempts)
lastb -n 10 root # show last 10 bad login attempts to root (which are likely break-in attempts)

grep "Accepted" /var/log/auth.log | tail --lines=10 # See last accepted password events for succeful logins
grep "Failed" /var/log/auth.log | tail --lines=10 # See last rejected password events for failed login attemps (which include possible break-in attempts)
grep "Failed password for invalid user" /var/log/auth.log | tail --lines=10 # See last rejected password events for failed login attemps with non existing users (which are likely break-in attempts)

Show failed login attempts:

To enable faillog:
-----
Create the faillog log file:
touch /var/log/faillog

jed /etc/pam.d/common-auth

Add these lines at the top of the file:

# Log failed logins to /var/log/faillog
auth required pam_tally.so per_user magic_root onerr=fail

jed /etc/pam.d/sshd

Add the following lines immediately before @include common-auth (generally at the beginning of the file):

# Log failed login attempts to /var/log/faillog
auth required pam_tally.so per_user onerr=fail

jed /etc/ssh/sshd_config

Enable PAM, search for UsePAM and set it to yes if not enabled, or add the whole line if it's missing:

UsePAM yes

Restart SSH:

service ssh restart
-----
faillog # Show all users who attempted to log in without success
faillog -a # Show all failed login attempts including the ones of users who eventually logged in
failed -u USER # Show all failed login attempts for the specified USER, even if he eventually logged in
faillog -t 30 # Show only failed logins occurred during the last 30 days
faillog -l 5 # Lock the account for 5 seconds after each failed attempt
faillog -m 5 # Disable the account after 5 failed attempts (0 means that infinite attempts are allowed. You'd better leave the value for root to 0 to prevent a DoS attack)
faillog -u USER -r # Reset the counter of failed logins for the given user, enabling his account again if it was locked because of too many failed attempts (as specified in faillog -m N)
faillog -r # Reset counters of failed logins for all users
cat /var/log/faillog # Show the actual failed attempts log file
grep "authentication failure" /var/log/messages # extract failed login attempts from the messages file

Show current user's name:
whoami

Show information about a user:
finger USER

Show groups to which current user belongs:
groups

Show user and group IDs:
id # show user and group IDs for the current user
id USER # show user and group IDs for the specified USER

Send a message to a logged user (to the output console of his terminal):
# Check the user device with a w or a who command first then redirect the output of an echo command to such device
echo -ne "Hello\n">/dev/pts/1 # Send "Hello" to the user logged with pts/1

# Send a message to a logged user (to the output console of his terminal) who's using a specific process (identified by its PID):
echo -ne "Hello\n" > /proc/PID/fd/0

Send keystrokes to a terminal:
# (use ps -A|grep "PROCESS_NAME" to find in which terminal is running a specific process, in case you need to send keystroke to a process waiting for an input)
perl -e '$TIOCSTI = 0x5412; $tty = "/dev/pts/TERMINAL"; $char = "\n"; open($fh, ">", $tty); ioctl($fh, $TIOCSTI, $char);'; # Send a single keystroke (ENTER in this example) to the given TERMINAL
perl -e '$TIOCSTI = 0x5412; $tty = "/dev/pts/TERMINAL"; $str = "TEXT_STRING\n"; for my $i (1..length($str)) { open($fh, ">", $tty); $char=substr($str, $i-1, 1); ioctl($fh, $TIOCSTI, $char); }'; # Send a TEXT_STRING to the given TERMINAL

Send a message to all logged users:
wall PATH/FILE # show the content of FILE to all logged users (max 20 lines)
wall # use standard input (normally the keyboard) to show a message to all logged users. Message must be terminated with an EOF (End Of File) character (usually CTRL+D)

Execute a command as another user (impersonated user password will be requested):
su USER COMMAND

Start a console as another user (impersonated user password will be requested):
su USER
su # if no user is specified, then root is assumed by default

-----

File status and information:
file PATH/FILE # identify file type
stat PATH/FILE # Show file properties (file status)
stat --printf="%U" PATH/FILE # Get user name of the owner of the file
stat --printf="%u" PATH/FILE # Get user ID of the owner of the file
stat --printf="%s" PATH/FILE # Get file size
wc -c PATH/FILE | awk '{print $1}' | tr -d '\n' # Get file size
basename PATH/FILE # return the file name (FILE) of the specified FILE
s1='PATH/FILE'; echo ${s1#$(dirname "$s1")/}; # return the file name (FILE) of the specified FILE
dirname PATH/FILE # return the path (from root) of the specified FILE
realpath PATH/FILE # Get the full path of a file
s1='DIR1/DIRN/FILE'; echo ${s1#$(dirname "$(dirname "$s1")")/}; # return the last directory (DIRN) and the file name (FILE) of the specified FILE

Search for a file:
find PATH -type f # find all files in PATH and its subdirectories
find PATH -type d # find all subdirectories in PATH and its subdirectories
find PATH -name FILE_MASK # find all files and directories with given mask inside PATH and its subdirectories. Example: find / -name a* # find all files and directories starting with a in the server
find PATH -type f -name FILE_MASK # find all files with given mask inside PATH and its subdirectories. Example: find / -type f -name *.php # find any PHP file in the server
find PATH -type d -name FILE_MASK # find all directories with given mask inside PATH and its subdirectories. Example: find / -type d -name test* # find all test directories in the server
find PATH -type f -regex ".*\.EXT$" # Find all files in PATH and its subdirectories with given EXTension
find PATH -type f -regex ".*\.[0-9]*$" # Find all files in PATH and its subdirectories with a numeric extension
whereis FILENAME # return the paths of files with matching FILENAME
locate NAME # return all paths and files with a matching NAME string
grep -r "STRING" PATH # search for all files containing the given STRING, within the PATH and its subdirectories (-r)
grep -r -l "STRING" PATH # search for all files containing the given STRING, within the PATH and its subdirectories (-r) and return just matching files name and not each occurrence of the string within every file (-l)
find PATH | xargs grep "STRING" -sl # search for all files containing the given STRING, within the PATH and its subdirectories, returns only filenames with path
find PATH -type f -newermt "2019-01-01" \! -newermt "2020-09-01" -exec ls -ld {} \; # Find all files in PATH and its subdirectories created between the two given dates
find PATH -type f -newermt "2019-01-01" \! -newermt "2020-09-01" -exec ls -ld {} \; # Find all subdirectories in PATH and its subdirectories created between the two given dates

Copy a file or a directory:
cp -rp SOURCE DESTINATION # copy a file or a directory tree preserving owners and properties
mkdir -p PATH && cp -rp SOURCE $_ # copy a file or a directory tree (preserving owners and properties) and create destination path if it doesn't exist

Copy a file over a network:
apt-get install netcat
# On the Destination computer (listener). Make sure the PORT you are listening isn't blocked. Timeout (-w) is set to 3 seconds:
nc -vvn -l -p PORT -w 3 > DEST_PATH/FILE
# On the Source computer (sender). Make sure the PORT you are sending to isn't blocked. It will quit 2 seconds after receiving an EOF. If -q is not specified, then netcat must be killed manually:
nc -vvn DESTINATION_IP PORT -q 2 < SOURCE_PATH/FILE 

Copy a file securely (through SSH) over a network (it will ask for passwords if needed for authentication on remote hosts):
# Use blowfish encryption (-c blowfish) instead of the default TripleDES (3DES) because it's more secure and faster.
scp -c blowfish SOURCEPATH/NAME DESTUSER@DESTHOST:DESTPATH/NAME  # copies a file from the local system to a remote host
scp -c blowfish SOURCEUSER@SOURCEHOST:DSOURCEPATH/NAME DESTPATH/NAME  # copies a file from a remote host to the local system
scp -c blowfish SOURCEUSER@SOURCEHOST:DSOURCEPATH/NAME DESTUSER@DESTHOST:DESTPATH/NAME  # copies a file from a remote host to another remote host
scp -c blowfish -r SOURCEUSER@SOURCEHOST:DSOURCEPATH DESTUSER@DESTHOST:DESTPATH/  # copies a directory tree from a remote host to another remote host

Move a file or a directory tree (or rename):
mv SOURCEPATH/NAME DESTPATH/NAME
for a in *; do mv -v "$a" "${a//STRING_SEARCH/STRING_REPLACE}"; done  # batch rename all files in the current directory
for a in *; do mv -v "$a" "${a}_`date --iso-8601`"; done  # append date as YYYY-MM-DD to all file names in the current directory (note that it will be appended after any extension)
find PATH -type f -name "*" -exec mv "{}" "{}".txt \; # add an extension (.txt in this example) to all files in given PATH
find PATH -type f -not -name "*.*" -exec mv "{}" "{}".txt \; # Add an extension (.txt in this example) to all files without an extension in given PATH
for i in `ls PATH -1`; do mv $i "${i,,}" ; done # rename all files in the given PATH to lowercase
for i in `ls PATH -1`; do mv $i "${i^^}" ; done # rename all files in the given PATH to uppercase
for f in * ; do mv "$f" "$f.txt"; done # add .txt extension (suffix) to all files (in the current directory)
for f in *.txt ; do mv "$f" "${f%txt}asc"; done # change extension (suffix) from .txt to .asc
for f in *.txt ; do mv "$f" "${f%.txt}"; done # remove .txt extension (suffix)
for f in * ; do mv "$f" "prefix_$f"; done # add _prefix prefix to all files (in the current directory)
for f in prefix_* ; do mv "$f" "${f#prefix_}"; done; # remove prefix_
for f in prefix_* ; do mv "$f" "x_${f#prefix_}"; done; # replace prefix_ with x_

Synchronize two directories:
apt-get install rsync
rsync -a -v /SOURCE_PATH/* /DEST_PATH/ # copy all files from /SOURCE_PATH that doesn't exist in /DEST_PATH
rsync -a --delete -v /SOURCE_PATH/* /DEST_PATH/ # copy all files from /SOURCE_PATH that doesn't exist in /DEST_PATH, and delete all files in /DEST_PATH that doesn't (no longer) exist in /SOURCE_PATH
rsync -a --delete -v rsync://www.example.com/dir/ /DEST_PATH # synchronize from an external rsync directory

Create a link to a file:
ln -s PATH/ACTUAL_FILE PATH/LINK_TO_CREATE

Create an empty file:
touch PATH/NAME # Create an empty file. If the given file already exists, it updates its access date

Create a dummy file with a given size:
fallocate -l 1K PATH/NAME # create a dummy file sized 1Kbyte
dd if=/dev/zero of=PATH/NAME bs=1K count=2 # create a dummy file sized 2Kbyte

Create a unique temporary file or directory with a pseudo random name:
mktemp "tmp_XXXXXX" # Create an empty file with a random name generated from the given mask, for example tmp_92pIKB or tmp_Z1Pdz1
mktemp -d "tmp_XXXXXX" # Create an empty directory with a random name generated from the given mask, for example tmp_olD9rA or tmp_5Fw0JI
Sample usage of mktemp (from mktemp man):
TMPFILE=`mktemp /tmp/example.XXXXXX` || exit 1
echo "program output" >> $TMPFILE

Split a file in chunks:
split -b=BYTES PATH/FILE PREFIX # split FILE in chunks of BYTES bytes, using PREFIX to name generated chunks
split -l LINES PATH/FILE.txt PREFIX # split PATH/FILE.txt in chunks of LINES lines, using PREFIX to name generated chunks

Rebuild a split file (concatenate files):
cat PREFIX* > PATH/FILE
sed -e '$a\' test?.txt > testx1b.txt # concatenate text files appending an empty line at the end of every line if there isn't already one
(for r in PREFIX*;do sed s/\$/\ $r/ < "$r";done) > PATH/FILE # concatenate text files appending the file name after each line
cat $(ls PREFIX* -t) > PATH/FILE # concatenate files sorted by date and time (newer first, older last)
cat $(ls PREFIX* -tr) > PATH/FILE # concatenate files sorted by date and time (older first, newer last)

Type a file:
cat PATH/FILE
cat -n PATH/FILE # show a text file with line numbers
cat PATH/FILE | tr [:upper:] [:lower:] # show a text all in lower case
cat PATH/FILE | tr [:lower:] [:upper:] # show a text all in Upper case
tac PATH/FILE # show a text from the last line to the first (thus listing lines backwards), useful to ordering elements in a last-in, first-out (LIFO) way
fold -w COLOUMN PATH/FILE # wrap a text file, cutting lines at the given coloumn. Useful to format data as input for a program
fmt -u -w80 file.txt # format a text file output: 1 space between words, 2 spaces between sentences, 80 coloumns
nl PATH/FILE  # show line numbers for every line (empty lines are not counted by default)
nl -ba PATH/FILE # show line numbers for every line and count all lines, including empty ones
nl -s": " PATH/FILE # show line numbers for every line and separate line numbers from line text using the specified separator (": " in this case)
pr -l 50 test.txt # show a text as pages, each page is 50 lines long (minimum 11, default is 66)
pr -l 50 --column 2 test.txt # show a text as pages, each page is 50 lines long and divided into 2 columns
more PATH/FILE  # type a file page by page
less PATH/FILE  # show a file page by page, and let scroll through pages. It shows escaped characters as plain text (get it with apt-get install less)
less -F PATH/FILE  # show a file page by page, and let scroll through pages (get it with apt-get install less). If the file output fits a single screen it quits without having to press "q"
grep "TEXT" PATH/FILE  # show only lines containing "TEXT"
grep -v "TEXT" PATH/FILE  # show only lines NOT containing "TEXT"
grep -i "TEXT" PATH/FILE  # show only lines containing "TEXT" (case insensitive)
grep -vi "TEXT" PATH/FILE  # show only lines NOT containing "TEXT" (case insensitive)
grep -m 1 "TEXT" PATH/FILE  # show only the first matching line containing "TEXT"
grep -v -m 1"TEXT" PATH/FILE  # show only the first matching line NOT containing "TEXT"
grep -i -m 1"TEXT" PATH/FILE  # show only the first matching line containing "TEXT" (case insensitive)
grep -vi -m 1"TEXT" PATH/FILE  # show only the first matching line NOT containing "TEXT" (case insensitive)
grep -B 1 -A 1 "TEXT" PATH/FILE  # show lines containing "TEXT" and show 1 line Before and 1 line After each match
grep -B 2 -A 2 -n "TEXT" PATH/FILE  # show lines containing "TEXT" and show 2 lines Before and 2 lines After each match, together with line numbers
head PATH/FILE  # show the head (first 10 lines) of FILE
head --lines=20 PATH/FILE  # show the first 20 lines of FILE
head --bytes=1024 PATH/FILE  # show the first 1024 bytes of FILE
head --lines=-50 PATH/FILE  # show all the FILE but the last 20 lines
head --bytes=-2048 PATH/FILE  # show all the FILE but the first 2048  bytes
tail PATH/FILE  # show the tail (last 10 lines) of FILE
tail --lines=20 -F PATH/FILE  # show the last 20 lines of FILE, and new lines as the file grows. If the files doesn't exist yet, it waits for it (useful for log files)
tail --lines=+20 -F PATH/FILE  # show the last lines of FILE starting from line 20
tail --bytes=1024 PATH/FILE  # show the last 1024 bytes of FILE
tail -f PATH/FILE # keep showing the last line of FILE whenever there are added new ones (useful for log files) until stopped
tail -f PATH/FILE | sed '/^ENDSTRING$/ q' # keep showing the last line of FILE whenever there are added new ones (useful for log files) until a given string appears
sed -ne "3p" PATH/FILE  # show line number 3 of FILE
sed -ne "2,4p" PATH/FILE # show lines 2 to 4 of FILE
sed -ne "1,5p" PATH/FILE # show lines 1 (beginning) to 5 of FILE
sed -ne "5p,$" PATH/FILE # show lines 5 to the end of FILE
sed -n -e '3,5p' -e '9,15p' PATH/FILE # show lines 3 to 5 and 9 to 15 of FILE
sed -e "5d" PATH/FILE # show all lines of FILE except for the ones from 2 to 30 included, and except for line 5
sed -e "2,30d" PATH/FILE # show all lines of FILE except for the ones from 2 to 30 included
sed -e "5d;2,30d" PATH/FILE # show all lines of FILE except for line 5, and except for the ones from 2 to 30 included
awk 'NR==2 {print;exit}' SOURCE # Print line number 2 from SOURCE file
awk 'NR==2,NR==4' SOURCE # Print line numbers 2 to 4 (inclusive) from SOURCE file
awk '{print $1}' PATH/FILE # print the first word of every line in FILE
awk '{print substr ($1,0,1)}' PATH/FILE # print the first character of every line in FILE
awk 'NF>=1{print $NF}' PATH/FILE # print the last word of every line in FILE
grep -o '.$' PATH/FILE # print the last character of every line in FILE
cat PATH/FILE | awk 'length($0) < 5' # show lines longer less than 5 characters in FILE
cat PATH/FILE | awk 'length($0) > 5' # show lines longer more than 5 characters in FILE
cat PATH/FILE | awk 'length($0) == 5' # show lines longer with exactly 5 characters in FILE
cat PATH/FILE | awk 'length($0) >= 5' # show lines with at least 5 characters in FILE
cat PATH/FILE | awk 'length($0) >= 5 { print NR ":" $0 }' # show lines with at least 5 characters in FILE, preceded by line numbers of matching lines
grep '.\{5\}' PATH/FILE # show lines with at least 5 characters in FILE

Binary dump of a file:
xxd -b PATH/FILE # binary dump of FILE
xxd -b PATH/FILE | less # navigate the binary dump of FILE

Hexadecimal dump of a file:
xxd PATH/FILE # hex dump of FILE
xxd PATH/FILE | less # navigate the hex dump of FILE
xxd -u PATH/FILE # hex dump of FILE. Show hex numbers in Upper case
xxd -a PATH/FILE # hex dump of FILE. Replace multiple NUL (00) lines with an asterisk ( * )
od -Ax -tx1z -v PATH/FILE # hex dump of FILE
hexdump -C PATH/FILE # hex dump of FILE
hexdump PATH/FILE # hex dump of FILE without the ASCII equivalent

Extrapolate strings of text from binary files:
strings -a -n 4 PATH/FILE # extrapolate strings of text from binary files (a string of text is considered at least 4 bytes long, you can change this value with the option -n) (get it with apt-get install binutils )
strings -a -t x PATH/FILE # extrapolate strings of text from binary files and return the offset of the string (-t x specifies the offset as hex, -t d decimal, -t o octal)

Search for data in a CSV (Comma Separated Values) / XSV (Character Separated Values) / SSV (Separator Separated Values) file:
awk -F',' '$1 == "TEXT"' /PATH/FILE.csv # If the first item (field) matches with the given TEXT in a comma (',') separated file, then return the whole line (row / record)
awk -F',' '{ if ($1 == "TEXT") { print $2 } }' /PATH/FILE.csv # If the first item (field) matches with the given TEXT in a comma (',') separated file, then return the second record in the line (row / record)

Count lines in text files:
wc PATH/FILE # line/word/byte count (handy options: --lines or -l print only the newline counts, --words print only the word counts, --bytes print only the byte counts, --chars print the character counts)
wc -L PATH/FILE # print the lenght of the longest line of FILE
grep "STRING" PATH/FILE|wc --lines # count how many lines in FILE contains STRING
wc -l PATH/*.txt # count the lines of all files with .txt extension in a directory
find PATH -name "*.txt" | xargs wc -l # count the lines of all files with .txt extension in a directory and its subdirectories

Sort a text file:
sort PATH/FILE.txt # sort FILE.txt alphabetically, case sensitive
sort -i PATH/FILE.txt # sort FILE.txt alphabetically, cosindering only printable characters
sort -d PATH/FILE.txt # sort FILE.txt alphabetically, cosindering only dictionary characters (alphanumeric characters and blanks)
sort -f PATH/FILE.txt # sort FILE.txt alphabetically, case insensitive
sort -fs PATH/FILE.txt # sort FILE.txt alphabetically, case insensitive and stable (sort according which case comes first for each letter)
sort -b PATH/FILE.txt # sort FILE.txt ignoring leading blank spaces
sort -r PATH/FILE.txt # sort FILE.txt in reverse alphabetical order
sort -n PATH/FILE.txt # sort numbers in FILE.txt
sort -h PATH/FILE.txt # sort human readable numbers (like 5K, 2G) in FILE.txt
sort -M PATH/FILE.txt # sort months (like Jan, Feb, Mar) in FILE.txt
sort -R PATH/FILE.txt # sort lines randomly in FILE.txt
sort -u PATH/FILE.txt # sort lines in FILE.txt and keep only unique lines
sort -k 2 PATH/FILE.txt # sort lines in FILE.txt using characters from column 2 (any blank character separates columns)
sort -t : -k 2 PATH/FILE.txt # sort lines in FILE.txt using characters from column 2 using ":" as a separator for columns
sort PATH/FILE.txt -o "PATH/DEST_FILE.txt" # sort FILE.txt and save the output in DEST_FILE.txt

Remove lines from a text file:
sed -i '1d' PATH/TEXT_FILE # Remove the first line from a file
cp PATH/TEXT_FILE PATH/TEXT_FILE.tmp ; sed '1d' PATH/TEXT_FILE.tmp > PATH/TEXT_FILE ; rm -f PATH/TEXT_FILE.tmp # sed older than v3.95 doesn't have the option -i and requires a temporary file
sed -i '$ d' PATH/TEXT_FILE # Remove the last line from a file
cp PATH/TEXT_FILE PATH/TEXT_FILE.tmp ; sed '$ d' PATH/TEXT_FILE.tmp > PATH/TEXT_FILE ; rm -f PATH/TEXT_FILE.tmp # sed older than v3.95 doesn't have the option -i and requires a temporary file

More file operations:
# All these commands should work also with "gawk" (GNU awk) and "nawk" (new awk by AT&T)
awk '1; { print "" }' SOURCE>DESTINATION # Add an empty line after each line
awk 'NF { print $0 "\n" }' SOURCE>DESTINATION # Add an empty line after each line, except when an empty line is already present
awk '{ print FNR ".\t" $0 }' SOURCE>DESTINATION # Add line numbers followed by dot and tab before than each line
awk '{sub(/^[ \t]+/, "")};1' SOURCE>DESTINATION # Remove leading spaces from each line
awk '{sub(/[ \t]+$/, "")};1' SOURCE>DESTINATION # Remove trailing spaces from each line
sed "s/[ \t]*$//" PATH/SOURCE_FILE>PATH/DESTINATION_FILE # Remove trailing spaces and tabs from a text file
awk '{gsub(/^[ \t]+|[ \t]+$/,"")};1' SOURCE>DESTINATION # Remove leading and trailing spaces from each line
awk '{sub(/A/,"B")}; 1' SOURCE>DESTINATION # Replace the first occurrence of A with B in each line
awk '{gsub(/A/,"B")}; 1' SOURCE>DESTINATION # Replace every occurrence of A with B in each line
awk '/X/{gsub(/A/, "B")}; 1' SOURCE>DESTINATION # Replace every occurrence of A with B in each line that contains X
awk '!/X/{gsub(/A/, "B")}; 1' SOURCE>DESTINATION # Replace every occurrence of A with B in each line that does not contain X
awk 'BEGIN{while (a++<100) s=s "."; print s}' >DESTINATION # Create a file containing 100 dots (.)
awk 'END{print NR}' SOURCE # count lines (emulates "wc -l")

# Strip trailing line break (new line) from output
echo "Test" | tr -d '\n' # In this example, this would be the same as echo -n "Test"

Return the average lenght in lines for all text files in the tree:
find PATH -name "*.txt" -exec wc -l {} \; | awk 'BEGIN {x=0;y=0} {x+=1; y+=$1} END {print y/x}'

Count the occurrences of all the words in a text file:
tr -c a-zA-Z '\n' < PATH/FILE.txt | sed '/^$/d' | sort -n | uniq -ci | sort -nr # case insensitive
tr -c a-zA-Z '\n' < PATH/FILE.txt | sed '/^$/d' | sort -n | uniq -c | sort -nr # case sensitive

Count the occurrences of a string within another string:
needle=","; haystack="one, two, three, four, five"; occurrences=$(grep -o "$needle" <<< "$haystack" | wc -l); echo $occurrences

Insert characters into a string:
echo '74657374' | sed 's/.\{2\}/&\./g' # Insert a dot every 2 characters and (it may leave a final dot)
echo '74657374' | sed 's/.\{2\}/&\./g' | awk '{sub(/\.$/, "")};1' # Insert a dot every 2 characters and remove the last dot if present (warning: it may remove a pre existing final dot)
echo '74657374' | sed 's/.\{4\}/& /g' # Insert a space every 4 characters and (it may leave a final space)
echo '74657374' | sed 's/.\{4\}/& /g' | awk '{sub(/ $/, "")};1' # Insert a space every 4 characters and remove the last space if present (warning: it may remove a pre existing final space)

Truncate a string:
echo $(cut -c 3 <<< "this_is_an_example") # Get the third character of a string. Example: before: "this_is_an_example", after: "h".
echo "this_is_an_example" | cut -c 3 # Get the third character of a string. Example: before: "this_is_an_example", after: "h".
echo "this_is_an_example" | cut -c 1 # Get the first character from a string. Example: before: "this_is_an_example", after: "t".
echo "this_is_an_example" | cut -c -3 # Get the first 3 characters from a string. Example: before: "this_is_an_example", after: "thi".
echo "this_is_an_example" | cut -c 4-9 # Get the characters ranging in position 4 up to 9 from a string. Example: before: "this_is_an_example", after: "s_is_a".
echo "this_is_an_example" | cut -c 3- # Cut the first 2 characters from a string (returns the string starting from the 3rd character). Example: before: "this_is_an_example", after: "is_is_an_example".

echo "this_is_an_example" | rev | cut -c 3 # Get the third character from the end of a string. Example: before: "this_is_an_example", after: "p".
echo "this_is_an_example" | rev | cut -c 1 # Get the last character of a string. Example: before: "this_is_an_example", after: "e".
echo "this_is_an_example" | rev | cut -c 1-3 | rev # Get the last 3 characters from the end of a string. Example: before: "this_is_an_example", after: "ple".
echo "this_is_an_example" | rev | cut -c 4-9 | rev # Get the characters ranging in position 4 up to 9 from the end of a string. Example: before: "this_is_an_example", after: "n_exam".
echo "this_is_an_example" | rev | cut -c 3- | rev # Cut the last 2 characters from a string (returns the string up to the 3rd character from the end). Example: before: "this_is_an_example", after: "this_is_an_examp".

Truncate a string using a marker:
# The marker (delimiter) to set the point where to truncate a string is ":" in the following examples
cut -d ":" -f1 <<< "this:is:an:example" # Truncate a string at the first occurrence of a marker (removing the marker). Example: before: "this:is:an:example", after: "this".
cut -d ":" -f2- <<< "this:is:an:example" # Truncate a string after the first occurrence of a marker (removing the marker). Example: before: "this:is:an:example", after: "is:an:example".
echo "this:is:an:example" | cut -d ":" -f1 # Truncate a string at the first occurrence of a marker (removing the marker). Example: before: "this:is:an:example", after: "this".
echo "this:is:an:example" | cut -d ":" -f2- # Truncate a string after the first occurrence of a marker (removing the marker). Example: before: "this:is:an:example", after: "is:an:example".
echo "this:is:an:example" | rev | cut -d ":" -f2- | rev # Truncate a string up to the last occurrence of a marker (removing the marker). Example: before: "this:is:an:example", after: "this:is:an".
echo "this:is:an:example" | rev | cut -d ":" -f1 | rev # Truncate a string after the last occurrence of a marker (removing the marker). Example: before: "this:is:an:example", after: "example".

Return a text string:
echo "STRING" # print STRING
echo -n "STRING" # print STRING without new line
echo {A,B,C}{A,B,C} # return all possible combinations of given characters (this example returns AA AB AC BA BB BC CA CB CC )

Text attributes:
tput bold; echo "Bold Text"; tput sgr0;
tput smul; echo "Underlined Text"; tput rmul;
tput bold; tput smul; echo "Bold and Underlined Text"; tput rmul; tput sgr0;
tput dim; echo "Dimmed (Half Bright) Text"; tput sgr0;
tput blink; echo "Blinking Text"; tput sgr0; # Note: most terminals don't support blinking text and will show a reverse color text instead.
tput rev; echo "Reverse Color Text"; tput sgr0;
tput smso;  echo "Standout Text"; tput rmso; # The visual output of a standout text depends on the terminal: generally is rendered as a reverse color text or a bold text.
tput sgr0 # Turn off all attributes.
txtbold=$(tput bold); txtnorm=$(tput sgr0); echo "Normal Text, ${txtbold}Bold Text${txtnorm}, Normal Text again."

Colored text:
# Note: Colors are not a POSIX (Portable Operating System Interface) feature in the Bash shell, so it might not be shown in some systems

tput colors # Show the number of available colors in the terminal (generally 8)

Typical ANSI colors in 8 color terminals:

0: Black
1: Red
2: Green
3: Yellow
4: Blue
5: Magenta
6: Cyan
7: White
8: Not used
9: Reset to default color

tput setaf COLOR # Set foreground color (ANSI).
tput setab COLOR # Set background color (ANSI).

tput setab 1; tput setaf 3; echo -n "TEXT"; tput setaf 9; tput setab 9; echo; # Type TEXT in yellow over red

# Escaped text colors
echo -e "\e[1;37m WHITE  \e[0;37m LIGHT GRAY  \e[1;30m GRAY  \e[0;30m BLACK  \e[0;31m RED  \e[1;31m LIGHT RED  \e[0;32m GREEN  \e[1;32m LIGHT GREEN  \e[0;33m YELLOW  \e[1;33m LIGHT YELLOW  \e[0;34m BLUE  \e[1;34m LIGHT BLUE  \e[0;35m PURPLE  \e[1;35m PINK  \e[0;36m CYAN  \e[1;36m LIGHT CYAN  \e[0m DEFAULT";

Remove Escaped colors and other control characters from a text file:
cat -v PATH/FILE | sed "s/\^\[\[.\;..m//g" | sed "s/\^\[\[.m//g" | sed "s/\^M$//g"

Type a text string (y by default) until interrupted:
yes # Type y until interrupted
yes n # Type n until interrupted
yes hello world | head --lines=5 # Type hello world 5 times
yes | COMMAND # automatically reply with "y" to all questions prompted by the COMMAND
yes n | COMMAND # automatically reply with "n" to all questions prompted by the COMMAND

Convert character encoding:
apt-get install recode
recode iso-8859-15..utf8 < FILE.TXT > file.utf8 # recode FILE.TXT from Latin Western Europe to UTF8
recode utf8..iso-8859-15 < FILE.UTF8 > file.txt # recode FILE.UTF8 from UTF8 to Latin Western Europe
recode ../b64 < FILE.TXT > file.b64 # recode FILE.TXT to Base 64
recode /qp.. < FILE.TXT > file.qp # recode FILE.TXT to quoted printable

Convert New Line Format:
sed -i "s/\r//" PATH/SOURCE_FILE>PATH/DESTINATION_FILE # From \r\n (Windows) to \n (Unix/Linux/Mac)
sed -i "s/\n/\r\n/" PATH/SOURCE_FILE>PATH/DESTINATION_FILE # From \n (Unix/Linux/Mac) to \r\n (Windows)

Convert tabs to spaces in a text file:
expand PATH/FILE  # convert only tabs at the beginning of the line
expand -a PATH/FILE  # convert all tabs, not just the ones at the beginning of the lines

Convert spaces to tabs in a text file:
unexpand PATH/FILE  # convert all tabs, not just the ones at the beginning of the lines
unexpand -i PATH/FILE # convert only spaces at the beginning of the lines

Search and Replace:
sed "s/SOURCE_STRING/REPLACEMENT/g" PATH/SOURCE_FILE>PATH/DESTINATION_FILE # Replace all occurrences of a string inside a text file, case sensitive
sed "s/SOURCE_STRING/REPLACEMENT/gi" PATH/SOURCE_FILE>PATH/DESTINATION_FILE # Replace all occurrences of a string inside a text file, case insensitive
sed "s/[Bb]ig/small/g" PATH/SOURCE_FILE>PATH/DESTINATION_FILE # Replace the word Big or big to small
sed "10,20 s/SOURCE_STRING/REPLACEMENT/gi" PATH/SOURCE_FILE>PATH/DESTINATION_FILE # Replace all occurrences of a string inside a text file, case insensitive, and process only lines 10 to 20 included
sed "/MATCH_STRING/ s/SOURCE_STRING/REPLACEMENT/g" PATH/SOURCE_FILE>PATH/DESTINATION_FILE # Replace SOURCE_STRING with REPLACEMENT only in lines where MATCH_STRING is present
sed 's/TAG1.*TAG2/TAG1 REPLACEMENT TAG2/g' PATH/SOURCE_FILE>PATH/DESTINATION_FILE # Replace anything between TAG1 and TAG2 keeping the tags, case sensitive
sed 's/TAG1.*TAG2/TAG1 REPLACEMENT TAG2/gi' PATH/SOURCE_FILE>PATH/DESTINATION_FILE # Replace anything between TAG1 and TAG2 keeping the tags, case insensitive
sed 's/TAG1.*TAG2/REPLACEMENT/g' PATH/SOURCE_FILE>PATH/DESTINATION_FILE # Replace anything between TAG1 and TAG2 removing the tags, case sensitive
sed 's/TAG1.*TAG2/REPLACEMENT/gi' PATH/SOURCE_FILE>PATH/DESTINATION_FILE # Replace anything between TAG1 and TAG2 removing the tags, case insensitive
find . \( -type d -name .git -prune \) -o -type f -name "*.txt" -print0 | xargs -0 sed -e 's/SOURCE_STRING/REPLACEMENT/g' # Replace all occurrences of SOURCE_STRING with REPLACEMENT (case sensitive) in all files which names ends with .txt inside the current directory and all its subdirectories except for the subdirectories called .git . This is a dry run ( -e flag for sed will print the result on standard output, without actually writing to the file )
find . \( -type d -name .git -prune \) -o -type f -name "*.txt" -print0 | xargs -0 sed -i 's/SOURCE_STRING/REPLACEMENT/g' # Replace all occurrences of SOURCE_STRING with REPLACEMENT (case sensitive) in all files which names ends with .txt inside the current directory and all its subdirectories except for the subdirectories called .git . This will actually replace the text in the files

Replace multiple spaces with a single space in a text file:
sed "s/  */ /g" PATH/SOURCE_FILE>PATH/DESTINATION_FILE

Replace multiple spaces with a single space, and add a space before of every character in a text file:
sed "s/ */ /g" PATH/SOURCE_FILE>PATH/DESTINATION_FILE

Remove comments from a text file:
sed "/ *#/d;" PATH/SOURCE_FILE>PATH/DESTINATION_FILE

Remove blank lines from a text file:
sed "/^ *$/d" PATH/SOURCE_FILE>PATH/DESTINATION_FILE

Remove comments and blank lines from a text file:
sed "/ *#/d; /^ *$/d" PATH/SOURCE_FILE>PATH/DESTINATION_FILE

Add a text at the end of each line:
sed "s/$/\;/g" PATH/SOURCE_FILE>PATH/DESTINATION_FILE  # Append a semicolon (";") at the end of each line

Add a text at the beginning of each line:
sed "s/^/LINE : /g" PATH/SOURCE_FILE>PATH/DESTINATION_FILE  # Add "LINE : " at the beginning of each line

Remove blank lines (empty lines) in a text file:
sed '/^$/d' PATH/SOURCE_FILE>PATH/DESTINATION_FILE
sed '/^[[:space:]]*$/d' PATH/SOURCE_FILE>PATH/DESTINATION_FILE # Remove all blank lines, including the ones that are not actually empty but contains spaces and/or tabs

Add blank lines after every line in a text file:
sed "G" PATH/SOURCE_FILE>PATH/DESTINATION_FILE # Add 1 blank line after every line in a text file
sed "G;G" PATH/SOURCE_FILE>PATH/DESTINATION_FILE # Add 2 blank lines after every line in a text file

Append a line of text to a file:
echo "line 1" >> PATH/FILE.txt

Append multiple lines of text to a file (in scripts):
echo "line 1
line 2" >> PATH/FILE.txt

Append multiple lines of text to a file (in scripts):
cat <> PATH/FILE.txt
line 1
line 2
EOT

Case conversion:
echo "Test"|tr '[:lower:]' '[:upper:]' # Return the text in UPPER case
echo "Test"|tr '[:upper:]' '[:lower:]' # Return the text in lower case
cat PATH/SOURCE_FILE|tr '[:lower:]' '[:upper:]'>PATH/DESTINATION_FILE # Convert a text file to Upper case
cat PATH/SOURCE_FILE|tr '[:upper:]' '[:lower:]'>PATH/DESTINATION_FILE # Convert a text file to lower case

Extract items from a delimited text file:
cut -d: -f1 PATH/SOURCE_FILE>PATH/DESTINATION_FILE  # choose the first item of a series delimited by ":"

Remove consecutive duplicate lines from a text file:
uniq PATH/FILE
uniq -u PATH/FILE # remove only successive duplicated lines (all occurrences)

Reverse lines of a file
rev PATH/FILE.txt > PATH/REVFILE.txt # Reverse the lines of a given file(s)

Get the checksum of a file:
sum PATH/FILE # return the checksum and block count for FILE
sha1sum PATH/FILE # return the SHA1 (160-bit) checksum for FILE
cksum PATH/FILE # return the CRC checksum and byte count for FILE

Compare and show differences between two binary files:
cmp -b -l PATH/FILE1 PATH/FILE2 # compare two files and returns the position of the different bytes followed by the value and the corresponding ASCII character for each file
cmp -b -l --bytes=10 PATH/FILE1 PATH/FILE2 # limit comparison to the first 10 bytes

Compare and show differences between two text files:
diff -abB PATH/FILE1 PATH/FILE2 # Report differences between two text files
diff -abBi PATH/FILE1 PATH/FILE2 # Report differences between two text files ignoring case differences in their content

Compare and show differences between three text files:
diff3 -e FILE1 FILE2 FILE3 # output unmerged changes from FILE2 to FILE3 into FILE1

Truncate a file:
cp /dev/null PATH/FILE # empty FILE. If FILE doesn't exist, it will be created.
truncate -s 0 PATH/FILE # truncate FILE to 0 bytes (empty FILE). If FILE doesn't exist, it will be created.
truncate -s 0 -c PATH/FILE # truncate FILE to 0 bytes (empty FILE). If FILE doesn't exist, it will not be created.
truncate -s 100 PATH/FILE # truncate FILE to 100 bytes. If FILE doesn't exist, or it's shorter than the given size, it will be filled by zeroes (0).
truncate -r PATH/REF_FILE PATH/FILE # truncate FILE to the size of REF_FILE.

Split or limit lines in a text file:
cut -c 80 PATH/FILE.txt # cut every line of FILE.txt to 80 characters (extra characters will be lost)
fold -w 80 PATH/FILE.txt # split every line of FILE.txt after 80 columns (extra characters will be wrapped to the next line)
fold -w 80 -s PATH/FILE.txt # split every line of FILE.txt after 80 columns, breaking at spaces (full words will be preserved)

Delete lines from a text file:
sed -e '10d' PATH/FILE.txt > PATH/DESTFILE.txt # remove line 10
sed -e '5,15d' PATH/FILE.txt > PATH/DESTFILE.txt # remove lines 5 to 15

Empty files:
# Delete the content of a file without removing the file. It can be useful to don't break programs which expect to find the file, or before to delete a very large file.
cp /dev/null > PATH/NAME # Empty a file. It will also work as "> PATH/NAME" (without quotes)
echo -n '' > PATH/NAME # Empty a file
find PATH -type f -exec cp /dev/null {} \; # Empty all files in PATH and its subdirectories
find PATH -type f -regex ".*\.EXT$" -exec cp /dev/null {} \; # Empty all files in PATH and its subdirectories with given EXTension
find PATH -type f -regex ".*\.[0-9]*$" -exec cp /dev/null {} \; # Empty all files in PATH and its subdirectories with a numeric extension

Delete files:
rm PATH/NAME
rm -i PATH/NAME # ask for confirmation before to delete
find PATH -maxdepth 1 -type f -empty -delete # Delete all empty files in the given PATH (also: find PATH -maxdepth 1 -type f -size 0 -delete )
find PATH -maxdepth 1 -type f ! -empty -delete # Delete all non empty files in the given PATH (also: find PATH -maxdepth 1 -type f -size +0 -delete )
find PATH -type f -empty -delete # Delete all empty files in the given PATH and its subdirectories (also: find PATH -type f -size 0 -delete )
find PATH -type f ! -empty -delete # Delete all non empty files in the given PATH and its subdirectories (also: find PATH -type f -size +0 -delete )
find /var/www/test/t5 -type f -delete # Delete all files in /var/www/test/t5 and its subdirectories
find /var/www/test/t5 -type f -regex ".*\.EXT$" -delete # Delete all files in /var/www/test/t5 and its subdirectories with given EXTension
find /var/www/test/t5 -type f -regex ".*\.[0-9]*$" -delete # Delete all files in /var/www/test/t5 and its subdirectories with a numeric extension

Wipe a file (secure file deletion):
# Important note: wiping overwrites a file one or more time before to delete it, to make it unrecoverable. This technique does not work with journaled file systems, like ReiserFS, Reiser4, or ext3 in journaled mode. It also doesn't work with solid state drives (such as SD cards).
apt-get install wipe # if it doesn't work, try apt-get install secure-delete
wipe PATH/FILE # wipe a file (asks for confirmation, write protected files will be spared)
wipe -c -f PATH/FILE # wipe a file
wipe -c -f -s PATH/FILE # wipe a file, quiet mode (no output returned except for fatal errors like "No such file or directory")
wipe -c -f -k PATH/FILE # wipes the content of a file, but keeps the file in place (with a lenght of 0 bytes)
wipe -c -f -e PATH/FILE # wipes the file up to its exact size, without wiping extra data in the last used block

Undelete a file:
apt-get install testdisk # PhotoRec
photorec MEDIA # attempt to recover deleted files from given MEDIA (for example /dev/sda2 ). You'll have then to confirm the media, choose partition, select recovery options, specify the file type of the files you want to recover, specify the filesystem type where the files were stored, choose if you want to search for deleted files in the empty space or the whole partition, choose the destination directory (in a separate disk) where you want to save the recovered files.

Create a directory:
mkdir PATH
mkdir -p PATH # Create the whole path (not just the final directory in the path) if it doesn't exist

Delete a directory:
rmdir PATH # Delete an empty directory with no subdirectories
rmdir -r PATH # Delete an empty directory with empty subdirectories (recursive)
find PATH -depth -type d -empty -exec rmdir {} \; # Delete an empty directory with empty subdirectories (recursive)

Prune a directory tree (remove a directory with all its subdirectories, even if not empty, recursive):
rm -rf PATH ; rm -rf PATH/.* # Prune a directory tree
mkdir /tmp/empty/ ; rsync -a --delete /tmp/empty/ PATH/DIRECTORY_TO_REMOVE/ ; rmdir PATH/DIRECTORY_TO_REMOVE # Delete a directory or prune a directory tree with a large number (thousands and more) of files. Note: you can use any empty directory, just make sure it's actually empty.

Wipe a directory tree (recursive secure file deletion):
apt-get install wipe
wipe -c -r PATH # wipe a directory tree (asks for confirmation)
wipe -c -f -r PATH # wipe a directory tree
wipe -c -f -r -s PATH # wipe a directory tree, quiet mode (no output returned except for fatal errors like "No such file or directory")

Show current directory:
pwd

Change directory:
cd PATH
cd .. # go to the parent directory (that is, one directory up)
cd - # go to the previous directory (which is stored in $OLDPWD)
echo $OLDPWD # see previous directory
pushd PATH # change directory and remember the current directory in the dirs stack
popd # return to the previous directory in the dirs stack
dirs # (or dirs -l ) list directories in the dirs stack keeping the output in a single line
dirs -p # list directories in the dirs stack showing the output in multiple lines
dirs -v # list directories in the dirs stack showing the output in multiple lines and line numbers
dirs -c # clear the dirs stack

Show the content of a directory:
ls PATH
ls -l -d PATH # show only the properties of a directory, not its content.
ls -laF --color=auto PATH # extended information and colored output
ls -laFXB PATH # sort by extension
ls -laFSr PATH # sort by size, small to large
ls -laFS PATH # sort by size, large to small
ls -laFtr PATH # sort by size, old to recent
ls -laFt PATH # sort by date, recent to old
ls -laFtur PATH # sort by access date, old to recent
ls -laFtu PATH # sort by access date, recent to old
ls -A # show hidden files
ls --group-directories-first # show directories first
ls -d */. # list only directories within current directory
ls -d PATH/*/. # list only directories within given PATH
ls -R PATH | grep ":$" | sed -e 's/:$//' -e 's/[^-][^\/]*\//--/g' -e 's/^/ /' -e 's/-/|/' # show directory tree (add | less at the end of the line to navigate the tree up and down)
find PATH -ls # The -ls parameters make find return results as the output of ls . You may also add the -print0 parameter and pipe to xargs -0 ls -lt (that is: -print0 | xargs -0 ls -lt )
find PATH -type f -mtime -30 # list files modified within last 30 days
find PATH -depth -type d -empty -exec echo {} \; # list empty subdirectories within given PATH (recursive)
find PATH -maxdepth 1 -type f -empty # Find all empty files in the given PATH (also: find PATH -maxdepth 1 -type f -size 0 )
find PATH -maxdepth 1 -type f ! -empty # Find all non empty files in the given PATH (also: find PATH -maxdepth 1 -type f -size +0 )
find PATH -type f -empty # Find all empty files in the given PATH and its subdirectories (also: find PATH -type f -size 0 )
find PATH -type f ! -empty # Find all non empty files in the given PATH and its subdirectories (also: find PATH -type f -size +0 )
find PATH/DIRECTORY -type f | perl -ne 'print $1 if m/\.([^.\/]+)$/' | sort -u # return all different file extensions in given DIRECTORY

Compare and show differences between two directories:
diff PATH/DIR1 PATH/DIR2 # Report different files between two directories
diff -q PATH/DIR1 PATH/DIR2 # Report different files between two directories without showing the different content of files with identical file names
diff -s PATH/DIR1 PATH/DIR2 # Report different files and identical files between two directories
diff --ignore-file-name-case PATH/DIR1 PATH/DIR2 # Report different files between two directories ignoring case differences in file names
diff -i PATH/DIR1 PATH/DIR2 # Report different files between two directories ignoring case differences in the content of text files
diff -r PATH/DIR1 PATH/DIR2 # Report different files between two directories and all their subdirectories
dir1="PATH/DIR1"; dir2="PATH/DIR2"; diff -q "$dir1" "$dir2" | grep "$dir2" | sed "s/$dir2\///g" | awk '{print $4}' # Report different files and files that are only present in DIR2

Pick a random file from a directory:
ls DIR | shuf -n 1 # Pick a random file from the given DIR directory
ls DIR -pR | grep -v / | sed '/^$/d' | shuf -n 1 # Pick a random file from the given DIR directory and its subdirectories
find DIR -maxdepth 1 -type f | shuf -n 1 # Pick a random file (including path) from the given DIR directory
find DIR -type f | shuf -n 1 # Pick a random file (including path) from the given DIR directory and its subdirectories

Change access permissions:
# Users: uuugggooo (ugo = u: owner user, g: all the users of the group, o: all other users)
# Modes: 0: ---, 1: --x, 2: -w-, 3: -wx, 4: r--, 5: r-x, 6: rw- , 7: rwx
chmod MOD FILE # Change permissions for a file
chmod -v MOD FILE # Change permissions for a file, with verbose output
chmod -R MOD DIRECTORY # Change permissions for a directory tree (recursive)
chmod -vR MOD DIRECTORY # Change permissions for a directory tree (recursive), with verbose output
find PATH -type f -exec chmod MOD {} + # Change permissions for all files (only files, not directories) in a directory tree (recursive)
find PATH -type d -exec chmod MOD {} + # Change permissions for all directories (only directories, not files) in a directory tree (recursive)

Change owner:
chown OWNER FILE # Change the owner of a file
chown OWNER:GROUP FILE # Change the owner and the group of a file
chown -R OWNER DIRECTORY # Change the owner of a directory tree (recursive)
chown -R OWNER:GROUP DIRECTORY # Change the owner and a group of a directory tree (recursive)

Change group:
chgrp OWNER FILE # Change the group of a file
chgrp -R OWNER DIRECTORY # Change the group of a directory tree (recursive)

File Permissions Backup and Restore
getfacl -R / > PATH/fsperms.txt # Back up file system permissions to a text file
setfacl --restore=PATH/fsperms.txt # Restore file system permissions from a text file

Raw copy:
dd if=/dev/fd0 of=/dev/fd1 # copy a floppy disk (using two floppy disk drives)
dd if=/dev/hda0 of=/dev/hda1 # copy a hard disk (or partition) into another one
dd if=/dev/hda | gzip -9v | dd of=/mnt/hdb/hda.img # back up the whole hard disk as a gzipped file (you may need to mount a partition of another hard disk as hdb: mount /dev/hdb2 /mnt/hdb )
dd of=/mnt/hdb/hda.img | gzip | if=/dev/hda # restore the gzipped image of a hard disk
dd if=/dev/fd0 of=/tmp/disk.bak # copy the whole floppy disk as a file
dd if=/dev/hda of=/tmp/mbr.bin count=1 bs=512 # back up MBR (boot loader and partition table)

Create and edit disk partitions:
apt-get install parted
parted # starts the partition utility
parted commands:
print # show current partitions
mkpart primary START END # create a new primary partition
mkpart extended START END # create a new extended partition
mkpart logical START END # create a new logical partition (within the range of an extended one)
resize PARTITON_N NEW_START NEW_END # resize an existing partition
rm PARTITION_N # remove a partition
quit # quit parted

Show available devices:
fdisk -l
fdisk -l --bytes # show size in bytes rather than human readable format
lsblk # list information about all available devices
lsblk -f # 
lsblk -a # list information about all available devices including empty devices and RAM disk devices
lsblk -b # list information about all available devices, show size in bytes rather than human readable format
blkid # show information about available block devices

Show device partitions:
fdisk -l /dev/DEVICE

Make a file system (format) a disk (partition):
mkfs.ext3 /dev/hda1 # Format hda1 as an ext3 partition
mkfs -V -t ext3 /dev/hda1 # Format hda1 as an ext3 partition (Verbose output)
mkfs -t msdos  /dev/hda2 # Format hda1 as a MS-DOS partition
mkfs -t msdos /dev/fd0 # Format a floppy disk using the MS-DOS file system
mkswap /dev/hda2 # Format hda2 as the swap partition

dd if=/dev/zero of=/dev/foo1 bs=512 count=1 # If you created or changed a DOS partition, you can use dd to zero the first 512 bytes

SMART Devices monitoring:
apt install smartmontools
smartctl -V # Show smartctl version and configure arguments
service smartd start ; chkconfig smartd on # Enable Service
smartctl -s on /dev/sda # Enable SMART on /dev/sda (use lsblk and fdisk -l to see all devices in your system). Some devices such as Virtual Disks do not support SMART
smartctl -i /dev/sda # Print SMART information about /dev/sda
smartctl --all /dev/sda # Print SMART information and data about /dev/sda
smartctl -t short /dev/sda # Short Test (it usually teakes minutes to complete)
smartctl -t long /dev/sda # Long Test (it usually teakes hours to complete)
smartctl -H /dev/sda # Check if the hard drive passed or failed last test

Create a RAMDisk:
mkdir -p /media/ramdisk
mount -t tmpfs -o size=2048M tmpfs /media/ramdisk # 2GB RAMDisk, it can use swap partition if needed, and doesn't preallocate disk space.
mount -t ramfs -o size=1024M ramfs /media/ramdisk # 1GB RAMDisk, it only uses physical RAM, not the swap partition, and preallocates disk space. Make sure you have enough spare RAM.
umount /media/ramdisk # unmount RAMDisk

Access a XFS partition:
# You typically need this to read disks from NAS devices, via USB.
mkdir /ramdisk/xfs1 # If you are using a Linux boot disk, like Knoppix, you'll probably need to mount the XFS partition in the RAMDisk
fdisk -l # Use fdisk to locate the XFS partition
mount -t xfs /dev/XFS_PARTITION /ramdisk/xfs1 # use the path to XFS partition you've found using fdisk instead of /dev/XFS_PARTITION
cp -rp /ramdisk/xfs1/* /DESTINATION # copy all files from the XFS formatted partition to another device ( /DESTINATION )

Check and repair a file system:
fsck
fsck -f -y FILE_SYSTEM # Note: Generally the device containing the file system to check can be /dev/hda0 on a single hard drive system or /dev/md0 on a RAID system); -y automatically reply yes (y) to all questions. Note that if it asks to connect to lost+found more than once, you may have some damaged files.

List RAID arrays:
mdadm --examine --scan

Check if a RAID array is up and working properly:
mdadm --detail /dev/md0 # assuming your RAID volume is /dev/md0

Check if two devices are in a RAID array:
mdadm --examine --scan /dev/sda1
mdadm --examine --scan /dev/sdb1
# Must return same MD array and UUID. Path to actual devices in your system may differ from /dev/sda1 and /dev/sdb1 of this example

Copy partitions from a device to another one:
sfdisk -d /dev/sda | sfdisk /dev/sdb # copy partitions from sda to sdb

Choose what partitions should be automatically mounted at system startup:
jed /etc/fstab # edit static file system information

Show free disk space: 
df -T # Show disk usage for all filesystems
df -Th # Show disk usage for all filesystems in human readable format(in Megabytes, Gigabytes...)
df -Th / # Show disk usage for the filesystem mounted on root in human readable format (Megabytes, Gigabytes...)
df -h / | xargs | awk '{print "Disk ("$8"): Size: " $9 " Used: " $11 " Free: " $10}'; # Show disk usage for the filesystem mounted on root in a easily human readable format
df -P -h | awk '0+$5 >= 90 {print "FS: "$1" ("$6") Size: "$2" Used: "$3" (\033[1;31m"$5"\033[0m) Free: "$4" (\033[1;31m"(100-$5)"%\033[0m)";}' # Only show near full file systems

Summarize disk usage:
# du shows size in KBytes by default (switch: -k), if you want to see the size in bytes, use the switch -b
du PATH/FILE # disk usage for specified file
du -bs PATH/DIRECTORY | awk '{print $1}' | tr -d '\n' # Total disk usage in bytes of the DIRECTORY in the given PATH
du -s PATH # disk usage for the whole directory (recursively), total size
du -s --exclude=*.mp3 PATH # disk usage for the whole directory (recursively), total size, excluding all files ending with .mp3
du PATH # disk usage for the whole directory (recursively)
du -a PATH # disk usage for the whole directory (recursively), showing disk usage for all files and directories, not just directories
du -S PATH # disk usage for the whole directory (recursively, but size is separated for each directory)
du -a PATH | sort -nr | less # show directories and files sorted by size for the given PATH and its subdirectories
du -a PATH | sort -nr | head -n 25 # show the 25 largest directories and files within the given PATH . Use root ( / ) as PATH to see the largest directories and files in the whole disk
find PATH -type f -exec file -b '{}' \; -printf '%s\n' | awk -F , 'NR%2 {i=$1} NR%2==0 {a[i]+=$1} END {for (i in a) printf("%12u %s\n",a[i],i)}' | sort -nr # show the size occupied by files within a given PATH sorted by file type (slow)
du -cs PATH/FILE* # show size of files matching a pattern
du -cs PATH/FILE* | tail --lines=1 | awk '{print $1}' # show total size of files matching a pattern
du -cs /var/log/apache2/*.gz | grep "$(printf '\t')total" # show total size of apache 2 compressed log files
du -csh PATH/FILE* # show size of files matching a pattern in human readable format
du -csh PATH/FILE* | tail --lines=1 | awk '{print $1}' # show total size of files matching a pattern in human readable format
du -csh /var/log/apache2/*.gz | grep "$(printf '\t')total" # show total size of apache 2 compressed log files in human readable format

Show deleted files locked by a running process:
# If you can't see more free space in your disk after deleting a large file it's probably locked by a running process so that it has been unlinked but not actually deleted yet. In this case you'll have to kill, stop, or restart the process to unlock the file and free its disk space
lsof +L1

Count files and subdirectories within directories:
ls -1 /PATH/ | wc -l # count files and subdirectories within a directory
ls -1R /PATH/ | wc -l # count files and subdirectories within a directory including subdirectories (recursively)
ls -1 /PATH/ | grep -i .*.EXT | wc -l # count files and subdirectories with a given EXTension within a directory
ls -1R /PATH/ | grep -i .*.EXT | wc -l # count files and subdirectories with a given EXTension within a directory including subdirectories (recursively)

Count files within directories:
find PATH -maxdepth 1 -type f | wc -l # count files within a directory
find PATH -type f | wc -l # count files within a directory including subdirectories (recursively)
find PATH -maxdepth 1 -type f | grep -i .*.EXT | wc -l # count files with a given EXTension within a directory
find PATH -type f | grep -i .*.EXT | wc -l # count files with a given EXTension within a directory including subdirectories (recursively)

Show mounted devices:
mount

Mount a device:
mount DEVICE # example: mount /dev/cdrom
mount -t FILESYSTEM DEVICE MOUNTPOINT # Mount a DEVICE at MOUNTPOINT as a given FILESYSTEM; example: mount -t ext3 /dev/sda1 /test

Mount an ISO image of a CD:
mount -o loop cd.iso /mnt/DIR_NAME/

Unmount a device:
umount DEVICE # example: umount /dev/cdrom 

Unmount and eject a device (like a CD-ROM):
eject # eject default device
eject -d # show default device
eject -r # eject CD-ROM

Force changed blocks cached in RAM to disk:
sync

-----

Mount a remote directory tree:
apt-get install sshfs fuse-utils

To let other users to handle the remote directory tree:
# create a group called fuse containing users allowed to mount the remote directory tree
chgrp fuse /usr/bin/fusermount
chmod u+s /usr/bin/fusermount
adduser USER fuse

Create the local directory where the remote directory tree will be mounted:
mkdir /mnt/sshdir
chown USER /mnt/sshdir

Mount a remote directory tree:
sshfs USER@HOST:/PATH /mnt/sshdir

-----

Configure boot devices and partitions:

# see GNU GRUB (GRand Unified Bootloader) configuration:
less /boot/grub/grub.cfg
# /boot/grub/grub.cfg replaces /boot/grub/menu.lst and it's not meant to be edited
# in distros other than Debian the path could be /boot/grub/grub.conf or /etc/grub.conf

# edit LILO (LInux LOader) configuration:
jed /etc/lilo.conf

-----

Print a file on a printer:
lpr -Pprinter PATH/FILE
lpr -Pprinter -#N PATH/FILE # print N copies of FILE on printer

List print queue:
lpq

Remove a job from the print queue:
lprm JOB_ID
lprm # remove all jobs from the print queue

-----

Indentation:

Indent C code (or similarly formatted code):
apt-get install indent

indent PATH/SOURCE -o PATH/DEST
indent -nprs PATH/SOURCE -o PATH/DEST # Doesn't add a space after an open bracket and before a closed one

sed -i 's/^[[:space:]]*//g' PATH/FILE # Remove indentation from any text file

Indent HTML code:
apt-get install hindent

hindent PATH/SOURCE.HTML>PATH/DEST.HTML
hindent -i 0 PATH/SOURCE.HTML>PATH/DEST.HTML # Remove indentation
hindent -c PATH/SOURCE.HTML>PATH/DEST.HTML # Force all tags to lowercase (otherwise all tags are forced to UPPERCASE by default)
hindent -c -i 2 PATH/SOURCE.HTML>PATH/DEST.HTML # Set indentation level to 2 and force all tags to lowercase
hindent -c PATH/SOURCE.HTML>PATH/DEST.HTML # Force all tags to lowercase
hindent -f PATH/SOURCE.HTML>PATH/DEST.HTML # Flow: dump only HTML tags discarding any data between
hindent -l PATH/SOURCE.HTML>PATH/DEST.HTML # List all HTML tags contained in the source document

-----

Scheduled jobs with CronTab:

# Edit CronTab file
crontab -e # Edit CronTab file using the default editor or the one set in export EDITOR=jed ; export VISUAL=jed ; # (or another editor)

Format:
# Line of comments start with the dash character (#)
minute (0-59) hour (0-23) day of month (1-31) month (1-12) day of week (0-6, Sunday=0) COMMAND

# Use asterisks to specify any occurrence of the time parameter
* * * * * echo "test" # execute the command every minute (not recommended)
30 * * * 1 echo "test" # execute the command every 30 minutes, every monday

# separate multiple istances of the same time parameter with a comma (,)
15,30,45 * * * 1,3,5 echo "test" # execute the command every 15 minutes, every monday, wednesday, friday

# separate two istances with a dash (-) to create a time range
0 0,12 * 6-9 1-5 echo "test" # execute the command at midnight amd at twelve, every day from monday to friday, during every month from June to September

# Percent character (%) must be escaped (\)
30 * * * 1 touch /PATH/file_$(date +\%Y\%m\%d).txt

# Normally, crontab sends a mail to the user with the output of the command executed, including errors. Append ">/dev/null 2>&1" to the command line to prevent this behaviour
* * * * * echo "test">/dev/null

# list all scheduled jobs
crontab -l

# erase the crontab file (not recommended because remove all jobs, including ones that might be necessary for the system, you'd better edit the crontab files and delete entries manually)
crontab -r

-----

Networking:

arp -a # show DNS, IP, and MAC address for all network interfaces
ifconfig -a # show DNS, IP, and MAC address for all network interfaces
ifconfig eth0 # show DNS, IP, and MAC address for all network interfaces # show DNS, IP, and MAC address for the given interface (eth0)

Change MAC Address of an interface (eth0):
ifconfig eth0 down
ifconfig eth0 hw ether xx:xx:xx:xx:xx:xx
ifconfig eth0 up

ifconfig # show local network interfaces
ifconfig INTERFACE IP_ADDRESS # change the IP address for a network interface
ifconfig INTERFACE down # disable a network interface (WARNING: if you are connected from a remote system and disable the network interface to which you are connected you'll be disconnected, and if there isn't another interface available or a script that sets the interface up again automatically, you'll need physical access to the computer to configure it)
ifconfig INTERFACE up # enable a network interface

iwconfig # show local wireless network interfaces
iwconfig INTERFACE IP_ADDRESS # change the IP address for a wireless network interface
iwconfig INTERFACE down # disable a wireless network interface (WARNING: if you are connected from a remote system and disable the network interface to which you are connected you'll be disconnected, and if there isn't another interface available or a script that sets the interface up again automatically, you'll need physical access to the computer to configure it)
iwconfig INTERFACE up # enable a wireless network interface

Wake On LAN (WOL):
apt-get install ethtool
ethtool -s eth0 wol g # Set Wake On LAN on the specified interface (eth0). The system must support Wake On LAN and it has to be enabled from the BIOS
ethtool -s eth0 wol d # Unset Wake On LAN on the specified interface (eth0)
apt-get install etherwake
etherwake xx:xx:xx:xx:xx:xx # Wake On LAN the remote system with the specified MAC address

Default Gateway:
route add default gw GATEWAY_IP # Set Default Gateway. GATEWAY_IP in most cases is 192.168.0.1 or 192.168.0.254 or 192.168.1.0 or 192.168.1.254 (should you make an educated guess)
ip r # Show Default Gateway
route -n | grep 'UG[ \t]' | awk '{print $2}' # Get the IP of the Default Gateway

# Specify DNS Servers to resolve domain names
jed /etc/resolv.conf
# OpenDNS servers
nameserver 208.67.222.222
nameserver 208.67.220.200

jed /etc/hosts # assign hosts to specific IPs

jed /etc/network/options
jed /etc/network/interfaces

/etc/init.d/networking restart

Show active Internet connections:
netstat -anp
netstat -an | awk '{print $5}' | grep -o "[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}" | egrep -v "(`for i in \`ip addr | grep inet |grep eth0 | cut -d/ -f1 | awk '{print $2}'\`;do echo -n "$i|"| sed 's/\./\\\./g;';done`127\.|0\.0\.0)" | sort -n | uniq -c | sort -rn # show open connections sorted by IP address
netstat -an | grep 'ESTABLISHED' | awk '{print $5}' | grep -o "[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}" | egrep -v "(`for i in \`ip addr | grep inet |grep eth0 | cut -d/ -f1 | awk '{print $2}'\`;do echo -n "$i|"| sed 's/\./\\\./g;';done`127\.|0\.0\.0)" | sort -n | uniq -c | sort -rn # show established connections sorted by IP address
netstat -anp | grep 'IP' # Show current connections for the given IP
netstat -anp | grep 'ESTABLISHED' | grep 'IP' # Show current estabilished connections for the given IP
netstat -plntu # show applications listening TCP/UDP sockets
netstat -plntu | grep /apache | awk '{print $4}' | rev | cut -d ":" -f1 | rev | uniq # Show ports on which Apache webserver is listening, if enabled
netstat -plntu | grep /apache | awk '{print $4}' | rev | cut -d ":" -f1 | rev | uniq | tr '\n' ',' | awk '{sub(/,$/, "")};1' # Show ports on which Apache webserver is listening, if enabled (comma separated)
netstat -plntu | grep /tor | awk '{print $4}' | rev | cut -d ":" -f1 | rev | uniq # Show ports on which TOR server is listening, if enabled
netstat -plntu | grep /mysqld | awk '{print $4}' | rev | cut -d ":" -f1 | rev | uniq # Show ports on which MySQL server is listening, if enabled
netstat -plntu | grep /sshd | awk '{print $4}' | rev | cut -d ":" -f1 | rev | uniq # Show ports on which SSH is listening, if enabled
netstat -plntu | grep /sendmail | awk '{print $4}' | rev | cut -d ":" -f1 | rev | uniq # Show ports on which sendmail is listening, if enabled
netstat -A inet -lnp # show only listening servers
netstat -ant | awk '{print $4 " " $6}' | grep ':22 ' | grep 'ESTABLISHED' | wc -l # count estabilished SSH connections
netstat -ant | awk '{print $4 " " $6}' | grep ':80 ' | grep 'ESTABLISHED' | wc -l # count estabilished HTTP connections
netstat -ant | awk '{print $4 " " $6}' | grep ':443 ' | grep 'ESTABLISHED' | wc -l  # count estabilished HTTPS connections
netstat -anp | grep -v ":443 " | grep -v ":80 " | grep 'ESTABLISHED' # show NOT HTTPS/HTTP estabilished connections (excluding ports 443, 80)

Show network sockets in use:
lsof -i tcp:80 # show all processes that are using port 80 TCP

Scan an IP for open ports:
echo "Open ports:"; for i in {1..65535}; do (echo < /dev/tcp/127.0.0.1/$i) &>/dev/null && echo $i; done # Scan localhost (127.0.0.1) for open ports

Show bandwidth usage:
apt-get install nload
nload # show traffic in real time

apt-get install iftop
iftop # show connections and their bandwidth usage

Network usage stats:

apt install vnstat

jed /etc/vnstat.conf # Configuration

vnstat -u -i eth0 # Create the database file. If the interface is not eth0 change it accordingly
chown vnstat:vnstat /var/lib/vnstat/eth0 # Set vnstat as the owner of the database file. If the interface is not eth0 type the file name accordingly

service vnstat start # Start vnstat

service vnstat stop # Stop vnstat

vnstat -h # Hourly Stats
vnstat -d # Daily Stats
vnstat -w # Weekly Stats
vnstat -m # Monthly Stats
vnstat -t # Top Transfers

Ping host:
ping HOST # ping a host until CTRL-C is pressed
ping -c N HOST # ping a host N times

Trace route to host:
traceroute HOST

Network Mapping:
apt-get install nmap

nmap -sS 192.0.2.0 # search for open ports in the given IP
nmap -sS 192.0.2.0/24 # search for connected IPs (and related open ports) in the given IP mask
nmap -sS -O -v 192.0.2.0 # return Operating System and uptime for the given IP

Port scanning:
apt-get install netcat

# scan TCP ports:
nc -vvn -z 192.0.2.2 1-80 # scan TCP ports from 1 upto 80 in the given IP address

# scan UDP ports:
nc -u -vvn -z 192.0.2.8 1-255 # scan UDP ports from 1 upto 255 in the given IP address

Calculate broadcast, network, Cisco wildcard mask, and host range
apt-get install ipcalc
ipcalc 192.0.2.0/24
ipcalc 192.0.2.0/255.255.255.0
ipcalc 192.0.2.0-192.0.2.100 # deaggregate address range

DNS lookup:
apt-get install host # If you want to use the host command

# Resolve Hostname to IP address
dig +short www.example.com | awk '{ print ; }'
getent hosts www.example.com | awk '{ print $1 ; }'
host www.example.com | awk '/has address/ { print $4 ; }'
nslookup www.example.com | awk '/^Address: / { print $2 ; }'

# Resolve Hostname to IP address (only get the first result if there are more then one IP)
dig +short www.example.com | awk '{ print ; exit }'
getent hosts www.example.com | awk '{ print $1 ; exit }'
host www.example.com | awk '/has address/ { print $4 ; exit }'
nslookup www.example.com | awk '/^Address: / { print $2 ; exit }'

# Resolve Hostname to IPv4 or IPv6 address
host www.example.com # return both IPv4 and IPv6 addresses
host -t A www.example.com # return the IPv4 addresses
host -t AAAA www.example.com # return the IPv6 addresses

# Reverse DNS lookup
host 192.0.2.0
nslookup 192.0.2.0

DNS queries:
dig www.example.com A # get the IP address of the given domain
dig www.example.com TXT # get text annotations about the given domain
dig www.example.com MX # get the mail servers of the given domain
dig www.example.com NS # get the authoritative DNS servers for the given domain
dig www.example.com ANY # get all DNS information for the given domain
whois example.com # Information about a domain name

# IP Tables:

# iptables work with IPv4 addresses. Use ip6tables to work with IPv6 addresses.

iptables -S # Show rules (doesn't resolve IPs)
iptables -L # List rules (resolve IPs)
iptables -L -n # List rules (doesn't resolve IPs)
iptables -L INPUT --line-numbers # List only INPUT rules together with their line numbers (resolve IPs)
iptables -L INPUT -n --line-numbers # List only INPUT rules together with their line numbers (doesn't resolve IPs)

iptables -I INPUT -s 192.0.2.100 -j DROP # Block a specific (single) IP
iptables -I INPUT -s 192.0.2.100/32 -j DROP # Block a specific (single) IP (CIDR 32 only includes a single IP)
iptables -I INPUT -s 192.0.2.100/255.255.255.255 -j DROP # Block a specific (single) IP (subnet mask 255.255.255.255 only includes a single IP)
iptables -I INPUT -s 192.0.2.0/255.255.255.0 -j DROP # Block a range of IPs using a subnet mask
iptables -I INPUT -s 192.0.2.0/24 -j DROP # Block a range of IPs using CIDR
iptables -I INPUT -s 192.0.2.100 -j DROP -m time --timestart 22:00 --timestop 23:00 # Block the given IP only for one hour (22:00 to 23:00) every day

iptables -I INPUT -s 192.0.2.100 -j REJECT --reject-with icmp-port-unreachable # Reject a specific (single) IP. The difference between DROP and REJECT is that REJECT returns an error packet, while DROP returns no reply. Generally, REJECT is preferred on local networks and DROP on the Internet.

iptables -D INPUT LINE_NUMBER # Delete the INPUT rule with specified LINE_NUMBER
iptables -D INPUT 1 # Delete the last INPUT rule
iptables -D INPUT -s 192.0.2.100 -j DROP; # Unblock the matching IP Address
iptables -D INPUT -s 192.0.2.0/255.255.255.0 -j DROP # Unblock the matching range of IPs using a subnet mask
iptables -D INPUT -s 192.0.2.0/24 -j DROP # Unblock the matching range of IPs using CIDR

iptables -S|grep " DROP"|awk '{sub(/-A /,"iptables -I ")}; 1' > iptablesdrop.sh ; chmod 777 iptablesdrop.sh ; # Back up Drop instances in IP Tables into a shell script
iptables -S|grep " REJECT"|awk '{sub(/-A /,"iptables -I ")}; 1' > iptablesdrop.sh ; chmod 777 iptablesdrop.sh ; # Back up Reject instances in IP Tables into a shell script
iptables -S|grep -E "( DROP| REJECT)"|awk '{sub(/-A /,"iptables -I ")}; 1' > iptablesdrop.sh ; chmod 777 iptablesdrop.sh # Back up Drop and Reject instances in IP Tables into a shell script
iptables -S|grep -E "( DROP| REJECT)"|grep -Ev "^-A f2b-"|awk '{sub(/-A /,"iptables -I ")}; 1' > iptablesdrop.sh ; chmod 777 iptablesdrop.sh # Back up Drop and Reject instances in IP Tables, excluding the ones created by fail2ban, into a shell script

iptables-save > /etc/iptables/rules.v4 # Back up all IP Tables for IPv4
iptables-restore < /etc/iptables/rules.v4 # Restore all IP Tables for IPv4
ip6tables-save > /etc/iptables/rules.v6 # Back up all IP Tables for IPv6
ip6tables-restore < /etc/iptables/rules.v6 # Restore all IP Tables for IPv6

Script to clear IPTables:
---
#!/bin/bash

# Clear iptables
iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -t raw -F
iptables -t raw -X
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT
---

Download files from the Internet:
# note: default ports: HTTP: 80, FTP: 21
wget "http://www.example.com/file.gz"  # download the document at the given URL to the local directory
wget "http://www.example.com/file.gz" -O PATH/downloaded_file.gz  # download the document at the given URL to the specified directory and with the specified file name
wget -S "http://www.example.com/file.gz"  # download the document at the given URL in the local directory, and save the HTTP header at the beginning of the file
wget -c --wait=2 --limit-rate=20k "http://www.example.com/file.gz" # continue an interrupted download, and pause 2 secods after getting every file and limit download speed to 20KBps
wget -t 50 "http://www.example.com/file.gz"  # retries up to 50 times if the download fails or it's incomplete
wget -t 0 "http://www.example.com/file.gz"  # retries forever if the download fails or it's incomplete, until it can get the requested file
echo 'wget "http://www.example.com/file.gz"'|at 02:00  # start a download at 2:00am
wget -O- "http://www.example.com/path/page.html" # dump the output of the page on the console (normally on the screen)
wget -q -O- "http://www.example.com/path/page.html"|grep 'a href'  # show all links within given document
wget -r -l1 "http://www.example.com/path/page.html"  # download the first layer of links from given document
wget -r -l1 --no-parent "http://www.example.com/path/page.html"  # download the first layer of links from given document, ignoring links that are not within the given URL path
wget -r -l1 --no-parent -nc "http://www.example.com/path/page.html"  # download the first layer of links from given document, ignoring links that are not within the given URL path, and avoid downloading files already present in the destination directory (useful when resuming an interrupted download)
wget "http://www.example.com/doc?id="{1..50} // Download all files called doc?id= ranging from 1 to 50 from the given URL
wget "http://www.example.com/doc?id="{01..50} // Download all files called doc?id= ranging from 01 to 50 from the given URL
wget -r -l0 "http://www.example.com/path/page.html"  # download a whole website to the local directory
wget --mirror "http://www.example.com/"  # download a whole website to the local directory
wget -r -l1 --no-parent -A "*.gif" "http://www.example.com/dir/"  # download all files matching a certain pattern (all GIF files in this case) from the specified URL
wget -U USER_AGENT_STRING "http://www.example.com/file.gz" # you may set the User Agent to pretend to get the file from the server with a normal browser

Dump the content of a webpage on the screen:
lynx -dump http://www.example.com/
lynx --source http://www.example.com/ # Show document source
lynx -head -dump http://www.elfqrin.com/ # Show document info (headers)
wget -qO- http://www.example.com/ | cat # Show document source
wget --server-response --spider -q http://www.example.com/ # Show document info (headers)
curl http://www.example.com/ # Show document source
curl -I http://www.example.com/ # Show document info (headers)

Run a script on the web server:
wget -O- "http://localhost/script.php" # run a script on the local server
wget -q -O- "http://localhost/script.php" # run a script on the local server hiding the output
wget -q -T 0 -O- "http://localhost/script.php" # By default wget drops connection after 900 seconds (15 minutes), in this case you'll get a "Read error (Connection timed out) in headers.  Retrying." and if the script always takes more than the given time, it will retry uselessly until the maximum allowed retries (by default 20, but it can be changed with -t). To avoid time outs you can set a longer time using -T followed by the number of seconds (or 0 for infinite time, in this case make sure the script is working or it will run forever and if it uses resources incrementally it may crash the server). Also, if you experienced a time out with wget while running a script on localhost you should better restart the webserver as the script may continue running despite of wget timing out and dropping the connection.
lynx -dump http://localhost/script.php >/dev/null # run a script on the local server (less reliable than wget for this purpose) hiding the output

Run a script:
php PATH/FILE.php # run a PHP script
php -r "echo 'Hello World';"; # Execute a PHP line in the shell
php -r 'echo "Hello World\n";'; # Execute a PHP line in the shell. Use quotes to print character sequences.
perl -e "print 'Hello World';"; # Execute a Perl line in the shell
python -c "print('Hello World')"; # Execute a Python line in the shell
python3 -c "print('Hello World')"; # Execute a Python3 line in the shell
ruby -e "puts 'Hello World'"; # Execute a Ruby line in the shell

# If you receive this error message from the PPP Daemon (PPPD)
# pppd: The remote system is required to authenticate itself but I couldn't find any secret (password) which would let it use an IP address.
# You can fix it adding the string "noauth" (or changing the existing string "auth" to "noauth") in /etc/ppp/options

-----

Mirror a disk through the network:

If you have two computers with identical hardware you can mirror a system into the other machine through the network
This is especially useful with clusters / virtual servers

Start both machines from a live CD (like Debian Live - https://www.debian.org/CD/live/ )
Connect both machine with a cross cable through their eth1 interfaces and set them this way:
root@source_pc# ifconfig eth1 192.168.0.1
root@destin_pc# ifconfig eth1 192.168.0.2

Then execute these commands on the machines to start transfers:
root@source_pc# dd if=/dev/sda bs=128M | nc _w2 192.168.0.2 9000
root@destin_pc# nc _l _p 9000 | dd of=/dev/sda

Depending from the size of the disk it may need many hours

-----

Maths:

Calculate the result of an expression:

# operators: + - * / %  < > = >= <= !=

echo $(( 10+5 )) # Sum
echo $(( 10-5 )) # Subtraction
echo $(( 10*5 )) # Multiplication
echo $(( 10/5 )) # Division
echo $(( 10**2 )) # Power
echo $(( 15%10 )) # Module

echo $(( (1+2)*3 )) # Result = 20

echo $(( 5>3 )) # Returns 1 (true)
echo $(( 3>5 )) # Returns 0 (false)

# Using expr, operators must be escaped and separated by a space

expr 10 \+ 5 # Sum
expr 10 \- 5 # Subtraction
expr 10 \* 5 # Multiplication
expr 10 \/ 5 # Division
expr 15 \% 10 # Module

expr \( 1 \+ 2 \) \* 3 # Result = 20

expr 5 \> 3 # Returns 1 (true)
expr 3 \> 5 # Returns 0 (false)

Return the absolute value of a number
echo $(( 2-3 )) | tr -d -

Show Multiplication Table:
for i in {1..9}; do for j in $(seq 1 $i); do echo -ne $i\x$j=$((i*j))\\t; done; echo; done;

Return the prime factors of a number:
factor NUMBER
factor NUMBER1 NUMBER2 NUMBER3... # return factors for the given numbers

Return a sequence of numbers:
echo {FIRST..LAST} # return a sequence of integer numbers from FIRST to LAST, separated by a space. Numbers can be negative, and LAST can be less than FIRST
printf '%s\n' {1..10} # return a sequence of integer numbers from FIRST to LAST, each in its own line of text (separated by new lines).
printf '%s,' {1..10} | rev | cut -c 2- | rev # return a sequence of integer numbers from FIRST to LAST, comma separated (rev and cut are used to remove the comma at the end of the string)
printf '%s, ' {1..10} | rev | cut -c 3- | rev # return a sequence of integer numbers from FIRST to LAST, comma and space separated (rev and cut are used to remove the comma and space at the end of the string)
for ((i=FIRST; i<=LAST; i+=INCREMENT)); do echo $i; done # create a sequence from FIRST up to LAST number with given INCREMENT
seq LAST # create a sequence up to LAST number (default FIRST is 1, INCREMENT is 1)
seq FIRST LAST # create a sequence from FIRST up to LAST number (default INCREMENT is 1)
seq FIRST INCREMENT LAST # create a sequence from FIRST up to LAST number with given INCREMENT. Numbers can be negative, and END can be less than START, if the INCREMENT is negative
seq -s "," FIRST INCREMENT LAST # create a sequence from FIRST up to LAST number with given INCREMENT. Numbers are separated by a comma (",") instead of a line break
seq -s "," -w FIRST INCREMENT LAST # create a sequence from FIRST up to LAST number with given INCREMENT. Numbers are separated by a comma (",") instead of a line break. Numbers with less digit than the greater number are padded with 0s (or the sign for negative numbers)
seq FIRST INCREMENT LAST | shuf # shuffle a sequence of numbers

PI:
apt-get install pi
pi N # Print the mathematical constant PI to the given number of decimal figures. Note that pi 0 returns an empty string, pi 1 returns "3.", pi 2 returns "3.1", pi 3 returns "3.14", and so on. Default parameter for pi is 100 (that is, if you enter "pi" you get the same result as "pi 100").

Random numbers:

echo $RANDOM  # return a random number between 0 and 32767 (16-bit integer)
echo $((RANDOM%(MAX+1-MIN)+MIN)) # return a random number between MIN and MAX
expr \( $RANDOM \% \( MAX \+ 1 \- MIN \) \+ MIN \) # return a random number between MIN and MAX


Base conversion:

apt-get install bc

echo 'ibase=10; obase=16; 155'|bc  # base conversion (155 from decimal to hexadecimal)
echo 'ibase=16; obase=2; F'|bc  # base conversion (F from hexadecimal to binary)
echo '2*(3+4)'|bc -l  # perform math operations


Measure conversion:

apt-get install units

units  # interactive
units "1 in" "cm"  # convert 1 inch to centimeters


Human Readable Values:

numfmt --to=iec-i --suffix=B BYTES # Convert the number of BYTES into a human readable value
numfmt --to=iec-i --suffix=B --padding=7 BYTES # Convert the number of BYTES into a human readable value, more instances will be padded
numfmt --to=iec-i --suffix=B --format="%.3f" BYTES # Convert the number of BYTES into a human readable value with 3 decimal digits precision 

-----

Time:

Set date and time:
date MMDDhhmmYYYY.ss

Set date from NTP (SNTP) server:
apt-get install ntpdate
ntpdate SERVER  # set the clock
ntpdate -q SERVER  # query only for the time, don't set the clock

SNTP servers:
pool.ntp.org # NTP Pool Project
ntp.ubuntu.com # Ubuntu
ntp1.inrim.it # INRiM, Torino, Italy
ntp.univ-lyon1.fr # University of Lyon, Lyon, France
vega.cbk.poznan.pl # Astrogeodynamic Observatory, Borowiec, Poland
bitsy.mit.edu # MIT, Cambridge, Massachusetts, USA
bonehed.lcs.mit.edu # MIT, Cambridge, Massachusetts, USA
clock.isc.org # Internet Systems Consortium, Palo Alto, California, USA

Start a NTP server:
apt-get install ntp-refclock (or another ntp* package)
jed /etc/ntp.conf
/etc/init.d/ntp-server restart

Date tokens:
%% : %
%a : Weekday's abbreviated name (for example, Mon)
%A : Weekday's full name (for example, Monday)
%b : Month's abbreviated name (for example, Jan; same as %h)
%B : Month's full name (for example, January)
%C : Century (for example, 21)
%d : Day of month (for example, 01)
%e : day of month, space padded; same as %_d
%F : Full date formatted as full year - month - day (same as %Y-%m-%d )
%g : last two digits of year of ISO week number (see %G)
%G : year of ISO week number (see %V); normally useful only with %V
%H : Hour in 24 hours format with padded 0 (00-23)
%I : Hour in 12 hours format with padded 0 (01-12)
%j : Day of year with padded 0 (001-366)
%k : Hour in 24 hours format with padded space ( 0-23)
%l : Hour in 12 hours format with padded space ( 1-12)
%m : Month with padded 0 (01-12)
%M : Minute with padded 0 (00-59)
%n : Newline
%N : Nanoseconds (000000000-999999999)
%p : AM or PM, uppercase
%P : AM or PM, lowercase
%R : 24-hour hour and minute (same as %H:%M )
%s : Seconds since Linux Epoch (1970-01-01 00:00:00 UTC)
%S : Second with padded 0 (00-60)
%t : Tab
%T : Time (same as %H:%M:%S )
%u : ISO Day of week (1-7: 1 is Monday, 7 is Sunday)
%U : Week number of year, with Sunday as first day of week and padded 0 (00-53)
%V : ISO Week number of the year, with Monday as first day of week and padded 0 (01-53)
%w : Day of week (0-6: 0 is Sunday, 6 is Saturday)
%W : ISO Week number of year, with Monday as first day of week and padded 0 (00-53)
%y : Last two digits of year (00-99)
%Y : Full Year
%z : Timezone as hhmm (e.g., -0200)
%:z : Timezone as hh:mm (e.g., -02:00)
%Z : Timezone's abbreviated name (e.g., CET)

Show current date and time:
date
date '+%a %d %b %Y %H:%M:%S %Z (UTC%:z)' # show day of the week, day of the month, month (short format), year (full digits), hour, minutes, seconds, timezone. Example: Wed 16 Sep 2020 21:18:15 CEST (UTC+02:00)
echo "$(date '+%a %d %b %Y %H:%M:%S %Z (UTC%:z)')"; # echo day of the week, day of the month, month (short format), year (full digits), hour, minutes, seconds, timezone

Show information about your local time:
timedatectl status

Show System Time Zone
cat /etc/timezone

Linux Epoch (1 January 1970 00:00:00)
date +"%s" # Timestamp: seconds from Epoch to current time
date -d "2000-01-01" +"%s" # Seconds from Epoch to the specified date (from 1 January 1970 to 1 January 2000 in this example)
date -d "2000-01-01 12:00:00" +"%s" # Seconds from Epoch to the specified date and time (from 1 January 1970 00:00:00 to 1 January 2000 12:00:00 in this example)
date -d @1474303375 # Converts seconds from Epoch to date

Dates:
date +"%F" -d now # Return current day in the specified format
date +"%F" -d yesterday # Return yesterday's day in the specified format
date +"%F" -d tomorrow # Return tomorrow's day in the specified format
date +"%F" -d last-week # Return last week's date on current day in the specified format
date +"%F" -d next-week # Return next week's date on current day in the specified format
date +"%F" -d last-month # Return last month's date on current day in the specified format
date +"%F" -d next-month # Return next month's date on current day in the specified format
date +"%F" -d last-year # Return last year's date on current day in the specified format
date +"%F" -d next-year # Return next year's date on current day in the specified format
date +"%F" -d last-WEEKDAY # Return the date of the last given weekday (for example last-monday) in the specified format
date +"%F" -d next-WEEKDAY # Return the date of the next given weekday (for example next-monday) in the specified format

Calculate dates:
date +"%F" -d "now +15 days" # Show the resulting date adding the specified time to the current date in the specified format (date only in this example)
date +"%F" -d "2000-01-01 +15 days" # Show the resulting date adding the specified time to the given date in the specified format (date only in this example)
date +"%F %H:%M:%S" -d "now +10 seconds" # Show the resulting date adding the specified time to the current date in the specified format (date and time in this example)
date +"%F %H:%M:%S" -d "2000-01-01 10:10:00 UTC +30 seconds" # Show the resulting date adding the specified time to the given date in the specified format (date and time in this example)

Difference between dates:
date1='20050901'; date2='20200912'; echo $(( ( $( date +%s -d "$date2" ) - $( date +%s -d "$date1" ) )/86400 )); # Difference between two dates in days
time1='10:08'; time2='11:10'; IFS=: read hour1 min1 <<< "$time1"; IFS=: read hour2 min2 <<< "$time2"; min1=$((10#$hour1*60 + 10#$min1)); min2=$((10#$hour2*60 + 10#$min2)); echo "$((min2 - min1))"; # Difference between two times in minutes
date1='2020-11-10 12:00:10'; date2='2020-11-30 17:56'; echo $((($(date -ud "$date2" +'%s') - $(date -ud "$date1" +'%s'))/60)) # Difference between two dates and times in minutes

Get the day of the week for a specified date:
date -d "2000-01-01" +"%A"

Extract Date and Time from Apache Logs:
head /var/log/apache2/access.log --lines=1 | sed 's#[^[]*[[]\([^]][^]]*\).*#\1#' # Get Date and Time (including Time Zone) of the first entry in the Apache Log access.log
head /var/log/apache2/access.log --lines=1 | sed 's#[^[]*[[]\([^]][^]]*\).*#\1#' | awk '{print $1}' # Get Date and Time (excluding Time Zone) of the first entry in the Apache Log access.log
tail /var/log/apache2/access.log --lines=1 | sed 's#[^[]*[[]\([^]][^]]*\).*#\1#' # Get Date and Time (including Time Zone) of the last entry in the Apache Log access.log
tail /var/log/apache2/access.log --lines=1 | sed 's#[^[]*[[]\([^]][^]]*\).*#\1#' | awk '{print $1}' # Get Date and Time (excluding Time Zone) of the last entry in the Apache Log access.log
date -d "$(echo 'dd/MMM/yyyy:hh:mm:ss' | sed -e 's,/,-,g' -e 's,:, ,')" "+%a %d %b %Y %H:%M:%S" # Convert Apache log date/time to a better human readable date
date -d "$(echo 'dd/MMM/yyyy:hh:mm:ss' | sed -e 's,/,-,g' -e 's,:, ,')" +"%s" # Convert Apache log date/time to Linux timestamp

Calculate how long takes a command to be executed:
time COMMAND COMMAND_PARAMETERS
time cat # works like a stopwatch: counts time until you send an EOF to stop it (which generally means until you press CTRL+D that closes the cat command)

Create filenames containing current date and time:
touch data_`date +%F`.bak # create a file with the name data_YYYY-MM-DD.bak
touch log_`date +%F_%H-%M-%S`.txt # create a file with the name log_YYYY-MM-DD_HH-MM-SS.txt

User and system times:
times

Pause:
sleep S # pause for S seconds

Wait for a key press:
read -rs -n 1 -p "Press any key to continue..." wkey # Wait for any key
read -rs -p "Hit ENTER to continue..." wkey # Wait for ENTER

Show a calendar:
# cal  # show dayweeks in a row, Sunday first (use -m to have Monday first)
# ncal # show dayweeks in a coloumn, Monday first
cal # show a calendar for the current month
cal -y # show a calendar for the current year
cal -y YEAR # show a calendar for the given year
cal MONTH YEAR # show a calendar for the given MONTH and YEAR
ncal -e YEAR # show date of Easter (for Western Churches)
ncal -o YEAR # show date of Orthodox Easter (Greek and Russian Orthodox Churches)

Memorable dates:
The directory /usr/share/calendar/ contains some collections of remarkable dates.
For example:
cat /usr/share/calendar/calendar.history # Important historical dates
cat /usr/share/calendar/calendar.music # Important dates in music history
cat /usr/share/calendar/calendar.lotr # Lord of the Rings dates

-----

Archiving and Compression:

Install archivers:
apt-get install bzip2
apt-get install zip unzip
apt-get install zoo
apt-get install lha
apt-get install rar
apt-get install unp

Archiving with ISO:
dd if=/dev/dvd of=dvd.iso # Create an ISO file from a DVD
dd if=/dev/cdrom of=cd.iso # Create an ISO file from a CD-ROM
dd if=/dev/scd0 of=cd.iso # Create an ISO file from a SCSI CD-ROM
mkisofs -o /PATH_DEST/cd.iso /PATH_SOURCE/ # Create an ISO file from a directory tree

Archiving with TAR:
tar -cvf file.tar text.txt # archive text.txt as file.tar
tar -cvf file.tar *.txt # archive all files ending with ".txt" in the current directory as file.tar
tar -xvf file.tar # extract all files from file.tar in the current directory

Compressed TAR archives (tarball):
# Use tar to compress a directory tree into a single file, and gzip or bzip2 to compress each file into a directory tree
tar -cjvf file.tbz text.txt # TAR+BZIP2 compression (canonical extension .tbz or .tar.bz2)
tar -xjvf file.tbz # extract all files from a TAR+BZIP2 compressed archive
tar -czvf file.tgz text.txt # TAR+GZIP compression (canonical extension .tgz or .tar.gz)
tar -xzvf file.tgz # extract all files from a TAR+GZIP compressed archive

Compression with GZIP:
gzip -9vk text.txt # compress text.txt as text.txt.gz using highest compression and verbose output, keeping the original uncompressed file
gzip -9v text.txt > file.gz # compress text.txt as file.gz using highest compression and verbose output, keeping the original uncompressed file
gzip -9v text.txt # compress text.txt as text.txt.gz using highest compression and verbose output, deleting the original uncompressed file 
gzip -r9v file.txt file2 file3 /home/docs # compress file.txt file2 file3 and every file in /home/docs (if such directory exists), deleting the original uncompressed files
gzip -l file.gz # show the content of the compressed file
gzip -dk file.gz # extract the content of file.gz in the current directory and keep the compressed file (file.gz)
gzip -d file.gz # extract the content of file.gz in the current directory and delete the compressed file (file.gz)

Compression with BZIP2:
bzip2 text.txt # compress text.txt as text.txt.bz2
bunzip2 file.bz2 # extract the content of file.bz2 into the current directory

Compression with ZIP:
zip file.zip file.txt # add file.txt to the compressed file file.zip (if it doesn't exist, it will created)
zip -r file.zip /home/docs # compress the whole content of the directory /home/docs into file.zip
zip -r filename.zip file.txt file2 file3 /home/docs # compress file.txt file2 file3 and the content of /home/docs (if such directory exists) into file.zip
zip -F brokenfile.zip # try to fix a broken zipped file (use -FF to try even harder)
unzip file.zip # extract the content of file.zip into the current directory
unzip file.zip -d PATH_DEST # extract the content of file.zip into the specified directory

Encrypted Compression with ZIP:
# Note that Zip encryption is not very strong
zip -e file.zip file.txt # encrypt file.txt into file.zip prompting for a password
zip -P PASSWORD FILE.zip file.txt # encrypt file.txt into file.zip with the given PASSWORD (beware of shoulder surfing)
unzip file.zip # extract the content of file.zip and prompt for password if encrypted
unzip -P PASSWORD file.zip # unencrypt and extract the content of file.zip with given PASSWORD (beware of shoulder surfing)

Compression with LHa (LHarc) / LZH:
lha a file.lha text.txt # compress text.txt as file.lha (or add text.txt to an already existing file.lha compressed archive)
lha e file.lha # extract the content of file.lha into the current directory (will prompt for existing files)

Compression with ZOO:
zoo a file.zoo text.txt # compress text.txt as file.zoo (or add text.txt to an already existing file.zoo compressed archive)
zoo e file.zoo # extract the content of file.zoo into the current directory (will ignore existing files)

Compression with RAR (RAR is shareware):
rar a file.rar text.txt # compress text.txt as file.rar (or add text.txt to an already existing file.rar compressed archive)
rar e file.rar # extract the content of file.rar into the current directory (will prompt for existing files)

Unpack most common formats:
unp FILE # Supported extensions and formats: tar[.gz,.bz2], gz, bz2, Z, ar/deb, rpm, shar, rar, arj, zip, LHa, cab, ace, tnef, uu (mail, news), mime, hqx, sea, zoo, pmd, cpio, afio, lzop

-----

Encoding:

Install encoders/decoders:
apt-get install mpack
apt-get install uudeview
apt-get install yencode

MIME/Base64:
mpack -s SUBJECT -o PATH/DEST_FILE PATH/SOURCE_FILE # encode (B64encode) within a MIME message with header
munpack PATH/FILE # decode (B64decode) to the original file name
uudeview -i PATH/FILE # decode (B64decode) to the original file name

Base64:
uuenview -b PATH/SOURCE_FILE>PATH/DEST_FILE # encode (B64encode). will not be decoded without MIME header.
base64 PATH/SOURCE_FILE>PATH/DEST_FILE # encode (B64encode)
base64 -d PATH/SOURCE_FILE>PATH/DEST_FILE # decode (B64decode)
echo "TEXT"|base64 # encode a TEXT (B64encode)
echo "VEVYVAo="|base64 -d # decode an encoded TEXT (B64decode)

UU (Unix-to-Unix):
uuenview -u PATH/SOURCE_FILE>PATH/DEST_FILE # encode (uuencode)
uudeview -i PATH/FILE # decode (uudecode) to the original file name

XX:
uuenview -x PATH/SOURCE_FILE>PATH/DEST_FILE # encode (xxencode)
uudeview -i PATH/FILE # decode (xxdecode) to the original file name

Yenc:
uuenview -y PATH/SOURCE_FILE>PATH/DEST_FILE # encode (yencode)
uudeview -i PATH/FILE # decode (ydecode) to the original file name # uudeview handy options: +o don't overwrite existing files, -c autoclear (delete) successfully decoded files, -d desperate mode (process incomplete files)
yencode PATH/SOURCE_FILE # encode (yencode) SOURCE_FILE as SOURCE_FILE.ync
ydecode PATH/FILE # decode (ydecode) to the original file name
ydecode -f PATH/FILE # decode (ydecode) to the original file name (force overwriting already existing files, without prompting)

-----

Hashing:

md5sum PATH/FILE # Return the MD5 hash for the content of the given FILE
md5sum PATH/FILE* # Return the MD5 hash for the content of the files matching the given mask
md5sum PATH/FILE1 PATH/FILE2 PATH/FILE3 # Return the MD5 hash for the given FILEs
echo -n "STRING" | md5sum | sed "s/  -//g" # Return the MD5 hash of the given STRING
VARIABLE=`echo -n "STRING" | md5sum | sed "s/  -//g"`; # Store the MD5 hash of the given STRING into a VARIABLE
VARIABLE=`echo -n "STRING" | md5sum | sed "s/  -//g"`; echo "[$VARIABLE]"; # Store the MD5 hash of the given STRING into a VARIABLE and output it between square brackets

-----

Obfuscation:


Text scrambling (anagrams):

apt-get install wordplay

wordplay "text string" # list all the possible anagrams of the given text string


Alphabet scrambling:

apt-get install rotix

rotix # accept a text from standard input and return its ROT-13 to the standard output
rotix -f PATH/SOURCE_FILE -o PATH/DEST_FILE # read SOURCE_FILE and return its ROT-13 to DEST_FILE (note: a ROT-13 can be deobfuscated performing the same operation again)
rotix -r 15 -L -f PATH/SOURCE_FILE -o PATH/DEST_FILE # read SOURCE_FILE and return its ROT-15 to the Left to DEST_FILE (note: this can be deobfuscated performing a ROT-15 to the Right)
rotix -r 12 -R -f PATH/SOURCE_FILE -o PATH/DEST_FILE # read SOURCE_FILE and return its ROT-12 to the Right to DEST_FILE (note: this can be deobfuscated performing a ROT-12 to the Left)


Steganography:

apt-get install steghide

steghide --embed -p PASSWORD -ef SOURCE_FILE_TO_HIDE -cf DEST_OBFUSCATION_FILE # hide a file into another (AES encryption and compression are applied by default)
steghide --embed -p PASSWORD -ef SOURCE_FILE_TO_HIDE -cf OBFUSCATION_FILE -sf DEST_FILE # hide a file into another (AES encryption and compression are applied by default), but save the result into a new file
steghide --info STEGOFILE -p PASSWORD # return information about the file obfuscated into STEGOFILE
steghide --extract -p PASSWORD -sf STEGOFILE -xf DESTFILE # extract the file obfuscated into STEGOFILE as DESTFILE
# steghide applies AES encryption and compression by default. It can hide information into JPEG, BMP, WAV, and AU files.

apt-get install stegdetect

stegdetect FILE.jpg # try to find steganographed information into a JPEG file

-----

Encryption with openssl:

# Note that AES-256-CBC method in unsecure as it does not provide authenticated encryption and is vulnerable to padding oracle attacks.

apt-get install openssl

openssl aes-256-cbc -pbkdf2 -salt -in PATH/CLEARTEXT -out PATH/ENCRYPTED # Encrypt CLEARTEXT file to ENCRYPTED file. It will prompt for password twice.
openssl aes-256-cbc -d -pbkdf2 -in PATH/ENCRYPTED -out PATH/CLEARTEXT # Decrypt ENCRYPTED file to CLEARTEXT file. It will prompt for password.
openssl aes-256-cbc -pbkdf2 -salt -pass pass:PASSWORD -in PATH/CLEARTEXT -out PATH/ENCRYPTED # Encrypt CLEARTEXT file to ENCRYPTED file using PASSWORD.
openssl aes-256-cbc -d -pbkdf2 -pass pass:PASSWORD -in PATH/ENCRYPTED -out PATH/CLEARTEXT # Decrypt ENCRYPTED file to CLEARTEXT file using PASSWORD.
echo -n "CLEARTEXT"|openssl aes-256-cbc -pbkdf2 -salt -pass pass:PASSWORD|base64 # Encrypt CLEARTEXT to ENCRYPTED using PASSWORD and encode it as Base64.
echo -n "U2FsdGVkX19YTYqwG/Dxp5t0k7mhGPYyvv70hiJfvOQ="|base64 -d|openssl aes-256-cbc -d -pbkdf2 -pass pass:PASSWORD # Decrypt Base64 encoded ENCRYPTED text to CLEARTEXT using PASSWORD.


Encryption with GnuPG (GPG) and Pretty Good Privacy (PGP) wrapper:

apt-get install pgpgpg

# PGP

pgp -h # Help
pgp -k # Key Manager functions help

pgp -kg # Generate key
pgp -kxa USER_ID PATH/DEST_FILE.ASC # Export User's public key to file
pgp -ka PATH/PUBLIC.KEY # Add public key to keyring

pgp -ea PATH/SOURCE_CLEARTEXT RECIPIENT_ID # Encrypt file for Recipient
pgp PATH/ENCRYPTED_FILE # Decrypt file (need password)

# Handy options:
# (Usage example: PGP -e = binary object file, PGP -ea = ASCII object file)
# -a   ASCII armour
# -w   Wipe (overwrite&delete) original file  (when crypting) 
# -m   No output file (on screen display, "eyes only")  (when decrypting)

# GPG

gpg -h # Help
gpg --version # Show GPG version and supported algorithms

gpg --gen-key # generate key
gpg --list-keys # list public keys in the public keyring
gpg --list-secret-keys # list secret (private) keys in the secret keyring
gpg --export -a > PATH/DEST_FILE.ASC # Export all public keys to a file
gpg --export -a "USER_ID" > PATH/DEST_FILE.ASC # Export User's public key to file
gpg --import PATH/PUBLIC.KEY # Add public key to keyring

gpg -s -r "USER_ID" PATH/SOURCE_CLEARTEXT # Sign file for Recipient, file name is automatically generated
gpg -s -a -r "USER_ID" PATH/SOURCE_CLEARTEXT # Create a signature for the given file for Recipient and output with an ASCII armour, file name is automatically generated
gpg -se -r "USER_ID" PATH/SOURCE_CLEARTEXT # Sign and encrypt file for Recipient, file name is automatically generated
gpg -se -a -r "USER_ID" PATH/SOURCE_CLEARTEXT # Sign and encrypt file for Recipient and output with an ASCII armour
gpg --verify PATH/SIGNATURE.SIG # Verify a signature (without showing the signed content)

gpg PATH/ENCRYPTED_FILE # Decrypt file (need password), file name is automatically generated
gpg -d PATH/ENCRYPTED_FILE > PATH/DEST_FILE # Decrypt file specifying the output file name (need password)
gpg -d PATH/ENCRYPTED_FILE # Decrypt file, on screen display, "eyes only" (need password)

-----

Passwords:

Password generation:

tr -dc 'a-z0-9' < /dev/urandom | head -c 10 # generate a random password of 10 characters with only lower case letters and numbers

apt-get install apg

apg -n 1 -M NL -a 1 -m 8 -x 12 # Generate a random password from 8 to 12 characters long made with only lower case letters and numbers
apg -n 1 -M NL -a 0 -m 10 -x 15 # Generate a random but human readable password from 10 to 15 characters long made with only lower case letters and numbers

apt-get install pwgen

pwgen -1 -A -n -B # Generate a password of 8 characters made with only lower case letters (-A) and at least one number (-n, default behavior) without ambiguous characters (-B)

-----

Audio:

Play a sound:
printf '\7' # Ring the Bell

Extract audio tracks from a CD:

apt-get install cdda2wav

cdda2wav -B # extract tracks as WAVs from the default CD player
cdda2wav -D  /dev/cdrom -B # extract tracks as WAVs from the specified CD player


Create an Audio CD writing WAVs as audio tracks:

apt-get install cdrecord

cdrecord dev=0,0,0 speed=4 -eject -pad -audio -raw -swab *.wav # write all WAVs from the current directory


Create an Audio CD writing MP3s/OGGs/flacs as audio tracks:

apt-get install mp3burn

mp3burn *.mp3 # create an audio CD from all MP3s in the current directory
mp3burn -p "PLAYLIST.m3u" # create an audio CD fetching files listed in a M3U playlist


Play MP3 files:

apt-get install mpg321

mpg321 FILE.mp3


Edit ID3v1 tags:

apt-get install id3ed

id3ed -i FILE.mp3 # return id3v1 tag
id3ed -r FILE.mp3 # remove id3v1 tag
id3ed -s SONGNAME -n ARTIST -a ALBUM -y YEAR -c COMMENT -k TRACKNUM -g GENRE_NUMBER FILE.mp3 # set id3v1 for FILE.mp3
id3ed -s SONGNAME -n ARTIST -a ALBUM -y YEAR -c COMMENT -k TRACKNUM -g GENRE_NUMBER *.mp3 # set id3v1 for all mp3 files in the current directory


Edit ID3v1 and ID3v2 tags:

apt-get install id3v2

id3v2 -l FILE.mp3 # return id3 tags
id3v2 -1 -l FILE.mp3 # return id3v1 tag
id3v2 -2 -l FILE.mp3 # return id3v2 tag
id3v2 --delete-v1 FILE.mp3 # remove id3v1 tag
id3v2 --delete-v2 FILE.mp3 # remove id3v2 tag
id3v2 --delete-all FILE.mp3 # remove all tags (id3 v1 and v2)
id3v2 -C FILE.mp3 # convert id3v1 to id3v2 tag
id3v2 -a "ARTIST" -A "ALBUM" -t "SONG" -c "COMMENT" -g GENRE_NUMBER -y YEAR -T TRACK FILE.mp3 # set id3 for FILE.mp3
id3v2 -a "ARTIST" -A "ALBUM" -t "SONG" -c "COMMENT" -g GENRE_NUMBER -y YEAR -T TRACK *.mp3 # set id3 for all mp3 files in the current directory
id3v2 -1 -a "ARTIST" -A "ALBUM" -t "SONG" -c "COMMENT" -g GENRE_NUMBER -y YEAR -T TRACK FILE.mp3 # set id3v1 for FILE.mp3
id3v2 -2 -a "ARTIST" -A "ALBUM" -t "SONG" -c "COMMENT" -g GENRE_NUMBER -y YEAR -T TRACK FILE.mp3 # set id3v2 for FILE.mp3


Check MP3 files:

apt-get install mp3check

mp3check FILE.mp3 # check crc and headers of mp3 files for consistency
mp3check -a FILE.mp3 # check mp3 file for differences from layer 3, 44.1kHz, 128kB, joint stereo, no emphasis, has crc


Lossless MP3 normalization (peak and loudness normalization):

apt-get install mp3gain

mp3gain FILE1.mp3 FILE2.mp3 FILE3.mp3 # normalize specified mp3 files (warning: overwrites source files)
mp3gain -r FILE1.mp3 FILE2.mp3 FILE3.mp3 # normalize specified mp3 files all to equal loudness (warning: overwrites source files)
mp3gain -a FILE1.mp3 FILE2.mp3 FILE3.mp3 # normalize specified mp3 files to the average loudness, intended for all audio files from a same album (warning: overwrites source files)
mp3gain -u FILE.mp3 # undo changes made by mp3gain (based on info stored on tag)


Convert WAV to MP3:

apt-get install lame

lame FILE.wav FILE.mp3


Convert MP3 to WAV:

apt-get install mpg321

mpg321 -w FILE.wav FILE.mp3 # convert FILE.mp3 to FILE.wav


Convert MP3 to OGG:

apt-get install mp32ogg

mp32ogg FILE.mp3 # convert FILE.mp3 to FILE.ogg
mp32ogg --rename=DEST.ogg SOURCE.mp3 # convert SOURCE.mp3 to DEST.ogg
mp32ogg --rename=%a_%l_%t.ogg *.mp3 # convert all mp3 files in the current directory renaming generated OGG files as ARTIST_ALBUM_TRACKTITLE.ogg (make sure that id3 tag contains all such information, otherwise files will be called as SOURCEFILE.ogg)


Convert an audio format into another audio format

apt-get install sox

sox SOURCE_FILE.SOURCE_EXT DEST_FILE.DEST_EXT # convert SOURCE_FILE.SOURCE_EXT to DEST_FILE.DEST_EXT. File format conversion is based on standard extensions
sox FILE.ogg FILE.mp3 # convert FILE.ogg into FILE.mp3 (note that MP3 support is optional. check sox --help for supported formats)

Reverse audio:
sox -V SOURCE_FILE.wav DEST_FILE.wav reverse

-----

Manipulate video files

apt-get install ffmpeg

Convert a video format into another video format:

ffmpeg -i PATH/SOURCE_VIDEO.avi PATH/DEST_VIDEO.mpg # convert from AVI to MPEG

ffmpeg -i PATH/SOURCE_VIDEO.avi -sameq FILE.avi PATH/DEST_VIDEO.mpg # convert from AVI to MPEG, keeping same quality of the source file

ffmpeg -i PATH/SOURCE_VIDEO.avi -ss 00:00:00 -t 00:45:00 PATH/DEST_VIDEO.mpg # convert from AVI to MPEG, taking only the first 45 minutes of the source video (from 00:00:00 to 00:45:00)

ffmpeg -i PATH/SOURCE_VIDEO.avi -i PATH/SOURCE_AUDIO.mp3 PATH/DEST_FILE.avi # create a video with audio taking video and audio from two different sources

ffmpeg -loop_input -i PATH/SOURCE_IMAGE.jpg -i PATH/SOURCE_AUDIO.mp3 -shortest -acodec copy PATH/DEST_FILE.mp4 # create a video with a static image and audio

Dump all frames from a video:
ffmpeg -i PATH/SOURCE_VIDEO -an -r 1/1 PATH/FILE_%06d.jpg # Dump all frames as JPG, you can dump them in .bmp or .png by changing the extension of the destination file. In this example the generated file name will have 6 digits

Speed up or slow down a video:
ffmpeg -i PATH/SOURCE_VIDEO -filter:v "setpts=SPEED*PTS" PATH/DEST_VIDEO # SPEED must be set to 1/(wanted speed x), for example set SPEED: 0.5 : double speed (2x), 0.25 : 4x speed, 2.0 = half speed (1/2x). If you want the destination video to have a specific duration, you can calculate the value of SPEED with this formula: 1 / (source video duration in seconds / destination video wanted duration in seconds)

Extract the audio track from a video:
ffmpeg -i PATH/SOURCE_VIDEO.mpg -vn -f mp3 AUDIO_TRACK.mp3 # extract the audio track as MP3 from a video
ffmpeg -i PATH/SOURCE_VIDEO.mpg -vn -ac 1 -f mp3 AUDIO_TRACK.mp3 # extract the audio track as MP3 from a video as mono. Use -ac 2 to force stereo even if the input file is mono.

-----

Play a video:

apt-get install vlc

vlc MOVIE.AVI # play a video (even in the text console)
vlc http://www.example.com:8081/file.mpg # play a video from the web

Streaming video server:

vlc -vvv file.mpg --sout '#standard{access=http,mux=mpeg1,url=www.example.com:8081/file.mpg}' --ttl 12 --loop # Stream a MPEG file (loop) from the url http://www.example.com:8081/file.mpg

# Note that you must have a streaming format like ASF to create a real stream of data
vlc -vvv file.asf --sout '#standard{access=mmsh,mux=asf,url=www.example.com:8082}' --ttl 12 --loop # Stream an ASF file (loop) from the url mms://www.example.com:8082/file.asf

-----

Manipulate and convert images:

apt-get install imagemagick

convert SOURCE.jpg DEST.png # Convert a JPEG image into PNG 
convert -resize 50% SOURCE.jpg DEST.jpg # resize SOURCE.jpg at 50% of the original size, and save it as DEST.jpg
convert -resize 600x300 SOURCE.jpg DEST.jpg # resize SOURCE.jpg to 600x300 (width x height) pixels, and save it as DEST.jpg
convert -resize 600 SOURCE.jpg DEST.jpg # resize SOURCE.jpg to a width of 600 pixels and relative height, and save it as DEST.jpg
convert -resize x300 SOURCE.jpg DEST.jpg # resize SOURCE.jpg to a height of 300 pixels and relative width, and save it as DEST.jpg
convert -thumbnail 150 SOURCE.jpg DEST.jpg # create a thumbnail of SOURCE.jpg with a width of 150 pixels and relative height, and save it as DEST.jpg # the difference between -resize and -thumbnail is that -thumbnail strips all meta data (like EXIF information) from the original image
convert -size 1x1 xc:none IMAGE.png # create a transparent .png image sized 1 x 1 pixels
convert -thumbnail x100 PATH/FILE.pdf[0] THUMBNAIL.png # create a thumbnail with a height of 100 pixels for the first page of a PDF document
convert -thumbnail x100 PATH/FILE.pdf THUMBNAIL.png # create a thumbnail with a height of 100 pixels for every page of a PDF document

-----

EXIF information:

apt-get install exiftool

exiftool FILE.jpg # show EXIF tags inside a JPEG image
exiftool -geotag= FILE.jpg # remove geographical coordinates from the given image
exiftool -all= FILE.jpg # remove all EXIF information from the given image
for i in PATH/*.jpg; do echo "Processing: $i"; exiftool -all= "$i"; done # remove all EXIF information from all the JPEG images in the given PATH


apt-get install exif

exif FILE.jpg # show EXIF tags inside a JPEG image
exif -im FILE.jpg # return EXIF tags inside a JPEG image in a machine readable format (-i ID instead of tags, -m tabs delimited output)
exif --ifd=0 -t TAG --set-value=VALUE SOURCE.jpg -o=DEST.jpg # set TAG as VALUE in SOURCE.jpg and save the resulting manipulated image as DEST.jpg
exif --remove --ifd=0 SOURCE.jpg -o=DEST.jpg # remove all EXIF information from SOURCE.jpg and save the resulting cleaned image as DEST.jpg
exif -r SOURCE.jpg -o=DEST.jpg # remove the thumbnail from SOURCE.jpg and save the resulting cleaned image as DEST.jpg
exif -e SOURCE.jpg -o=THUMBNAIL.jpg # extract the thumbnail from SOURCE.jpg and save it as THUMBNAIL.jpg 
exif FILE.jpg -n=THUMBNAIL.jpg # add THUMBNAIL.jpg to FILE.jpg and save the resulting manipulated image as FILE.jpg.modified.jpeg

-----

GPS data:

apt-get install gpsbabel

gpsbabel -i gdb -f PATH/SOURCE_FILE.gdb -o gpx -F PATH/DEST_FILE.gpx # convert a Garmin .gdb file into .gpx (GPS eXchange Format, XML)

-----

Disk quotas:

apt-get install quota quotatool

quota # display disk usage and limits
quotacheck # scan a file system for disk usage
quotactl # set disk quotas

-----

Quitting a session, turning off (halting) or rebooting system:

# Note that when a process receives a signal may behave differently from the default behavior
CTRL+C # terminate an application (sends a SIGINT (2) signal to the process, exits with Exit Code 130)
CTRL+\ # terminate the process and dump to a core file (sends a SIGQUIT signal to the process)
CTRL+D # send an EOT (End Of Tranmission) signal to the process (exits with Exit Code 0)
CTRL+Z # suspend an application (sends a SIGSUSP (20) signal to the process, exits with Exit Code 148)
return # return from a function or a sourced script
exit # exit from a non login shell
logout # logout from current session (you can also press CTRL+D )
shutdown -h now # halt the system
shutdown -h -P now # halt the system and turn off the machine (power off)
shutdown -r now # reboot the system
shutdown -r 22:00 # reboot the system at 22:00 (10:00 pm)
shutdown -r +5 # reboot the system after 5 minutes
reboot # reboot the system (like shutdown -r), might not work on older systems
reboot -f # force reboot without using shutdown

Show system reboot history:
last reboot
last -x|grep shutdown
for wtmp in `ls -t /var/log/wtmp*`; do last reboot -f $wtmp; done|grep reboot
last reboot|head -1 # Show only last reboot

Show uptime:
# A sysadmin will be judged by his uptime and by how many reboots he performed
uptime # Show uptime, total users, and load average of the system over the last 1, 5, and 15 minutes.
uptime -p # Show only uptime in pretty form
uptime -s # Show system up since (last boot)

# Edit system startup script
# Commands that will be executed when the system boots. Different runlevels are allowed.
jed /etc/inittab

Choose what command must be performed when CTRl+ALT+DEL is pressed:
jed /etc/inittab
# What to do when CTRL-ALT-DEL is pressed.
ca:12345:ctrlaltdel:/sbin/shutdown -t1 -a -r now

-----

Install a LAMP webserver:

Install servers and related software:
apt-get install apache2
apt-get install lynx
apt-get install openssl
# apt-get install php4 libapache2-mod-php4 php4-pear php4-gd php4-mcrypt # PHP 4
# apt-get install php4-mysql # MySQL support for PHP 4
# apt-get install php5 libapache2-mod-php5 php-pear php5-gd php5-mcrypt php5-sqlite # PHP 5
# apt-get install php5-mysql # MySQL support for PHP 5
# apt install php7.0 libapache2-mod-php7.0 php-pear php7.0-gd php7.0-mcrypt php7.0-mbstring php7.0-sqlite3 # PHP 7
# apt install php7.0-mysql # MySQL support for PHP 7
apt install php7.3 libapache2-mod-php7.3 php-pear php7.3-gd php7.3-mbstring php7.3-sqlite3 # PHP 7.3
apt install php-imagick # ImageMagick for PHP
apt install php7.3-mysql # MySQL support for PHP 7.3
# apt-get install mysql-server # Obsolete since Debian GNU/Linux 10 (buster)
apt install mariadb-server-core-10.3 mariadb-server-10.3 mariadb-plugin-tokudb mariadb-plugin-spider mariadb-plugin-oqgraph mariadb-plugin-mroonga mariadb-plugin-connect
apt install phpmyadmin


Apache 2 configuration:

/usr/sbin/apache2 -v  # Return Apache 2 version
openssl version -v # Get OpenSSL version

apache2 -l # list compiled in Apache 2 modules
apache2ctl -l # list compiled in Apache 2 modules
apache2ctl -M # list loaded Apache 2 modules
a2query -M # Return Apache Multi-Processing Module (MPM): event, prefork, worker

a2enmod # Add a module choosing it from a list. You'll have to restart Apache 2 then.
a2enmod rewrite # Add rewrite module. Restart Apache to enable it.

jed /etc/apache2/apache2.conf

If your webserver is too slow, you may have to adjust the value for MaxClients: if it's too low it will allow too few simultaneous accesses, if it's too high it will require too much resources to handle them. Try with MaxClients 50 or MaxClients 100 .

MaxClients 100


Also, you'd better set MaxKeepAliveRequests to a value higher than the default:
MaxKeepAliveRequests 2000

You can also add the expires module and set expiry periods for the cache of file types you don't change often on the server (like images):

ExpiresActive On
ExpiresDefault "access plus 2 days"
ExpiresByType text/php "access plus 1 second"
ExpiresByType text/x-php "access plus 1 second"
ExpiresByType application/php "access plus 1 second"
ExpiresByType application/x-php "access plus 1 second"
ExpiresByType application/x-httpd-php "access plus 1 second"
ExpiresByType text/html "access plus 2 days"
ExpiresByType text/plain "access plus 2 days"
ExpiresByType image/jpg "access plus 1 month"
ExpiresByType image/jpeg "access plus 1 month"
ExpiresByType image/gif "access plus 1 month"
ExpiresByType image/png "access plus 1 month"
ExpiresByType text/css "access plus 1 month"
ExpiresByType application/pdf "access plus 1 month"
ExpiresByType text/x-javascript "access plus 1 week"
ExpiresByType application/x-shockwave-flash "access plus 1 month"
ExpiresByType image/x-icon "access plus 1 year"


If you use .htaccess files in your web directories to have them to be executed by the web server, you must be sure that AllowOverride is set to All for such directories:

        Options Indexes FollowSymLinks
        AllowOverride All
        Require all granted


Specify available charsets and default charset:

AddCharset ISO-8859-1  .iso8859-1  .latin1
AddCharset UTF-8       .utf8
# AddDefaultCharset UTF-8
AddDefaultCharset ISO-8859-1

jed /etc/apache2/ports.conf # Port listened by the webserver (normally 80 for HTTP connections and 443 for HTTPS secure connections)
jed /etc/apache2/sites-enabled/default.conf # set DocumentRoot and create Virtual Hosts (vhosts). Older versions of Apache2 stores sites by default in /etc/apache2/sites-available/default

This is an example of a VirtualHost:

ServerName www.example.com
ServerAdmin webmaster@example.com
DocumentRoot "/var/www/example.com/www"
ServerSignature On
LogLevel Warn
ErrorLog /var/log/apache2/example-com_error.log
CustomLog /var/log/apache2/example-com_access.log combined


Check Apache configuration for errors:
apache2ctl configtest

Restart Apache after modifying its configuration:
/etc/init.d/apache2 restart

See if Apache is running:
netstat -plntu | grep apache # if you get a line containing the socket listened by the Apache webserver, then it's running

See Apache error log:
cat /var/log/apache2/error.log

Delete Apache access and error logs
rm /var/log/apache2/*.gz # Delete archived logs
rm /var/log/apache2/* # In case you want to delete access and error logs for all websites. IMPORTANT: Note that you'll have to restart Apache to set up logs and have them running again


Make sure the PHP package is enabled on Apache:
a2query -m php7.3
If it replies with "No module matches php7.3", then enable it:
a2enmod php7.3
Then restart Apache

PHP configuration on Apache 2:
jed /etc/php4/apache2/php.ini # PHP 4 configuration
jed /etc/php5/apache2/php.ini # PHP 5 configuration
jed /etc/php/7.x/apache2/php.ini # PHP 7.x configuration (make sure to replace x with your current subversion of PHP 7 )

Make sure engine = On in the PHP configuration file or PHP will not work at all.

You may want to set short_open_tag = On if your PHP code uses short tags, which means < ? ... ? > instead of < ?php ... ? >

You may specify the charset (character encoding) used by PHP generated pages. Note that it overrides charsets specified in the Apache configuration or in the HTML document (such as  in HTML 4 or  in HTML 5), so you'd probably better leave it empty
default_charset = ""
however in case you need to set it, most popular settings are:
default_charset = "UTF-8"
or
default_charset = "iso-8859-1"

Show PHP version:
php -v

Show PHP info:
php -i

List compiled-in PHP modules:
php -m

To send e-mails from PHP scripts, you need to install sendmail
apt-get install sendmail sendmail-bin sendmail-doc rmail

Then edit the hosts file ( jed /etc/hosts ) and set the alias for the IP 127.0.0.1 as
127.0.0.1 DOMAIN localhost localhost.localdomain HOSTNAME
where DOMAIN is a domain name (for example: example.com ) addressed to your server's IP and HOSTNAME is the hostname of your server (that you can get with the command hostname ). The domain name is not necessary but strongly recommended because some antispam services blacklist servers which send e-mails identifying themselves (EHLO) as "localhost.localdomain".

After setting the hosts file you have to reconfigure and restart sendmail:
sendmailconfig # Normally you have to just reply with Y to all questions
service sendmail restart

To send a test e-mail from the console:
echo "Subject: test"; echo "Body Test"; |/usr/lib/sendmail -v -F SENDER@MAILBOX.EXAMPLE.COM -t RECEIVER@MAILBOX.EXAMPLE.COM

You should check sendmail_path in PHP configuration (php.ini) and make sure it's either disabled or set to the default value of sendmail -t -i

To send an e-mail from a PHP page you have to use the mail command, like in this example:
mail($emailto, $emailsubject, $emailbody, "From: ".$emailfrom."\nReply-To: ".$emailreplyto."\nX-Mailer: ".$xmailer);
Of course all sample variables used must be set properly. This is a test without using variables, you need to replace email addresses in this example with actual working ones:
mail('user2@mailbox.example.com', 'Test', "This is a test\nfrom PHP", "From: ".'user1@mailbox.example.com'."\nReply-To: ".'user1@mailbox.example.com'."\nX-Mailer: ".'PHP');

Check the log of last sent e-mails:
tail --lines=20 /var/log/mail.log


Show MySQL version:
mysql -V

See if MySQL is running:
netstat -plntu | grep mysql # if you get a line containing the socket listened by the MySQL server, then it's running

Set MySQL root password:
$ mysql
 mysql> SET PASSWORD FOR root@localhost = PASSWORD('newpassword');

Create an alternate user with root-like access:
$ mysql -u root -pROOT_PASSWORD mysql

 mysql> CREATE USER 'USER_NAME'@'localhost' IDENTIFIED BY 'SET_PASSWORD_HERE'; GRANT ALL PRIVILEGES ON *.* TO 'USER_NAME'@'localhost'; FLUSH PRIVILEGES;

Create a new mysql user and assign a database to him
$ mysql -u root -pROOT_PASSWORD mysql

 mysql> insert into user (Host, User, Password, Select_priv) values ('localhost', 'USERNAME', password('USER_PASSWORD'), 'N');
 Query OK, 1 row affected (0.00 sec)

 mysql> insert into db (Host, Db, User, Select_priv, Insert_priv, Update_priv, Delete_priv, Create_priv, Drop_priv, Grant_priv, References_priv, Index_priv, Alter_priv, Create_tmp_table_priv, Lock_tables_priv) values ('localhost', 'USER_DATABASE', 'USER_NAME', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y', 'Y');
 Query OK, 1 row affected (0.00 sec)

 mysql> quit

$ mysqladmin -u root -pROOT_PASSWORD reload

The user can access the assigned database:
$ mysql -u USERNAME -pUSER_PASSWORD USER_DATABASE

Remove an user:
mysql> drop user USERNAME@HOSTNAME;
Or
mysql> delete from mysql.user where User='USERNAME' and Host='HOSTNAME';
mysql> revoke all privileges on *.* from USERNAME@HOSTNAME;
mysql> revoke grant option on *.* from USERNAME@HOSTNAME;
mysql> flush privileges;

Show all MySQL users:
mysql> select user, host from mysql.user; # show only user name and related host name for every user
mysql> select * from mysql.user; # show detailed information for every user

Show all databases:
mysql> show databases;

Show all tables in the current database:
mysql> show tables;
mysql> show table status; # provide information about every table

Show all fields (coloumns) of a table:
mysql> show fields from TABLE; # same as describe TABLE;
mysql> show fields from TABLE from DB; # show fields of a table from another database
mysql> show full fields from TABLE; # also lists privileges available to the current user

Show all indexes (keys) of a table:
mysql> show index from TABLE;
mysql> show index from TABLE from DB; # show indexes of a table from another database

Create an index (key) for a field (coloumn):
mysql> alter table TABLE add index (`FIELD`) ;

Count all records within a table:
mysql> select count(*) from TABLE;

Backup a database to a MySQL dump file (structure only):
mysqldump -no-data -u MYSQLUSER -pPASSWORD DATABASE > MYSQLDUMP.sql

Backup a database to a MySQL dump file (data only):
mysqldump -c --no-create-db --no-create-info -u MYSQLUSER -pPASSWORD DATABASE > MYSQLDUMP.sql

Backup a database to a MySQL dump file (structure and data):
mysqldump -c --add-drop-database --add-drop-table --add-locks -u MYSQLUSER -pPASSWORD DATABASE > MYSQLDUMP.sql

Back up a MySQL database into a GZIP file:
mysqldump -c --add-drop-database --add-drop-table --add-locks -u MYSQLUSER -pPASSWORD DATABASE | gzip -9v > MYSQLDUMP_`date +%Y%m%d`.sql.gz

Restore a database from a MySQL dump file:
mysql -h localhost -u MYSQLUSER -pPASSWORD DATABASE < MYSQLDUMP.sql

Delete a database:
mysql> drop database if exists DATABASE;

Delete a table:
mysql> drop table if exists TABLE;

Delete a field:
mysql> alter table TABLE drop FIELD;

Delete an index:
mysql> drop index INDEX on TABLE;

Empty a table without deleting it:
mysql> truncate table TABLE;

Change engine type:
mysql> alter table TABLE type = MYISAM; # change the engine tipe of TABLE to MyISAM

Analyzes indexes of a table:
# for MyISAM and BDB tables only
mysql> analyze table TABLE;

Check a table for errors:
mysql> check table TABLE;

Attempt to repair a corrupted table
mysql> repair table TABLE;

Optimize a table (defragment and rebuild indexes):
mysql> optimize table TABLE;

Show MySQL server status:
mysql> show status;

Check running MySQL processes (queries):
$ mysqladmin -h localhost -u root -pROOT_PASSWORD processlist
Or, from the MySQL console:
mysql> show processlist;

Kill a MySQL process:
$ mysqladmin -u root -pROOT_PASSWORD kill ID
Or, from the MySQL console:
mysql> kill ID;

Quit MySQL console:
mysql> quit

Show MySQL version:
mysql -V

To restart MySQL:
/etc/init.d/mysql restart
# If MySQL fails to start make sure the directories /var/log/mysql and /var/log/mysqld are owned by mysql:adm

Run a MySQL file and store output into another file:
mysql -u USER --password=PASSWORD DATABASE_NAME < SOURCE.sql > DESTINATION.txt


This may fix these errors that may happen after an upgrade:
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock'
ERROR 1577 (HY000) at line 1: Cannot proceed because system tables used by Event Scheduler were found damaged at server start

apt-get install php5-mysql mysql-server mysql-server-5.1
mv /etc/mysql/my.cnf /etc/mysql/my.cnf.bak1
mv /etc/mysql/my.cnf.dpkg-dist /etc/mysql/my.cnf
/etc/init.d/mysql start
apt-get install php5-mysql mysql-server mysql-server-5.1


Upgrading from PHP 4 to PHP 5:

# Note: this will also upgrade from MySQL 4 to MySQL 5, if MySQL 4 is installed
# Warning: newer versions of MySQL often use different formats for tables than earlier versions. In some cases you can attempt to make restore such tables compatible using  repair table TABLE;  and  optimize table TABLE; You'd better dump all your tables before to upgrade, in case you'll need to restore them after. In fact you may need to DROP tables which are no longer working as expected and rebuild them. A common issue are tables containing floating point fields: queries containing comparisons based on those fields will no longer work as they used to.

# stop the servers
/etc/init.d/apache2 stop
/etc/init.d/mysql stop

# remove PHP 4
apt-get remove libapache2-mod-php4 php4 php4-gd php4-pear php4-mcrypt php4-mysql

# You have to remove MySQL as well
apt-get remove mysql-server

# Install PHP 5
apt-get install php5 libapache2-mod-php5 php5-sqlite php-pear php5-gd php5-mcrypt # PHP 5
apt-get install php5-mysql # MySQL support for PHP 5

# Install MySQL again
apt-get install mysql-server

# Reinstall phpmyadmin (if you need it)
apt-get install phpmyadmin # Removing PHP 4 will remove phpmyadmin as well, you'll have to reinstall it if you need it

# Make sure that there are no references to PHP 4 left in the Apache configuration
jed /etc/apache2/apache2.conf
# Change  as 
# Add .php5 to AddType application/x-httpd-php
# Comment out  ...  and everything between

# restart the servers
/etc/init.d/mysql restart
/etc/init.d/apache2 restart


Switch from PHP 5 to PHP 7:

If you have two different versions of PHP, for example PHP 5.6 and PHP 7.0, as it can happen after upgrading Debian from jessie to stretch, you can switch between them with these commands:

update-alternatives --config php # Set the default PHP version for the command php (this will not influence the version used on the Apache webserver)

a2dismod php5 # Dismount the Apache2 Module for PHP5
a2enmod php7.0 # Mount the Apache2 Module for PHP7
systemctl restart apache2 # Restart Apache


phpMyAdmin:

# If you can't see the phpmyadmin directory into /var/www/ then the package likely installed it into /usr/share/ , in this case you have to link it manually to access it from the web
ln -s /usr/share/phpmyadmin /var/www/

Install phpMyAdmin manually:

# You may want to install phpMyAdmin manually, especially if your Debian is set to Stable and there's no 'phpmyadmin' package available.

# Create a directory for phpMyAdmin
mkdir /usr/share/phpmyadmin

# Create a directory to save phpMyAdmin source
mkdir /usr/share/phpmyadmin/src

# Go to phpmyadmin source directory and use it as a work directory
cd /usr/share/phpmyadmin/src

# Download latest version of phpMyAdmin
wget -P /usr/share/phpmyadmin/src https://www.phpmyadmin.net/downloads/phpMyAdmin-latest-all-languages.tar.gz

# Download phpMyAdmin public keyring to verify the source
wget -P /usr/share/phpmyadmin/src https://files.phpmyadmin.net/phpmyadmin.keyring

# Import phpMyAdmin public keyring into GPG
gpg --import phpmyadmin.keyring

# Get phpMyAdmin public key
wget -P /usr/share/phpmyadmin/src https://www.phpmyadmin.net/downloads/phpMyAdmin-latest-all-languages.tar.gz.asc

# Verify phpMyAdmin source
gpg --verify phpMyAdmin-latest-all-languages.tar.gz.asc

# You should get something like:
----------
gpg: assuming signed data in 'phpMyAdmin-latest-all-languages.tar.gz'
gpg: Signature made Thu 15 Oct 2020 20:10:40 CEST
gpg:                using RSA key 3D06A59ECE730EB71B511C17CE752F178259BD92
gpg: Good signature from "Isaac Bennetch " [unknown]
gpg:                 aka "Isaac Bennetch " [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 3D06 A59E CE73 0EB7 1B51  1C17 CE75 2F17 8259 BD92
----------
# Verify that the key returned by GPG is the same as shown in phpMyAdmin's website for the given release manager person:
https://docs.phpmyadmin.net/en/latest/setup.html#verifying-phpmyadmin-releases
# In this case, Isaac Bennetch's fingerprint is 3D06 A59E CE73 0EB7 1B51  1C17 CE75 2F17 8259 BD92

# If everything is OK, unpack the compressed file into phpMyAdmin's directory
tar xvf phpMyAdmin-latest-all-languages.tar.gz --strip-components=1 -C /usr/share/phpmyadmin/

# You can now remove the download directory (note that if left in the position given in this tutorial, it will also be available from the Web)
cd /usr/share/phpmyadmin/
rm /usr/share/phpmyadmin/src/*
rmdir /usr/share/phpmyadmin/src

# Create a configuration file for phpMyAdmin
cp /usr/share/phpmyadmin/config.sample.inc.php /usr/share/phpmyadmin/config.inc.php

# Edit phpMyAdmin configuration file
jed /usr/share/phpmyadmin/config.inc.php
# And set a password for the cookies
$cfg['blowfish_secret'] = 'PUT_YOUR_PASSWORD_HERE';

# Set modes for the configuration file
chmod 660 /usr/share/phpmyadmin/config.inc.php

# Set user and group for phpMyAdmin's directory
chown -R www-data:www-data /usr/share/phpmyadmin

# Create a symlink to make it accessible from the Webserver
ln -s /usr/share/phpmyadmin /var/www/

# Make phpMyAdmin available from the Web, for example:
jed /etc/apache2/sites-enabled/default.conf
---------

ServerName mysql.EXAMPLE.COM
ServerAdmin webmaster@EXAMPLE.COM
DocumentRoot "/var/www/phpmyadmin"
ServerSignature On
LogLevel Warn
ErrorLog /var/log/apache2/phpmyadmin_error.log
CustomLog /var/log/apache2/phpmyadmin_access.log combined

---------

# If you want to upgrade manually phpMyAdmin:

# Make a backup of the current phpMyAdmin directory
cp -rp /usr/share/phpmyadmin /usr/share/phpmyadmin_bak

# Empty phpMyAdmin directory without removing it
rm -r /usr/share/phpmyadmin/*

# Follow the steps to install phpMyAdmin manually except for: Create a directory for phpMyAdmin (if you've emptied the existing one without removing it), Create a symlink to make it accessible from the Webserver, Make phpMyAdmin available from the Web.

# If the new version works properly, you can remove the backup you've created
rm -r /usr/share/phpmyadmin /usr/share/phpmyadmin_bak


# Restart Apache Webserver
/etc/init.d/apache2 restart


HTTPS:

If you want to have https websites (so that the client can access the server securely through an encrypted connection to the server) hosted in your webserver, follow this procedure:

Certificates and keys should be stored in these directories:
/etc/ssl/certs # for certificates
/etc/ssl/private # for keys

Generate a Key:
openssl genrsa -out /etc/ssl/private/DOMAIN.key 2048
Enter a valid domain name 

Generate a request certificate:
openssl req -new -sha256 -key /etc/ssl/private/DOMAIN.key -out www.example.com.csr

The only fields you must fill are Organization Name (eg, company) and Common Name (e.g. server FQDN or YOUR name), you can leave empty all other fields. Note that fields with a given default value must be filled with a dot (.) to make them empty, if you simply press RETURN the default value will be used instead:

----------
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:.
State or Province Name (full name) [Some-State]:.
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:COMPANY
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:DOMAIN
Email Address []:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
----------

If you want to validate your domain (to claim its possession) via e-mail, enable the e-mail address admin@DOMAIN (that is, make sure you can receive e-mails to admin) for validation

Buy a certificate from a Certificate Authority (CA)
https://www.digicert.com/
https://www.ssl.com/ (via comodo)
https://www.comodo.com/

Validate the domain (claim it as yours) according to the procedure requested by the Certificate Authority

Download the certificates ( ca-bundle-client.crt , DOMAIN.crt )
Upload the certificates into /etc/ssl/certs (the private key you've generated should already be into /etc/ssl/private/ )

Edit the Apache configuration file:
jed /etc/apache2/sites-enabled/default.conf

Create a VirtualHost for the DOMAIN for port 443 with the following directives:


SSLEngine on
SSLCertificateFile /etc/ssl/certs/DOMAIN.crt
SSLCertificateKeyFile /etc/ssl/private/DOMAIN.key
SSLCertificateChainFile /etc/ssl/certs/ca-bundle-client.crt
# HSTS (mod_headers is required) (15768000 seconds = 6 months)
# Header always set Strict-Transport-Security "max-age=15768000"
# ...


For example:


SSLEngine on
SSLCertificateFile /etc/ssl/certs/example_com.crt
SSLCertificateKeyFile /etc/ssl/private/example_com.key
SSLCertificateChainFile /etc/ssl/certs/ca-bundle-client.crt
# HSTS (mod_headers is required) (15768000 seconds = 6 months)
# Header always set Strict-Transport-Security "max-age=15768000"
ServerName www.example.com
ServerAdmin webmaster@example.com
DocumentRoot "/var/www/example.com/www"
ServerSignature On
LogLevel Warn
ErrorLog /var/log/apache2/example-com_error.log
CustomLog /var/log/apache2/example-com_access.log combined


If you still want the website to be available via http (without encryption), you also have to configure it without SSL (normally for port 80). Note that it can be configured to have a different document directory () and thus provide different content from the https (encrypted) website.


ServerName www.example.com
ServerAdmin webmaster@example.com
DocumentRoot "/var/www/example.com/www"
ServerSignature On
LogLevel Warn
ErrorLog /var/log/apache2/example-com_error.log
CustomLog /var/log/apache2/example-com_access.log combined


If you want to redirect all http traffic to https for a portion of the website, add this line inside the VirtualHost for http (port :80)
Redirect /ALWAYS_HTTPS_DIRECTORY_NAME https://www.example.com/ALWAYS_HTTPS_DIRECTORY_NAME
If you want to redirect all http traffic to https for the whole website, add this line inside the VirtualHost for http (port :80)
Redirect / https://www.example.com/

Restart the webserver when you're done with editing the configuration file:
/etc/init.d/apache2 restart

When you renew a certificate, you'll have to upload the newly reissued certificates ( ca-bundle-client.crt , DOMAIN.crt ) replacing the old ones into /etc/ssl/certs and restart the webserver.



Password protected directories in Apache:

# Create a password file, it should be placed in a directory not served by the webserver (for example, /var/ , which normally is out of the htdocs directory)
touch PATH/.htpasswd
chmod 644 PATH/.htpasswd

# Create users and assign passwords to them
htpasswd -b PATH/.htpasswd USER PASSWORD # Create a password for every user. The same statement can be used to change a password of an already existing user

htpasswd -D PATH/.htpasswd USER # Delete a user from the password file

# Create an access control file in the directory containing the files with restricted access
# Use Require valid-user to allow access to any user in the password file, or list specific users allowed to access the restricted files
jed PATH_TO_PROTECT/.htaccess
---
AuthUserFile PATH/.htpasswd
AuthName "Enter Password"
AuthType Basic

# Require valid-user
Require user USER1
Require user USER2

---

Web site statistics:

grep " 404 " /var/log/apache2/ACCESS.log | awk -F\" '$9=404{print $9" "$2}' | sort | uniq -c | sort -rg | head --lines=25 # Show top 404 error ("Not Found") requested pages
grep "GET /" /var/log/apache2/ACCESS.log | awk {'print $7'} | sort | uniq -c | sort -g | tail --lines=15 | tac # Show top accessed pages
awk {'print $7'} /var/log/apache2/ACCESS.log | grep ".php" | sort | uniq -c | sort -g | tail --lines=15 | tac # Show top accessed PHP pages
grep -i "GET /PATH/DOCUMENT" /var/log/apache2/ACCESS.log | awk '{print $1}' | sort -n | uniq -c | sort -rn | head --lines=25 # Show top IPs accessing the given DOCUMENT (web page)


Webalizer:

apt-get install webalizer
jed /etc/webalizer.conf # or jed /etc/webalizer/webalizer.conf # or another custom .conf file
webalizer /var/log/apache2/access.log.1 # create a webalizer report for access.log.1
webalizer /var/log/apache2/example-com_access.log # create a webalizer report for example-com_access.log

If you get the following error message:
Error Opening file /usr/share/GeoIP/GeoIP.dat
you can fix it installing the GeoIP database:
apt-get install geoip-database

To have statistics available on the web for many websites, create a different access log, webalizer configuration file and output directory, for every website and a configuration file, and call webalizer from a script.

Example:

# 1. Set Apache log files
jed /etc/apache2/sites-enabled/default.conf


[...]
CustomLog /var/log/apache2/example-com_www_access.log combined
[...]



[...]
CustomLog /var/log/apache2/example-net_my_access.log combined
[...]


/etc/init.d/apache2 restart

# 2. Create output directories
mkdir /var/www/webalizer
mkdir /var/www/webalizer/www.example.com
mkdir /var/www/webalizer/my.example.net

# 3. Assign ownership of the directories to www-data
chown www-data:www-data /var/www/webalizer/www.example.com
chown www-data:www-data /var/www/webalizer/my.example.net

# 4. Create custom Webalizer configuration files
jed /etc/webalizer/webalizer_www.example.com.conf
LogFile         /var/log/apache2/example-com_www_access.log
OutputDir       /var/www/webalizer/www.example.com
Incremental     yes
HostName        www.example.com
HideSite        *example.com
HideReferrer    example.com/
[...]

jed /etc/webalizer/webalizer_my.example.net.conf
LogFile         /var/log/apache2/example-net_my_access.log
OutputDir       /var/www/webalizer/my.example.net
Incremental     yes
HostName        my.example.net
HideSite        *example.com
HideReferrer    example.com/
[...]

# 5. Create a script that invokes webalizer for every website

mkdir /etc/scripts

jed /etc/scripts/webalize.sh
---
#!/bin/bash
# Webalize
webalizer -c /etc/webalizer/webalizer_www.example.com.conf>/dev/null
webalizer -c /etc/webalizer/webalizer_my.example.net.conf>/dev/null
---

Sample lines to add to Crontab:
---
1 0,12 * * * /var/www/scripts/webalize.sh>/dev/null
21 6 * * * /var/www/scripts/webalize.sh>/dev/null
---
The first line tells Crontab to invoke Webalizer every day at 00:01 and 12:01, the second line makes Crontab invoke Webalizer every day at 6:21 . This is because logs start and end at a certain hour of the day (for example, 06:26), and you'd miss all the hours between last call of Webalizer, every day. To know the exact time when your logs starts and stops check the head and tail of the previous log, for example: head --lines=10 /var/log/apache2/www.example.net_access.log.1 , tail --lines=10 /var/log/apache2/www.example.net_access.log.1

# If you want to reset information cached from previous runs of Webalizer (especially if you've set Incremental yes in the webalizer configuration file), delete the files webalizer.current and webalizer.hist in the Webalizer OutputDir.
# Example:
# rm /var/www/webalizer/webalizer.current
# rm /var/www/webalizer/webalizer.hist

-----

SQLite:

apt-get install sqlite sqlite3 libsqlite3-dev

# To fix these errors: sqlite_exec(): attempt to write a readonly database; sqlite_exec(): unable to open database file
chmod 666 *.sdb
chmod 777 the directory containing the .sdb file

-----

FTP server:
# Note that FTP protocol is deprecated in favor of SSH connections, because it's unencrypted and unsecure.

PROFTP:

apt-get install proftpd
(run as standalone)

Create an user called "www" to access the www directory:

adduser --home /var/www www

Allow FTP access to the user "www":

jed /etc/proftpd.conf

Add the following lines in the user section of the proftpd configuration file:


User www
Group www
DirFakeUser on ftp
DirFakeGroup on ftp
RequireValidShell off
MaxClients 2
DisplayLogin welcome.msg
DisplayFirstChdir .message
 
  
   AllowAll
  
 


/etc/init.d/proftpd restart


VSFTP:
# VSFTP allows virtual users, besides real shell users and anonymous users (you can simulate a chroot'ed FTP)

apt-get install vsftpd
apt-get install libpam-pwdfile

Create a password file for virtual users:
# Note: htpasswd only allows passwords up to 8 characters long
htpasswd -bc /etc/vsftpd_passwd USERNAME PASSWORD

Additional users can be created without the -c parameter. The same statement can be used to change the password for an existing user:
htpasswd -b /etc/vsftpd_passwd USERNAME PASSWORD

If you want to enter the password manually, omit the -b parameter:
htpasswd /etc/vsftpd_passwd USERNAME

Use this command to remove a virtual user:
htpasswd -D /etc/vsftpd_passwd USERNAME

You'll need to create a directory for each virtual user (unless you configure vsftp otherwise):

First, create the main directory for all virtual users:
mkdir /var/www/ftp

Then create a directory for each virtual user with proper read/write permissions:
mkdir /var/www/ftp/USERNAME
chmod 755 /var/www/ftp/USERNAME
chown ftp /var/www/ftp/USERNAME


Configure PAM (Pluggable Authentication Modules aka X/Open Single Sign-on) for vsftp:
jed /etc/pam.d/vsftpd
# Log in using htpasswd password file
auth    required pam_pwdfile.so pwdfile /etc/vsftpd_passwd
account required pam_permit.so


jed /etc/vsftpd.conf
# Config file /etc/vsftpd.conf
listen=YES
#listen_ipv6=YES
anonymous_enable=NO
local_enable=YES
write_enable=YES
local_umask=022
anon_upload_enable=NO
anon_mkdir_write_enable=NO
dirmessage_enable=YES
xferlog_enable=YES
connect_from_port_20=YES
#chown_uploads=YES
#chown_username=ftp
xferlog_file=/var/log/vsftpd.log
#xferlog_std_format=YES
idle_session_timeout=600
data_connection_timeout=120
#nopriv_user=ftp
#async_abor_enable=YES
#ascii_upload_enable=YES
#ascii_download_enable=YES
ftpd_banner=Welcome
#deny_email_enable=YES
#banned_email_file=/etc/vsftpd.banned_emails
chroot_local_user=YES
#chroot_list_file=/etc/vsftpd.chroot_list
#ls_recurse_enable=YES
secure_chroot_dir=/var/run/vsftpd
pam_service_name=vsftpd
rsa_cert_file=/etc/ssl/certs/vsftpd.pem
guest_enable=YES
virtual_use_local_privs=YES
#user_config_dir=/etc/vsftpd_user_conf
#guest_username=ftp
user_sub_token=$USER
local_root=/var/www/ftp/$USER
hide_ids=YES
#cmds_allowed=PASV,BYE,LIST,ABOR,CWD,NLST,PORT,PWD,QUIT,RETR,SIZE,TYPE

# Note that the option user_config_dir lets you create custom vsftp configuration files for each virtual users
# You can create a config file with the name of the virtual user within the specified path
jed /etc/vsftpd_user_conf/USERNAME
# VSFTP extra config for user USERNAME
local_root=/var/www/example.com/www/USERNAME


Restart VSFTP to make it work with the new users or configuration:
/etc/init.d/vsftpd restart

-----

Samba:

apt-get install samba smbfs smbclient samba-doc

mkdir /public
mkdir /public/shared
chmod -v 0777 /public
chmod -v 0777 /public/shared

jed /etc/samba/smb.conf

---
# samba configuration file

[global]
allow hosts = 192.168.0.0/255.255.255.0
workgroup = network
server string = %h server (Samba %v)
log file = /var/log/samba/log.%m
max log size = 1024

; security = user
encrypt passwords = true
passdb backend = tdbsam guest
guest account = nobody
invaluid users = root

preserve case = yes
short preserve case = yes

; include /home/samba/etc/smb.conf.%m
socket options = TCP_NODELAY

[homes]
comment = Home directories
browseable = no
writable = no
create mask = 0700
directory mask = 0700

[shared]
comment = Public shared directory
path = /public/shared
public = yes
browseable = yes
; read only = no
writable = yes
; printable = no
create mask = 0766
directory mask = 0766
guest ok = yes
; valid users = user_name
---

testparm /etc/samba/smb.conf

/etc/init.d/samba restart

smbpasswd -a nobody
New SMB password:
Retype new SMB password:

log file (default):
/var/log/samba/log.%m

smbtree # find Windows machines

-----

UFW (Uncomplicated Firewall):

apt-get install ufw

ufw version # Show UFW version
ufw 0.36
Copyright 2008-2015 Canonical Ltd.

# Enable support for IPv6
jed /etc/default/ufw
IPV6=yes

# Restart (disable and enable) ufw to make changes to the configuration into effect

ufw app list # List of available apps on ufw

ufw default deny incoming
ufw default allow incoming

ufw default deny incoming
ufw default allow outgoing

ufw allow ssh
ufw allow 22/tcp

ufw allow http
ufw allow 80/tcp

ufw allow https
ufw allow 443/tcp

ufw allow 5000:6000/tcp
ufw allow 5000:6000/udp

ufw allow from 192.0.2.100
ufw deny from 192.0.2.100
ufw allow from 192.0.2.110 to any port 22

ufw allow from 192.0.2.100/24
ufw allow from 192.0.2.100/24 to any port 22

ufw delete deny from 192.0.2.100
ufw delete allow from 192.0.2.100

ufw status numbered

ufw delete NUMBER # delete the rule with the given number

ufw reset # reset all rules

ufw disable # stop ufw
Firewall stopped and disabled on system startup

ufw enable # start ufw
Command may disrupt existing ssh connections. Proceed with operation (y|n)? y
Firewall is active and enabled on system startup

echo 'y' | ufw enable # start ufw with automatic confirmation
Command may disrupt existing ssh connections. Proceed with operation (y|n)? Firewall is active and enabled on system startup

ufw --force enable # start ufw without confirmation prompt
Firewall is active and enabled on system startup

ufw disable; ufw enable; # restart ufw

ufw disable; ufw --force enable; # force restart ufw


ufw status
Status: inactive

ufw status
Status: active

To                         Action      From
--                         ------      ----
Anywhere                   DENY        192.0.2.100
22                         ALLOW       192.0.2.110
22/tcp                     ALLOW       Anywhere
80/tcp                     ALLOW       Anywhere
443/tcp                    ALLOW       Anywhere
22/tcp (v6)                ALLOW       Anywhere (v6)
80/tcp (v6)                ALLOW       Anywhere (v6)
443/tcp (v6)               ALLOW       Anywhere (v6)


ufw status verbose
Status: active
Logging: on (low)
Default: allow (incoming), allow (outgoing), disabled (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
Anywhere                   DENY IN     192.0.2.100
22                         ALLOW IN    192.0.2.110
22/tcp                     ALLOW IN    Anywhere
80/tcp                     ALLOW IN    Anywhere
443/tcp                    ALLOW IN    Anywhere
22/tcp (v6)                ALLOW IN    Anywhere (v6)
80/tcp (v6)                ALLOW IN    Anywhere (v6)
443/tcp (v6)               ALLOW IN    Anywhere (v6)


ufw version ; ufw status verbose | head --lines=4 ; ufw status numbered | grep -v 'Status' ;
Status: active
Logging: on (low)
Default: allow (incoming), allow (outgoing), disabled (routed)
New profiles: skip

     To                         Action      From
     --                         ------      ----
[ 1] Anywhere                   DENY IN     192.0.2.100
[ 2] 22                         ALLOW IN    192.0.2.110
[ 3] 22/tcp                     ALLOW IN    Anywhere
[ 4] 80/tcp                     ALLOW IN    Anywhere
[ 5] 443/tcp                    ALLOW IN    Anywhere
[ 6] 22/tcp (v6)                ALLOW IN    Anywhere (v6)
[ 7] 80/tcp (v6)                ALLOW IN    Anywhere (v6)
[ 8] 443/tcp (v6)               ALLOW IN    Anywhere (v6)

-----

Scripting / Linux Shell scripts:

# First line headers

Bourne Shell (sh)
#!/bin/sh

Bourne Again Shell (bash)
#!/bin/bash

C-Shell (csh)
#!/bin/csh

Korn Shell (ksh)
#!/bin/ksh

Perl
#!/usr/bin/perl

PHP
#!/usr/bin/php

Python
#!/usr/bin/python

Python3
#!/usr/bin/env python3

Turbo Shell (tcsh)
#!/usr/bin/tcsh

TCL/Tk
#!/usr/bin/wish -f

True
#!/bin/true
# Do not execute the script and exit with a Success (0) exit status

False
#!/bin/false
# Do not execute the script and exit with a Failure (1) exit status

Unless you expressely need to code for a specific shell, your choice of header should be #!/bin/bash (that is, Bourne Again Shell (bash))


Set the executable flag:
chmod +x FILE # declares the file as executable


# Parameters
if [ "$#" -eq 0 ] || [ -z "$1" ]; then echo "Missing parameters"; fi

# Variables
varn=1 # number (floating or integer)
vars="hi" # string of characters

echo $varn
echo $vars

# Arrays
arr=(10 4 5 20 8)

echo "Array Length: ${#arr[@]}";
echo -n "Highest number in array: "; printf "%d\n" "${arr[@]}" | sort -nr | head --lines=1
echo -n "Lowest  number in array: "; printf "%d\n" "${arr[@]}" | sort -nr | tail --lines=1

msga[0]="Hello"; msga[1]="World";
msgb=("Hello" "World")
msgc=([3]="Hello" [5]="World")

echo "${msga[0]} ${msga[1]}"
echo "${msgb[0]} ${msgb[1]}"
echo "${msgc[3]} ${msgc[5]}"

printf '%s\n' "${arr[@]}" # print the whole content of the array arr, separated by new lines
printf '%s, ' "${arr[@]}" # print the whole content of the array arr, separated by commas and spaces (note that there will be a comma and a space after the last element in the array too)
printf '%s, ' "${arr[@]}"  | rev | cut -c 3- | rev # print the whole content of the array arr, separated by commas and spaces and removing the trailing comma and space

# Functions

# Declare a function with function
function helloworld { echo "Hello World"; }

# Declare a function with round brackets
hellow () { echo "Hello World"; }

# Declare a function that accepts parameters
helloany () { echo "Hello $1"; }

# Return a value from a function
function sum() {
local r=0;
if [ $# -ge 1 ]; then
for i in "$@"; do
local r=$(( $r+$i ));
done
echo -n $r;
return 0;
else
echo -n $r;
return 1;
fi
}

# Call functions
helloworld
hellow
helloany "everybody"
echo $( sum 3 4 2 ); echo $( sum 5 5 ); echo $( sum 2 0 ); echo $( sum 6 ); echo $( sum );


# Branching

# if ... then ... elif ... else ... fi

# Check if a variable is empty
if [ -z "$v1" ]; then echo "The variable is empty"; fi
if [ -n "$v1" ]; then echo "The variable is NOT empty"; fi
if [ ! -z "$v1" ]; then echo "The variable is NOT empty"; fi
if [ ! -n "$v1" ]; then echo "The variable is empty"; fi

# Compare numbers
$v1=5; $v2=8;
if [ $v1 -eq $v2 ]; then echo "$v1 is equal to $v2"; fi
if [ $v1 -ne $v2 ]; then echo "$v1 is not equal to $v2"; fi
if [ $v1 -gt $v2 ]; then echo "$v1 is greater than $v2"; fi
if [ $v1 -ge $v2 ]; then echo "$v1 is greater than or equal to $v2"; fi
if [ $v1 -lt $v2 ]; then echo "$v1 is less than $v2"; fi
if [ $v1 -le $v2 ]; then echo "$v1 is less than or equal to $v2"; fi
if (( "$v1" == "$v2" )); then echo "$v1 is equal to $v2"; fi
if (( "$v1" != "$v2" )); then echo "$v1 is not equal to $v2"; fi
if (( "$v1" > "$v2" )); then echo "$v1 is greater than $v2"; fi
if (( "$v1" >= "$v2" )); then echo "$v1 is greater than or equal to $v2"; fi # >= and <= don't work with [[ ]]
if (( "$v1" < "$v2" )); then echo "$v1 is less than $v2"; fi
if (( "$v1" <= "$v2" )); then echo "$v1 is less than or equal to $v2"; fi # >= and <= don't work with [[ ]]
if (( "$v1" == "$v2" )); then echo "$v1 is equal to $v2"; elif (( "$v1" < "$v2" )); then echo "$v1 is less than $v2"; else echo "$v1 is greater than $v2"; fi

# Compare strings
$v1="hello"; $v2="world";
if [[ "$v1" == "$v2" ]]; then echo "$v1 is equal to $v2"; fi
if [[ "$v1" != "$v2" ]]; then echo "$v1 is not equal to $v2"; fi
if [[ "$v1" == "$v2"* ]]; then echo "$v1 begins with $v2"; fi # Wildcards don't work with (( ))
if [[ "$v1" == *"$v2" ]]; then echo "$v1 ends with $v2"; fi # Wildcards don't work with (( ))
if [[ "$v1" == *"$v2"* ]]; then echo "$v1 contains $v2"; fi # Wildcards don't work with (( ))
if [[ "$v1" == *"$v2"* ]] && [[ "$v1" != *"$v2" ]] && [[ "$v1" != "$v2"* ]]; then echo "$v1 contains with $v2, but not at the beginning or at the end."; fi # Wildcards don't work with (( ))
if (echo "$v1" | grep -Eq $regex); then echo "$v1 matches $regex"; fi # This is the POSIX way to match Regular Expressions.
if [[ "$v1" =~ $regex ]]; then echo "$v1 matches $regex"; fi # This is Bash specific to match Regular Expressions. If you get a "[[: not found" error in your script, remember to add #!/bin/bash at the top of your script.
if [[ "$v1" > "$v2" ]]; then echo "$v1 is greater than $v2"; fi
if [[ "$v1" < "$v2" ]]; then echo "$v1 is less than $v2"; fi
if [[ "$v1" == "$v2" ]]; then echo "$v1 is equal to $v2"; else echo "$v1 is not equal to $v2"; fi

# Compare versions
v1="3.50.2"; v2="4.2.3";
if [[ "$v1" == "$v2" ]]; then echo "$v1 is equal to $v2"; fi
if [[ "$(sort --version-sort <<< "$(printf '%s\n' "$v1" "$v2")" | head --lines=1)" = "$v1" ]]; then echo "$v1 is lower than or equal to $v2"; fi
if [[ "$(sort --version-sort <<< "$(printf '%s\n' "$v1" "$v2")" | head --lines=1)" != "$v1" ]]; then echo "$v1 is higher than $v2"; fi
if [[ "$v1" == "$v2" ]]; then echo "$v1 is equal to $v2"; elif [[ "$(sort --version-sort <<< "$(printf '%s\n' "$v1" "$v2")" | head --lines=1)" = "$v1" ]]; then echo "$v1 is lower than $v2"; else echo "$v1 is higher than $v2"; fi

# File System
if [ -e $v1 ]; then echo "$v1 exists as a File or a Directory"; fi
if [ -f $v1 ]; then echo "$v1 is a File"; fi
if [ -d $v1 ]; then echo "$v1 is a Directory"; fi
if [ -L $v1 ]; then echo "$v1 is a SymLink"; fi
if [ -s $v1 ]; then echo "$v1 file size is greater than 0"; fi
if [ -N $v1 ]; then echo "$v1 was modified after its last read"; fi
if [ -O $v1 ]; then echo "$v1 is owned by current user"; fi
if [ -G $v1 ]; then echo "$v1 is owned by current user's group"; fi
if [ -r $v1 ]; then echo "$v1 has Read permission"; fi
if [ -w $v1 ]; then echo "$v1 has Write permission"; fi
if [ -x $v1 ]; then echo "$v1 has Execute permission"; fi
if [ $v1 -ef $v2 ]; then echo "$v1 and $v2 are hard links to the same file"; fi
if [ $v1 -nt $v2 ]; then echo "$v1 is newer than $v2"; fi
if [ $v1 -ot $v2 ]; then echo "$v1 is older than $v2"; fi

# Check for File or Directory
if [[ -d $v ]]; then
echo "Directory"
elif [[ -f $v ]]; then
echo "File"
else
echo "Not found"
fi


# case

# Config
onekeyc=1; # Single Keypress for menu choice, rather than having to hit Enter (1: True, 0: False)

echo -n "Enter choice ";
if [ $onekeyc -eq 1 ]; then
read -r -n 1 -p "" choice # Key press
echo ;
else
read -r -p "" choice # Type choice and hit Enter
fi

case $choice in
"1") echo "1" ;;
"2") echo "2" ;;
"q"|"Q"|"x"|"X") echo "Q or X, upper or lower case" ;;
*) echo "Default" ;;
esac


# Looping

# for

for i in 1 2 3 4 5
do
echo "N. $i"
done
echo

for i in {1..5}
do
echo "N. $i"
done
echo

for (( i=1; i<=5; ++i ))
do
echo "N. $i"
done
echo

for (( i=1; i<=10; i=i+2 ))
do
echo "N. $i"
done
echo

# Bash v4.0 and greater support step
for i in {1..10..2}
do
echo "N. $i"
done
echo

# Before than Bash v4.0 step could be achieved with the additional seq command
for i in $(seq 1 2 10)
do
echo "N. $i"
done
echo

# List all elements in given list
i=0;
for el in Hello World !;
do
i=$(( i + 1 ))
echo "Element N. $i: $el"
done
echo

# List all elements in given list in a variable
str="Hello World !"
i=0;
for el in $str;
do
i=$(( i + 1 ))
echo "Element N. $i: $el"
done
echo

# List all elements in given array
arr=("Hello" "World" "!")
i=0;
for el in "${arr[@]}";
do
i=$(( i + 1 ))
echo "Element N. $i: $el"
done

# List all files in PATH
for file in PATH/* ;
do
echo $file
done

for (( ; ; ))
do
echo "infinite loop (kill the process, or terminate it with CTRL+C)"
done


# while

i=1
while (( $i <= 5 ))
do
echo "N. $i"
(( ++i ))
done
echo

i=0
while (( $i <= 10 ))
do
i=$(( i + 2 ))
if (( i == 4 )); then continue; fi
echo "N. $i"
if (( i >=8 )); then break; fi
done
echo

while :
do
echo "infinite loop (kill the process, or terminate it with CTRL+C)"
done


# until

i=1
until (( $i > 5 ))
do
echo "N. $i"
(( ++i ))
done
echo

i=0
until (( $i > 10 ))
do
i=$(( i + 2 ))
if (( i == 4 )); then continue; fi
echo "N. $i"
if (( i >=8 )); then break; fi
done
echo

until false :
do
echo "infinite loop (kill the process, or terminate it with CTRL+C)"
done

-----

Invoke Linux shell commands from PHP:

PHP can perform calls to the Linux shell using the shell_exec() function:

 $output = shell_exec('ls -laF');
 echo '
'.$output.'
';

or enclosing the Linux shell command between backticks:

 $output = `ls -laF`;
 echo '
'.$output.'
';

PHP also has a system() function that returns the value returned by the Linux shell command and the last line of output of such command:

 $last_line = system('ls -laF', $returned_value);

-----

GUI commands
# Commands for the Graphical User Interface

xwinifo # Enter this command and then click on any window to get information about it such as size in pixels and color depth. You may also click on the desktop.

sudo rm -rf  ~/.local/share/Trash/files/* # Empty trashcan

-----

Fortune cookies:
apt-get install fortunes
/usr/games/fortune  # return a fortune cookie (fetches a random sentence (% separated lines) from a random file in /usr/share/games/fortunes/ )
/usr/games/fortune PATH/FILE  # return a fortune cookie fetching a random sentence (% separated lines) from the given FILE

To create your own fortune files:
- Create a text file separating each fortune cookie with lines containing only a percent (%) sign.
- Create a database file for that text file, with the same file name followed by a .dat extension, and place it in the same directory: strfile PATH/FILENAME PATH/FILENAME.dat

-----

ASCII Art:

Labyrinth:
c=(\/ \\); n=${#c[@]}; clear; printf "%s" "${c[RANDOM%n]"{1..1920}"}"
c=(\/ \\); n=${#c[@]}; clear; while :; do printf -- "${c[RANDOM%n]}"; done # Endless labyrinth (break with CTRL+C)

apt-get install figlet

figlet "Hello" # Type Hello using ASCII Art

apt-get install cowsay

/usr/games/cowsay "Hello" # Show a cow saying Hello
/usr/games/cowthink "Hello" # Show a cow thinking Hello
/usr/games/cowsay -f CHARACTER "Hello" # Use alternate CHARACTERs, for example: tux, moose, sheep, milk
/usr/games/cowthink -f CHARACTER "Hello" # Use alternate CHARACTERs, for example: tux, moose, sheep, milk
ls /usr/share/cowsay/cows # Show available cows (characters)
/usr/games/fortune|/usr/games/cowsay # Fortune teller cow (requires fortune cookies to be installed)
/usr/games/cowsay -f "$(ls /usr/share/cowsay/cows | sort -R | head -1)" "$(/usr/games/fortune -s)" # Random character tells fortune

apt-get install sl

sl # show a running steam locomotive in ASCII Art
sl -c # show a small steam locomotive
sl -F # show a flying steam locomotive
sl -a # show people crying for help into a steam locomotive
sl -a # allow the animation to be interrupted with CTRL+C

apt-get install lolcat

echo 'Somewhere over the rainbow' | /usr/games/lolcat # Return the given text as a raibow
echo 'Somewhere over the rainbow' | /usr/games/lolcat -a # Return the given text as an animated raibow
/usr/games/fortune|/usr/games/cowsay|/usr/games/lolcat # Fortune teller cow in a rainbow (requires fortune cookies and cowsay to be installed)
/usr/games/fortune|/usr/games/cowsay|/usr/games/lolcat -a # Fortune teller cow in an animated rainbow (requires fortune cookies and cowsay to be installed)

-----

Text editors:

Jed

apt-get install jed

jed FILE.txt # Create or edit FILE.txt
jed FILE.txt -g LINE # Open FILE.txt for editing, set cursor position at the given LINE
jed FILE.txt -s "TEXT" # Open FILE.txt for editing, set cursor position at the first occurrence of the given TEXT


Vim

apt-get install vim

vim FILE.txt # Create or edit FILE.txt (cursor on the first line)
vim + FILE.txt # Edit FILE.txt and place cursor on the last line
vim +N FILE.txt # Edit FILE.txt and place cursor on line N

# While Jed is an easy and intuitive text editor, it might happen that it's not available in the system on which you are working.
# In this case you'll likely have to use vim instead which is powerful in trained hands but much less intuitive.
# This is a list of some basic and useful commands

Cursor keys / h, l, j, k : Cursor movement (left, right, down, up)
b : Jump to the word on the left
w : Jump to the word on the right
( : Jump to the sentence on the left
) : Jump to the sentence on the right
^ / 0 : Jump to the beginning of the line
$ / gm : Jump to the end of the line
- : Jump to the first non blank character of the previous line
+ : Jump to the first non blank character of the next line
{ : Jump to the previous paragraph
} : Jump to the next paragraph
[[ : Jump to the beginning of the file
]] : Jump to the end of the file
[LINE_NUMBER]G / [LINE_NUMBER]gg : Jump to the given LINE NUMBER
[VALUE]% : Jump to the specified position of the document
[COLOUMN]| : Jump to the specified COLOUMN in the current line
[LINE]H : Jump to the given LINE visible in the current window, from the top
[LINE]L : Jump to the given LINE visible in the current window, from the bottom
M : Jump to the line in the middle of the current window

zt : Place current line at the top of the window
zz : Place current line at the middle of the window
zb : Place current line at the bottom of the window

i : Enter Insert mode
I : Jump at the beginning of the line and enter Insert mode
s : Delete the character under the cursor and enter Insert mode
R : Enter Replace mode
ESC : Exit Edit mode (either Insert or Replace (Overwrite))

Commands in Edit mode (^ means CONTROL):
^w : Delete characters from cursor position to the beginning of the current word
^u : Delete characters from cursor position to the beginning of the current line
^t : Add a Tab at the beginning of the current line (Add indentation)
^r : Remove a Tab from the beginning of the current line (Remove indentation)

r : Replace a single character at the cursor position
o : Create a new empty line below the current one and enter Insert mode
O : Create a new empty line above the current one and enter Insert mode
gu[CURSOR_KEY] : Change a single character to lowercase in the direction specified with the cursor key
gU[CURSOR_KEY] : Change a single character to uppercase in the direction specified with the cursor key

x : Remove the character at the cursor (Delete)
X : Remove the character before than the cursor (Backspace)
D : Delete until the end of the current line
dd : Delete the current line
J : Join the current line with the next one adding a space inbetween

Y / yy : Copy the current line
P / p : Paste the line in the clipboard 

ga : Show the ASCII value of the character at the cursor position

/[TEXT] : Search for the given TEXT
n : Repeat last search, forward
N : Repeat last search, backwards

:s/[SEARCH]/[REPLACE]/ : Search and Replace the first occurrence in the current line only, case sensitive
:s/[SEARCH]/[REPLACE]/g : Search and Replace every occurrence in the current line only, case sensitive
:%s/[SEARCH]/[REPLACE]/g : Search and Replace every occurrence in the whole file, case sensitive
:%s/[SEARCH]/[REPLACE]/gI : Search and Replace every occurrence in the whole file, case sensitive
:%s/[SEARCH]/[REPLACE]/gi : Search and Replace every occurrence in the whole file, case insensitive
:%s/[SEARCH]/[REPLACE]/gic : Search and Replace every occurrence in the whole file, case insensitive, and ask for confirmation

:%s/\s\+$// : Remove trailing spaces in the whole file
:%s/^\s\+// : Remove leading spaces in the whole file

:set number / :set nu : Display line numbers
:set nonumber / :set nonu : Turn off line numbers
:set number! / :set nu! : Toggle line numbers
:set relativenumber : Display line numbers relative to the cursor line
:set norelativenumber : Remove line numbers relative to the cursor line
:set relativenumber! : Toggle line numbers relative to the cursor line
# If number and relativenumber are both enabled the absolute line number will be shown in the cursor line (instead of 0)

:set list : Display special characters
:set nolist : Hide special characters
:set list! : Toggle special characters
:set listchars=eol:$,tab:>-,trail:~,nbsp=! : Configure how to show special characters
:help listchars : Display help for listchars (quit with :q)

:syntax on : Enable syntax
:syntax off : Disable syntax
:set syntax=[LANGUAGE] : Display syntax for the specified language (example: html, php, whitespace)

:w : Save current file
:wq / :x / ZZ : Save current file and quit
:q : Quit (it only quits if the current file is saved, otherwise returns an error message)
:q! : Quit without saving (doesn't ask for confirmation)

-----

Easter eggs:

$ apt-get moo
         (__)
         (oo)
   /------\/
  / |    ||
 *  /\---/\
    ~~   ~~
...."Have you mooed today?"...

$ apt-get -h
[...]
                       This APT has Super Cow Powers.

$ aptitude -h
[...]
                  This aptitude does not have Super Cow Powers.

$ aptitude moo
There are no Easter Eggs in this program.
$ aptitude -v moo
There really are no Easter Eggs in this program.
$ aptitude -vv moo
Didn't I already tell you that there are no Easter Eggs in this program?
$ aptitude -vvv moo
Stop it!
$ aptitude -vvvv moo
Okay, okay, if I give you an Easter Egg, will you go away?
$ aptitude -vvvvv moo
All right, you win.

                               /----\
                       -------/      \
                      /               \
                     /                |
   -----------------/                  --------\
   ----------------------------------------------

Happy?
$ aptitude -vvvvvv moo
What is it?  It's an elephant being eaten by a snake, of course.

# Note: aptitude's easter egg is a reference to "The Little Prince" written in 1943 by Antoine de Saint-Exupéry.

-----