Bash for SAs

Why Bash?

The simple answer? Pretty much any Unix-based system is going to have some version of Bash on it. Even languages such as python aren't guaranteed because not every system uses systemd and thus doesn't require it (eg. AIX) - but you'll be hard-pressed to find a system without bash in it.

Bash version differences cheatsheet

Each new version of bash adds new features and changes or removes others. It's important to know the capabilities of all of the systems you plan on running your bash scripts on.

I took the liberty of summarizing the most notable changes in each version of bash. The full list can be found here.

5.2 (09/26/2022)

kg-card-begin: html

Feature Summary
read Timeout read command now supports fractional timeouts.
Quoting Adds $'...' and $"..." quoting options.
Parameter Transformation Adds @k operator for splitting strings into words after transformation.
printf Specifier Adds %Q specifier for quoting strings.
Globskip Dots Prevents . and .. from appearing in pathname expansions by default.
Suspend Option suspend -f now forces suspension even without job control.

kg-card-end: html

5.1 (12/07/2020)

kg-card-begin: html

Feature Summary
read with File Descriptors read -e now supports custom file descriptors (e.g., read -u N).
wait PID Storage wait can store the PID of processes using -p VARNAME.
Case Conversion Adds U, u, and L options to change string case.
Enhanced Job Reporting jobs can show completed jobs when used with -c.
Array Variable Display PROMPT_COMMAND can now be an array, with each item as a command.

kg-card-end: html

5.0 (01/07/2019)

kg-card-begin: html

Feature Summary
wait with Process Substitution wait now supports waiting for the last process substitution.
Epoch Time Variables Adds EPOCHSECONDS and EPOCHREALTIME for Unix epoch time.
history -d Enhancements Allows deleting ranges of history entries with -d start-end.
Case-Insensitive Completion Command completion matches aliases and functions case-insensitively.
Flexible umask Values umask supports modes and masks beyond octal 777.

kg-card-end: html

4.4 (09/15/2016)

kg-card-begin: html

Feature Summary
mapfile Enhancements mapfile supports custom delimiters with -d and can strip delimiters with -t.
Command Timing Supports time ; othercommand to time null commands.
Filename Ignore List EXECIGNORE variable can ignore specific filenames when searching for commands.
Prompt String Expansion New PS0 string for showing prompts before command execution.
unset Enhancement unset can now unset scalar variables using subscript 0.

kg-card-end: html

4.3 (02/26/2014)

kg-card-begin: html

Feature Summary
Unlimited History Option Setting HISTSIZE or HISTFILESIZE to negative values enables unlimited history.
Enhanced read read now skips NUL bytes in input and is interruptible by signals in POSIX mode.
Negative Array Indexing Allows referencing array elements using negative indices (e.g., array[-1]).
Wait for Any Child wait -n waits for any child process to change state.
New Special Variable BASH_COMPAT sets compatibility level for different Bash versions.

kg-card-end: html

4.2 (02/13/2011)

kg-card-begin: html

Feature Summary
Unicode Escapes $'...', echo, and printf now support \u and \U Unicode escapes.
test Enhancement Adds -v variable option to check if a variable is set.
Negative Array Indexes Indexed arrays support negative subscripts, counting from the end.
printf Time Formatting New %(fmt)T format specifier for time formatting with strftime-like syntax.
Last Pipe Option lastpipe option allows last pipeline command to run in the current shell.

kg-card-end: html

4.1 (12/31/2009)

kg-card-begin: html

Feature Summary
printf Array Support printf -v can assign values to array indices.
Enhanced read New -N nchars option reads exact character counts, ignoring delimiters.
Conditional String Comparison [[ < and > comparisons respect locale settings.
Syslog History Option New option to forward all command history entries to syslog.

kg-card-end: html

4.0 (02/20/2009)

kg-card-begin: html

Feature Summary
BASHPID Variable New $BASHPID variable provides the process ID of the current shell.
Automatic cd The autocd option allows automatic directory changes with directory name input.
globstar Option Enables ** to match directories and files recursively.
Append Redirect Operator &>> appends both stdout and stderr to a file.
Case Modification Adds expansions for uppercase (^) and lowercase (,), applicable globally.

kg-card-end: html

3.2 (10/11/2006)

kg-card-begin: html

Feature Summary
Pattern Matching Update [[ =~ ]] now supports quoted strings for exact string matching.
Home Variable in POSIX Mode $HOME is no longer automatically set in POSIX mode.
MacOS Compatibility Loadable builtins are now supported on MacOS 10.3 and 10.4.

kg-card-end: html

3.1 (12/08/2005)

kg-card-begin: html

Feature Summary
Append Operator Adds += operator for appending to strings and arrays.
Case-Insensitive Matching New nocasematch option for case-insensitive matching in case and [[ commands.
printf Output to Variable printf now supports -v var option to write output directly to a variable.
Environment Inheritance Inherits $_ from environment if set at startup.
POSIX Conformance Options Strict POSIX mode added, setting compliance by default if enabled.

kg-card-end: html

3.0 (08/03/2004)

kg-card-begin: html

Feature Summary
Brace Expansion Adds {x..y} expansion for sequences, e.g., {1..5}.
Case-Insensitive History Ignore HISTCONTROL's erasedups is now case-insensitive.
Regular Expression Matching [[ =~ ]] supports extended regex matching in conditional expressions.
Fail Glob Option New failglob option triggers error if pathname expansion has no match.
pipefail Option set -o pipefail returns failure if any command in a pipeline fails.

kg-card-end: html

2.05a/b (04/09/2001, 11/16/2001, 07/17/2002)

kg-card-begin: html

Feature Summary
/dev/tcp and /dev/udp Supports service names in redirections, not just port numbers.
complete Options Adds options for fallback completion (e.g., directory or filename) if no match.
Here-String Redirection Adds <<< operator for here-strings, redirecting a string as stdin.
printf Escape Updates New %q format for quoting special characters and \\cX for Control-X.

kg-card-end: html

2.04 (03/21/2000)

kg-card-begin: html

Feature Summary
Enhanced read Options read command adds -t (timeout), -n (character limit), -d (delimiter), and -s (silent).
TCP and UDP Redirection Supports /dev/tcp/host/port and /dev/udp/host/port for network connections.
Programmable Completion Adds complete and compgen builtins for custom autocompletions.
FUNCNAME Variable New variable, FUNCNAME, stores the name of the currently executing function.
HISTCONTROL Enhancements Adds erasedups to remove duplicate entries from command history.

kg-card-end: html

2.03 (02/19/1999)

kg-card-begin: html

Feature Summary
Restricted Shell Option New shopt option, restricted_shell, to indicate restricted shell mode.
Auto-Export OLDPWD OLDPWD is now automatically exported, aligning with POSIX.2 requirements.
Login Option for Non-Interactive Shells Non-interactive shells with --login source login startup files.

kg-card-end: html

2.02 (04/18/1998)

kg-card-begin: html

Feature Summary
Extended Pattern Matching Adds ksh-style extended globbing (e.g., [@+*?!](pattern)) with extglob option.
[[ Command New [[ command adds extended test functionality.
printf Builtin Adds printf as a builtin, following POSIX standards.
Command Substitution $(<filename) syntax for reading a file’s contents without cat.
Case-Insensitive Globbing New nocaseglob option for case-insensitive file pattern matching.

kg-card-end: html

2.01 (06/05/1997)

kg-card-begin: html

Feature Summary
GROUPS Variable Adds GROUPS array variable, listing all groups the user belongs to.

kg-card-end: html

2.0 (12/31/1996)

kg-card-begin: html

Feature Summary
time Reserved Word Adds time keyword for timing pipelines, commands, and functions.
ANSI and Locale Quoting $'...' and $"..." quoting for ANSI-C escapes and locale-specific translations.
Array Support Adds integer-indexed arrays with flexible indexing and assignment.
Enhanced Prompt Customization New prompt expansions: \a, \e, \H, \T, \@, \v, \V.
Indirect Expansion Adds ${!var} syntax for indirect variable references (equivalent to eval).

kg-card-end: html

Basics

This is intended as a cheatsheet and crash-course for SAs looking to automate things with Bash, so I won't be going into great detail about any particular thing. If you're an SA, you know how to use Google. I hope.

Bash scripts and the command-line are effectively the same thing. You can paste an entire script into your terminal and it will run fine, and vice-versa.

When I write bash scripts, I'll usually test each line I'm unsure of on the machine or machines on which I plan to run the script- as long as the line doesn't actually make any changes to the OS that I can't easily undo.

The main thing to note is the shebang at the top of a script file (#!/bin/bash etc) - it's usually recommended to use #!/usr/bin/env bash for portability reasons but there's a lot of history and debate online around anything that you can put into a script and its "portability" - so, using #!/bin/bash, as long as it works for your use-case, is perfectly fine.

Variables

Variables can be set for the current shell by simply doing the following:

variable_1="x"
VARIABLE_2="y"

You can set variables for the current shell and any subprocesses it spawns by doing the following instead:

export variable_1="x"
export VARIABLE_2="y"

or, if the variable was already set somewhere else:

export variable_1
export VARIABLE_2

To use variables in your script later, it's usually recommended to use the "${}" syntax.

eg.

echo "${variable_1}"

this allows for fancy things like replacements directly in the variable and also avoids problems like shell expansions.

Sometimes, though, you really need to do weird tuff like run yum or dnf based on the OS:

if command -v dnf; then
  pkg_cmd="dnf"
else
  pkg_cmd="yum"
fi

$pkg_command -y update

The above script would run dnf -y update if the system has dnf, and yum -y update if not.

If/elif/else

The basic syntax for an if/else if/else statement is:

if condition-1; then
  do-thing-1
elif condition-2; then
  do-thing-2
elif condition-3; then
  do-thing-3
else
  do-thing-4
fi

Return codes

Return codes in Bash indicate the success or failure of a command. A return code of 0 generally means success, while any non-zero code indicates an error. You can access the return code of the previous command using $?.

eg.

ls /nonexistent_directory
if [[ $? -ne 0 ]]; then
  echo "Command failed."
fi

Checking if a command exists

You can use command -v or which to check if a command is available:

if command -v cat &>/dev/null; then
  echo "cat is installed."
else
  echo "cat is not installed."
fi

Note that this works for Bash builtins (eg. timeout) as well as "real" commands located on the system.

$()

The $() syntax is used for command substitution in Bash. It allows you to capture the output of a command and use it as part of another command or assign it to a variable.

eg.

date_output=$(date)
echo "The current date and time is: $date_output"

This is equivalent to the older backtick syntax, but $() is preferred as it is more readable and can be nested easily.

The older syntax would look like this:

date_output=`date`
echo "The current date and time is: $date_output"

Combining the previous two sections, a handy trick I use sometimes is ls -l $(which cat) or vi $(which custom-command) to quickly find and edit a particular script registered as a command (eg. for Bash scripts located in ~/.local/bin)

Checking OS type and version

Almost every Unix-based distro will support os-release, which is really just a file located at /etc/os-release.

For example, to determine if a system is RHEL-based or Debian-based, you can use the ID_LIKE variable located in it:

ID_LIKE=$(grep ^ID_LIKE /etc/os-release | awk -F'[="]' '{print $3}')

or, even better:

source /etc/os-release

Sourcing the file will put all of the content into variables like NAME, VERSION, ID, ID_LIKE, etc for use in your script.

Then, you can do something like the following:

if [[ "${ID_LIKE}" == *rhel* ]]; then
  echo "RHEL OS!"
elif [[ "${ID_LIKE}" == *debian* ]]; then
  echo "Debian OS!"
fi

Strings

Strings in Bash can be enclosed with single (') or double (") quotes. Single quotes preserve literal value, while double quotes allow variable interpolation.

name="John"
echo 'Hello, $name'  # Output: Hello, $name
echo "Hello, $name"  # Output: Hello, John

Numbers & Math

Bash allows simple arithmetic using $((...)) syntax:

num1=5
num2=10
sum=$((num1 + num2))
echo $sum  # Output: 15

These operators are used for numeric comparisons in conditional expressions:

  • -eq: Equal
  • -ne: Not equal
  • -gt: Greater than
  • -lt: Less than

eg.

if [[ num1 -gt 2 ]]; then
  echo "$num1 > 2"
else
  echo "2 > $num1"
fi

-n and -z

-n and -z are used to test strings:

  • -nchecks if a string is non-empty.
  • -zchecks if a string is empty.

eg.

if [[ -n "$MY_VAR" ]]; then
  echo "MY_VAR is not empty."
fi

if [[ -z "$MY_VAR" ]]; then
  echo "MY_VAR is empty."
fi

Arrays

Bash supports indexed arrays and (from version 4) associative arrays:

# Indexed Array
my_array=("apple" "banana" "cherry")
echo ${my_array[1]}  # Output: banana

# Associative Array
declare -A fruits
fruits["yellow"]="banana"
echo ${fruits["yellow"]}  # Output: banana

Difference between [, [[, and no brackets

  • [is a synonym fortestand is used for basic conditional tests.
  • [[is a Bash-specific keyword that supports more complex conditions and pattern matching.
  • No brackets are used for simple commands or variable assignment.

eg.

if command -v cat; then
  echo "Using no brackets."
fi

if [ "$var" = "value" ]; then
  echo "Using single bracket."
fi

if [[ "$var" == "value" ]]; then
  echo "Using double brackets."
fi

To terminal or not to terminal?

You can check if a script is connected to a terminal to decide whether to enable terminal-specific features, like colored output:

if [[ -t 1 ]]; then
  TERMINAL=true
else
  TERMINAL=false
fi

Inline if/then

Bash supports inline if/then statements using the && and || operators:

[[ -n "$MY_VAR" ]] && echo "MY_VAR is not empty" || echo "MY_VAR is empty"

Output redirection

Output redirection in Bash allows you to direct the output of commands to files or other outputs. The most common operators are:

  • >: Redirect standard output to a file (overwrites the file).
  • >>: Redirect standard output to a file (appends to the file).
  • 2>: Redirect standard error to a file.
  • &>: Redirect both standard output and standard error to a file.

eg.

# Overwrite output to a file
ls > output.txt

# Append output to a file
echo "Additional line" >> output.txt

# Redirect error messages to a file
ls /nonexistent_directory 2> error.log

# Redirect both output and errors
echo "Test" &> combined.log

or, if you're feeling extra fancy, you can do some cool stuff with cat:

cat <<EOF > myfile.txt
# look, ma, a comment!
do x and y plz
and try ${favorite_piza}!
EOF

You can also output to devices, like /dev/null. As a useful example, to run a find command on your entire (local) system but skip any errors:

find / -xdev -type f -name "*xyz*" -ls 2> /dev/null

Checking connectivity

Connectivity checks can be done in a few ways, but the most reliable would be with a pure socket implementation, which has been supported in bash since version 2.04.

Pure sockets w/ timeout

This method uses timeout with Bash's /dev/tcp feature to check if a port is reachable:

timeout 10 bash -c "cat < /dev/null > /dev/tcp/server/port" &>/dev/null

This only works in modern Bash versions where timeout is available, however, so it's best to check if timeout is available with a quick if command -v timeout; then beforehand.

Pure sockets w/o timeout

This method uses /dev/tcp but has no set timeout, meaning it will block for the OS's default time if the connection doesn't succeed:

</dev/tcp/server/port &>/dev/null

In either case, you can get the result with a simple $? on the next line to see the return code. If it's non-zero, you have a problem.

Fancy techniques

Sometimes you need some fancy stuff to make your scripts pretty or make it work well. Here's a couple fun ones.

Getting current script path

To reliably determine the current script's path in a one-liner:

SCRIPT_PATH="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &> /dev/null && pwd)"

This method works even if the script is being sourced. Thanks to this SO answer.

Stripping ANSI colors

To strip ANSI color codes from a string:

echo -e "$1" | sed 's/\x1B\[[0-9]*\(\?:;[0-9]\)*\?m//g'

This is useful when you want to remove color formatting for logging or processing output, or when you don't have a terminal. You can check for all of these by simply doing a if [[ -t 1 ]]; then if/else case.

Largely thanks to the answers in this SO post, but I had to hand-edit the regex for it to work reliably across multiple versions and OSes.

For your bashrc

Here's a few useful things you can throw in your .bashrc (or .bash_profile)

enhanced grep

alias grep="grep --color=auto"
alias lgrepi="ls -la | grep --color=auto -i"
alias rgrepi="find \${PWD} -name '.*' -prune -o -type f -print0 | xargs -r0 -P2 grep --color=auto -HIni"
alias grepi="find \${PWD} -maxdepth 1 -name '.*' -prune -o -type f -print0 | xargs -r0 -P2 grep --color=auto -HIni"

And a quick summary of the aliases:

  • grephas been updated to add colors to the output when it can
  • lgrepisearches your current directory for filenames matching your query (case-insensitive)
  • rgrepisearches your current directory (and subdirectories) for content matching your query (case-insensitive)
  • grepiis the same asrgrepi, but won't recursively search subdirectories

rclone copy

Sometimes you need to copy a bunch of files fast, and with progress bars. rclone is a great way to do that but it doesn't really match the syntax of cp at all. Here's an rcp command that does just that:

function is_remote_mount() {
    dir="$1"
    grep -E ' nfs[3,4]* | cifs ' /proc/mounts | awk '{print $2}' | while read -r mount_point; do
        if [[ "$dir" == "$mount_point"* ]]; then
            return 0
        fi
    done
    return 1
}

function rclone_cp() {
    args=("$@")
    last_index=$(( $# - 1 ))
    dest="${args[$last_index]}"
    unset 'args[$last_index]'

    base_flags="--verbose --update --copy-links"
    # Enable progress if running in an interactive terminal
    if [ -t 1 ]; then
        base_flags="$base_flags --progress"
    fi

    temp_dir=$(mktemp -d)
    echo "Created temporary directory ${temp_dir}"
    trap 'rm -rf -- "$temp_dir"' EXIT

    for src in "${args[@]}"; do
        # Convert relative path to absolute path
        if [[ "$src" != /* ]]; then
            src="$PWD/$src"
        fi

        if [[ -d "$src" ]] && [[ "$src" != */ ]]; then
            # If source is a directory and does not end with '/', copy the directory itself
            ln -s "$src" "$temp_dir/"
        elif [[ -f "$src" ]]; then
            # If source is a file
            ln -s "$src" "$temp_dir/"
        fi
    done

    temp_flags="$base_flags"
    # Determine if source or destination is a remote mount
    if is_remote_mount "$dest"; then
        temp_flags="$temp_flags --transfers=16 --checkers=16 --buffer-size=512M"
    else
        temp_flags="$temp_flags --transfers=4 --checkers=4 --buffer-size=256M"
    fi
    # Determine if source or destination is remote (via rclone)
    if [[ "$dest" == *":"* ]]; then
        temp_flags="$temp_flags --transfers=32 --checkers=32 --buffer-size=1G --fast-list"
    fi

    rclone copy "$temp_dir" "$dest" $temp_flags
}

alias rcp='rclone_cp'