Shell notes

January 9, 2025
December 8, 2014

Notes on shell usage.
Most of the following are compatible to bash and zsh.

We all <3 Terminals. - Terminals Are Sexy
Philip James, Asheesh Laroia - Type python, press enter. What happens? - PyCon 2015 - YouTube how shell invocation works
Command Line Power User — A free video series for web developers on learning a modern command line workflow with ZSH, Z and related tools. !important, by Wes Bos
Coding like a Hacker in the terminal – Caleb Taylor – Medium
You’re Missing Out on a Better Mac Terminal Experience
The birth of the Bash shell | Opensource.com
Conquering the Command Line | Softcover.io
Bash Reference Manual
OverTheWire: Wargames learn shell commands through games

The Modern CLI Renaissance | Gabe Venberg
CLIs Are Making A Comeback - YouTube

Bash Automation & Scripting Basics (Part 1) – CloudSavvy IT
Bash Automation and Scripting Basics (Part 2) – CloudSavvy IT
Bash Automation and Scripting Basics (Part 3) – CloudSavvy IT

Choosing shell

Unix Shells: Bash, Fish, Ksh, Tcsh, Zsh - Hyperpolyglot
Evolution of shells in Linux
Rich’s sh (POSIX shell) tricks
the xonsh shell — xonsh 0.8.12 documentation

Most distro comes with bash as default login shell.
Since bash is ubiquitous and more feature-rich than POSIX shell (/bin/sh), it is recommended to write shell scripts in bash.
Since zsh is bash (and hence POSIX) -compatible. Most command lines found on the web can be pasted in zsh directly (unlike fish which is not POSIX-compatible). It is recommended to use zsh as the interactive shell.
However I found fish design quite sensible. I'll try to use external binaries as mush as possible instead of relying on shell features. On the same line of thought, even-though zsh has more powerful syntax than bash, it may not be worthy to use them to avoid lock-in and maintain portability.

Evolution of shells in Linux

Starship Successor of Fish shell's Spacefish
starship/starship: ☄🌌️ The cross-shell prompt for astronauts.
Configuration | Starship

igor-petruk/scriptisto: A language-agnostic "shebang interpreter" that enables you to write scripts in compiled languages.
rust-script — command-line utility in Rust // Lib.rs

rash-sh/rash: Declarative shell scripting container oriented

login shell

You can set a shell as your default shell with chsh.
But interactive shells must be added to /etc/shells first before used as a login shell.

http://fishshell.com/docs/2.1/faq.html#faq-default

Rust userland

Oxidise Your Life - YouTube

cargo-bins/cargo-binstall: Binary installation for rust projects
light4/cargo-info: cargo-info subcommand - show crates info from crates.io
kbknapp/cargo-graph: A cargo subcommand for creating GraphViz DOT files and dependency graphs
RazrFalcon/cargo-bloat: Find out what takes most of the space in your executable.
flamegraph-rs/flamegraph: Easy flamegraphs for Rust projects and everything else, without Perl or pipes <3
eza — command-line utility in Rust // Lib.rs

cargo-binstall nu # shell
cargo-binstall coreutils
cargo-binstall cargo-info
cargo-binstall eza bat fd-find du-dust ripgrep zellij
cargo-binstall mprocs # process manager
cargo-binstall wiki-tui
cargo-binstall rtx-cli # Polyglot runtime manager

Beginner's Guide to rtx (mise) - DEV Community
jdx/mise: dev tools, env vars, task runner

dotfiles

most contain useful aliases, functions and scripts

The Ultimate Guide to Mastering Dotfiles
Secure data in Git with the clean/smudge filter | Red Hat Developer

It is recommended to use source control to store your config files (dotfiles, as they usually begins with a .) and use symlinks (ln -s) to deploy them to the machine.

bash - Difference between .bashrc and .bash_profile - Super User
talks/linuxcon_eu-2013-10-gitify_your_life.pdf at main · RichiH/talks

GitHub does dotfiles - dotfiles.github.io
webpro/awesome-dotfiles: A curated list of dotfiles resources.
dotphiles/dotphiles: A community driven framework of dotfiles.

TODO: add my links in ~/caravan/github-watch/dotfiles/

Honorable mentions:

Yet Another Dotfiles Manager - yadm
direnv – unclutter your .profile | direnv
I'm never using .env files ever again - YouTube

RichiH/vcsh: config manager based on Git

AGWA/git-crypt: Transparent file encryption in git
.gitattributes · dots · GammaFn / dotfiles.cli · GitLab

lra/mackup: Keep your application settings in sync (OS X/Linux)
tuning — command-line utility in Rust // Lib.rs
hm — command-line utility in Rust // Lib.rs
quickcfg — command-line utility in Rust // Lib.rs

How to make your Dotfile management a painless affair
ajmalsiddiqui/autodot: A dotfile management system that makes sharing your dotfiles easy while keeping you in the loop.
RCM(7)

TechDufus/dotfiles: Fully automated development environment for TechDufus using ansible.
Automating your Dotfiles with Ansible: A Showcase - YouTube

dotfiler

svetlyak40wt/dotfiler: Shell agnostic git based dotfiles package manager, written in Python.
put your dotfiles into environments, each mirroring files/folder in your ~/
environments allow you to separate your configs into app-specific folders
dotfiler supports common folder in different environments by creating the common folder and symlinking each file from the environments
multiple repos can be used to merge common and machine-specific configs

chezmoi

Home | chezmoi.io also manage default apps with run_once_*
twpayne/chezmoi: Manage your dotfiles across multiple machines, securely.

Linux Fu: The Kitchen Sync | Hackaday
The ultimate dotfiles setup - YouTube chezmoi + ansible

gbck

jukben/gbck: 🗳 Intuitive lightweight tool for an easy and seamless backup of your files into Git repository
gbck— an easy way how to back up your dotfiles – Jakub Beneš – Medium

Stow

suppose do not support common folder in different environments

stow: manage farms of symbolic links | stow System Administration | Man Pages | ManKier

Use Stow for configuration management of multiple machines | Opensource.com
Brandon Invergo - Using GNU Stow to manage your dotfiles
Managing dotfiles using GNU Stow
Managing dotfiles with stow - Apiumhub
Managing dotfiles with GNU Stow | Bastian Venthur's Blog
Manage dotfiles with GNU Stow | Deepak Ramani

Manage your dotfiles across multiple machines with GNU Stow and Git - YouTube
Stow has forever changed the way I manage my dotfiles - YouTube
Give Your Dotfiles a Home with GNU Stow - YouTube
Manage Your Dotfiles Like A Superhero - YouTube

omerxx/dotfiles
eeowaa/stow-dotfiles

switching OS and host

SYS_OS=`uname -a`   # linux or mac
SHORT_HOSTNAME=`hostname -s`

# system aliases
if [[ "$SYS_OS" == 'Linux' ]]; then
     PATH="..LINUX PATH..."
     source ~/dotfiles/aliases.lx
 else
     PATH="... MAC PATH..."
     source ~/dotfiles/aliases.mac
fi

# run host specific profile
if [[ -e ~/dotfiles/profile.$SHORT_HOSTNAME ]]; then
    source ~/dotfiles/profile.$SHORT_HOSTNAME
fi

Environment variable

Environment variable - Wikiwand
Environment variables - ArchWiki

Frameworks

niieani/bash-oo-framework: Bash Infinity is a modern boilerplate for bash
jmmv/shtk: Application toolkit for programmers writing POSIX-compliant shell scripts

exec

The Uses of the Exec Command in Shell Script | Baeldung on Linux
bash - What does an "exec" command do? - Ask Ubuntu

Subprocess

How to Use Multi-Threaded Processing in Bash Scripts – CloudSavvy IT
Bash Process Termination Hacks – CloudSavvy IT

Common tasks

Useless Use of Cat Award

Follow The Art of Unix Programming, implement filters.
Simple filters in Perl, Ruby, and Bourne shell - TechRepublic

Capturing output of find . -print0 into a bash array - Stack Overflow
Filenames and Pathnames in Shell (bash, dash, ash, ksh, and so on): How to do it Correctly
Writing Safe Shell Scripts

Ten Things I Wish I’d Known About bash – zwischenzugs
Ten More Things I Wish I'd Known About bash – zwischenzugs
How to debug bash scripts with two tools - TechRepublic

shebang line

The #! magic, details about the shebang/hash-bang mechanism
Sambal : Passing options to node on the shebang (#!) line
Executing Python Scripts With a Shebang – Real Python

envns:

#!/bin/bash

ARGS=( $1 )  # separate $1 into multiple space-delimited arguments.
shift # consume $1

PROG=`which ${ARGS[0]}`
unset ARGS[0] # discard executable name

ARGS+=( "$@" ) # remainder of arguments preserved "as-is".
exec $PROG "${ARGS[@]}"
#!/usr/local/bin/envns node --harmonic

node-es6: (alias won't work in shebang)

#!/bin/bash
node --harmonic "$@"

Use -s to pass extra arguments

#!/usr/bin/env -S python3 -i

print("Hello, World!")
# you will be drop to a interactive shell

args parsing

Argbash: Bash argument parsing made easy

SHELLdorado - cmdargs
Command Line Options: How To Parse In Bash Using “getopt” — BahmanM.com

#!/bin/bash
# http://www.bahmanm.com/blogs/command-line-options-how-to-parse-in-bash-using-getopt

# "a" and "arga" have optional arguments with default values.
# "b" and "argb" have no arguments, acting as sort of a flag.
# "c" and "argc" have required arguments.

# e.g.:
# ./getopt.sh  1 -a'a' -b 2 -c'c' 3 4
# ./getopt.sh  1 --arga='a' 2 --argc='c' 3 4

# set an initial value for the flag
ARG_B=0

# read the options
TEMP=`getopt -o a::bc: --long arga::,argb,argc: -n 'getopt.sh' -- "$@"`
if [[ $? -ne 0 ]]; then
    echo -e "\e[1;31m""getopt error""\e[0m"
    exit 2
fi

eval set -- "$TEMP"

# extract options and their arguments into variables.
while true ; do
    case "$1" in
        -a|--arga)
            case "$2" in
                "") ARG_A='some default value' ; shift 2 ;;
                *) ARG_A=$2 ; shift 2 ;;
            esac ;;
        -b|--argb) ARG_B=1 ; shift ;;
        -c|--argc)
            case "$2" in
                "") shift 2 ;;
                *) ARG_C=$2 ; shift 2 ;;
            esac ;;
        --) shift ; break ;;
        *) echo "Internal error!" ; exit 2 ;;
    esac
done

# do something with the variables -- in this case the lamest possible one :-)
echo "ARG_A = $ARG_A"
echo "ARG_B = $ARG_B"
echo "ARG_C = $ARG_C"

# positional args
args=("$@")
echo \#: [${#args[@]}]
echo @: [${args[@]}]
for ((i=0; i<${#args[@]}; i++)); do
    echo ${args[i]};
done

Redirection

Redirections
Bash One-Liners Explained, Part III: All about redirections - good coders code, great coders reuse

All these redirections are equal:

$ > file command arg1 arg2 arg3
$ command > file arg1 arg2 arg3
$ command arg1 > file arg2 arg3
$ command arg1 arg2 > file arg3
$ command arg1 arg2 arg3 > file

Process Substitution

Process Substitution

Takes output of process as file

diff <(ls -a folder1) <(ls -a folder2)
diff <(grep -r -e foo src1/) <(grep -r -e foo src1/)

Command Substitution

Command Substitution

Takes output of process as text

vim `which a.sh`
vim $(which a.sh)

Here Doc

linux - How can I write a here doc to a file in Bash script? - Stack Overflow
Here Documents

This is a technique to embed string block in scripts and feed it to an interactive command.

interactive-program <<LimitString
command #1
command #2
...
LimitString

Single quote (') the commands to prevent string interpolation.

cat << 'EOF'
I want to print ${FOO} literally
EOF

<<- will suppress leading tabs (not spaces).

Create file with Here Doc

cat << EOF > ${FILE}
This is the file content.
Second line.
EOF

detect undefined variables

it's more meaningful in scripts to report error then to continue with empty string

set -u  # throw error on uninitialized variable, `set -o nounset`
set -e  # exit on statement error, `set -o errexit`

key bindings

Bash Shortcuts Quick Reference

bracketed paste mode

bracketed paste mode

Make pasting literals easier · Issue #967 · fish-shell/fish-shell

exit code

Exit and Exit Status
Exit Codes With Special Meanings

Command Exit Status — Brandur Leach

color

bash:tip_colors_and_formatting - FLOZz' MISC
True Colour (16 million colours) support in various terminal applications and terminals
256 Terminal colors and their 24bit equivalent (or similar)

A color escape sequence: <Esc>[<FormatCode>m, ``` can be:

zsh have color macros predefined, see themes in oh-my-zsh

scripts/ansi2html.sh at master · pixelb/scripts
ralphbean/ansi2html: Convert text with ansi color codes to HTML

eliukblau/pixterm: Draw images in your ANSI terminal with true color

power tools

mbrubeck/compleat

zstyle -L ":completion:*"

clvv/fasd

rupa/z

calculation

# do math calculation in $(( )), it is faster then `expr`
# and we can specify the number system with 8#, 10#, 16#
# useful for doing operation on variables with leading zeros
a=005
$((10#${a} + 1))
$(expr ${a} + 1)  # slower

or use bc, or use math (inspired by fish)

#!/bin/sh
# thin wrapper around bc for easier math expression in shell
# inspired by fish
echo "$@" | bc -l

trap

bash specific?

SignalTrap - Greg's Wiki
Bourne Shell Builtins (Bash Reference Manual)
How "Exit Traps" Can Make Your Bash Scripts Way More Robust And Reliable
Writing shell scripts - Lesson 15: Errors and Signals and Traps (Oh My!) - Part 1
Writing shell scripts - Lesson 16: Errors and Signals and Traps (Oh, My!) - Part 2

Linux Fu: It’s A Trap! | Hackaday
wd5gnr/trappist: A Bash function for managing signal traps easily

# clean up
trap cleanUp INT TERM EXIT
# cannot be killed
trap - INT TERM EXIT
NumberSIGMeaning
00On exit from shell
1SIGHUPClean tidyup
2SIGINTInterrupt (usually Ctrl-C)
3SIGQUITQuit (usually Ctrl-\)
6SIGABRTAbort
9SIGKILLDie Now (cannot be trap'ped)
14SIGALRMAlarm Clock
15SIGTERMTerminate

proper critical section

bash specific?

if [ ! -e $lockfile ]; then
    trap "rm -f $lockfile; exit" INT TERM EXIT
    touch $lockfile
    critical-section
    rm $lockfile
    trap - INT TERM EXIT
else
    echo "critical-section is already running"
fi

Shell Script Constructs

http://zsh.sourceforge.net/Doc/Release/Shell-Grammar.html

test command

The classic test command [Bash Hackers Wiki]

if

# if construct
if [[ -z "$1" || -z "$2" ]]; then
    echo "Usage: $0 <command> <arg>"
    exit 1
fi
# test existence of binary
if [[ -z "$(which wget)" ]]; then
  doSomething
fi

[[ $(which colordiff) ]] && alias diff='colordiff '

case

case '$param' in
A)
    ;;
B)
    ;;
*)
    ;;
esac
# choose 'open' command
case $(uname) in
    Darwin) OPEN_CMD=open ;;
    Linux) OPEN_CMD=xdg-open ;; # assuming X is present
    CYGWIN*) OPEN_CMD=cygstart ;;
    *) echo "Unknown OS"; exit 1 ;;
esac

for

for file in `ls -v` ; do cat $file >> concat.file ; done

for file in `ls -v` ; do
    cat $file >> concat.file ;
done

# {START..END..INCREMENT}
for i in {1..5}; do
   echo "Welcome $i times"
done

while

while [ -z $1 ] ; do
    echo $1
    shift
done

until

until [ -z $1 ] ; do
    echo $1
    shift
done

array

Bash associative array examples | Andy Balaam's Blog
Bash arrays | Andy Balaam's Blog
The Ultimate Bash Array Tutorial with 15 Examples

cherries+=(54758)
cherries+=(54761)
cherries+=(55075)
# array is 1-based
echo ${cherries[1]}  # 54758
# stringify array
echo ${cherries[@]}

# construct array form string
FILES=("${@}")
NUMFILES=${#FILES[@]}
for (( i = 0; i < ${NUMFILES}; i++ )); do
    echo ${FILES[${i}]}
done
for i in ${!FILES[@]}; do
    echo ${FILES[$i]}
done
# "${!array[*]}" "${array[*]}"?
# keys: ${!array[@]}
# values ${array[@]}

declare -A CAPITAL
CAPITAL[Alabama]=Montgomery
CAPITAL[Alaska]=Juneau
CAPITAL[Arizona]=Phoenix
CAPITAL[New Mexico]="Santa Fe"

# ${!ARRAY[@]} get the keys
for STATE in "${!CAPITAL[@]}"; do
    echo "The capital of $STATE is ${CAPITAL[$STATE]}."
done

passing array to function

bash how to pass array as an argument to a function - Stack Overflow
Passing arrays as parameters in bash - Stack Overflow

# stringify and reconstruct
foo()
{
    array=("$@")
    echo "array is ${array[@]}"
}

array=(one two three)
foo "${array[@]}"
# by reference
foo()
{
    # note: must use a name different from "array"
    local -n a=$1
    echo "array is ${a[@]}"
}

array=(one two three)
foo array
# by reference
foo()
{
    # note: must use a name different from "array1", "array2"
    declare -a argAry1=("${!1}")
    declare -a argAry2=("${!2}")
    declare -p argAry1
    declare -p argAry2
}

array1=(one two three)
array2=(a b c)
foo array1[@] array2[@]

returning values from function

Returning Values from Bash Functions | Linux Journal

Libraries

jwerle/q: Simple message queuing
jmcantrell/bashful: A collection of modules to simplify writing bash scripts.
dominictarr/JSON.sh: a pipeable JSON parser written in Bash
ShellCheck – Shell script analyzer

linux - Unit testing for shell scripts - Stack Overflow
kward/shunit2
Wiki Pages - shunit2 - shUnit2 - xUnit based unit testing for Unix shell scripts - Google Project Hosting
sstephenson/bats

Reference

Bash Shell Quick Reference, by Aurelio Jargas
The Linux Command Line by William E. Shotts, Jr.
Shell / Command Line | Linux.org
GNU/Linux Command-Line Tools Summary
Heiner's SHELLdorado
An Introduction to Shell Scripting | DigitalOcean
Portable Shell - Autoconf
Shell 特殊变量： $0, $#, $*, $@, $?, $$, $!, $_和命令行参数$n - Gikor - CSDN 博客

Writing Secure Shell Scripts | Linux Journal

Bash:
denysdovhan/bash-handbook: For those who wanna learn Bash
BASH Programming - Introduction HOW-TO
Bash Guide for Beginners
Advanced Bash-Scripting Guide
Bash Hackers Wiki Frontpage [Bash Hackers Wiki]
Bash Scripting Tutorial - Ryans Tutorials
http://justinlilly.com/bash/forgotten_friend_1.html
http://justinlilly.com/bash/forgotten_friend_2.html
http://justinlilly.com/bash/forgotten_friend_3.html
http://www.dsj.net/compedge/shellbasics.html
http://www.dsj.net/compedge/shellbasics1.html
http://www.dsj.net/compedge/regex.html
http://www.dsj.net/compedge/morebashfun2.html
http://www.dsj.net/compedge/bashit.html

Zsh:
http://zsh.sourceforge.net/Doc/Release/zsh_toc.html
http://grml.org/zsh/zsh-lovers.html
http://www.zzapper.co.uk/zshtips.html
http://fendrich.se/blog/2012/09/28/no/
http://www.tuxradar.com/content/z-shell-made-easy

Fish:
https://wiki.archlinux.org/title/Fish