Shell notes

Notes on shell usage.
Most of the following are compatible to bash and zsh.

We all <3 Terminals. - Terminals Are Sexy
Philip James, Asheesh Laroia - Type python, press enter. What happens? - PyCon 2015 - YouTube how shell invocation works
Command Line Power User — A free video series for web developers on learning a modern command line workflow with ZSH, Z and related tools. !important, by Wes Bos
Coding like a Hacker in the terminal – Caleb Taylor – Medium
You’re Missing Out on a Better Mac Terminal Experience
The birth of the Bash shell | Opensource.com
Conquering the Command Line | Softcover.io
Bash Reference Manual
OverTheWire: Wargames learn shell commands through games

Bash Automation & Scripting Basics (Part 1) – CloudSavvy IT
Bash Automation and Scripting Basics (Part 2) – CloudSavvy IT
Bash Automation and Scripting Basics (Part 3) – CloudSavvy IT

Choosing shell

Unix Shells: Bash, Fish, Ksh, Tcsh, Zsh - Hyperpolyglot
Evolution of shells in Linux
Rich’s sh (POSIX shell) tricks
the xonsh shell — xonsh 0.8.12 documentation

Most distro comes with bash as default login shell.
Since bash is ubiquitous and more feature-rich than POSIX shell (/bin/sh), it is recommended to write shell scripts in bash.
Since zsh is bash (and hence POSIX) -compatible. Most command lines found on the web can be pasted in zsh directly (unlike fish which is not POSIX-compatible). It is recommended to use zsh as the interactive shell.
However I found fish design quite sensible. I'll try to use external binaries as mush as possible instead of relying on shell features. On the same line of thought, even-though zsh has more powerful syntax than bash, it may not be worthy to use them to avoid lock-in and maintain portability.

Evolution of shells in Linux

Starship Successor of Fish shell's Spacefish
starship/starship: ☄🌌️ The cross-shell prompt for astronauts.
Configuration | Starship

igor-petruk/scriptisto: A language-agnostic "shebang interpreter" that enables you to write scripts in compiled languages.
rust-script — command-line utility in Rust // Lib.rs

rash-sh/rash: Declarative shell scripting container oriented

You can set a shell as your default shell with chsh.
But interactive shells must be added to /etc/shells first before used as a login shell.

http://fishshell.com/docs/2.1/faq.html#faq-default

dotfiles

most contain useful aliases, functions and scripts

It is recommended to use source control to store your config files (dotfiles, as they usually begins with a .) and use symlinks (ln -s) to deploy them to the machine.

bash - Difference between .bashrc and .bash_profile - Super User
talks/linuxcon_eu-2013-10-gitify_your_life.pdf at main · RichiH/talks

GitHub does dotfiles - dotfiles.github.io
webpro/awesome-dotfiles: A curated list of dotfiles resources.

Honorable mentions:

http://chneukirchen.org/dotfiles/
https://github.com/jukben/dotfiles (Tmux)

TODO: add my links in /caravan/github-watch/dotfiles
SukkaW/dotfiles: 🔧 My development environment and config install script to bootstrap

Steve's Blog: bash aliases

jukben/gbck: 🗳 Intuitive lightweight tool for an easy and seamless backup of your files into Git repository
gbck— an easy way how to back up your dotfiles – Jakub Beneš – Medium
Yet Another Dotfiles Manager - yadm
direnv – unclutter your .profile | direnv

RichiH/vcsh: config manager based on Git

AGWA/git-crypt: Transparent file encryption in git
.gitattributes · dots · GammaFn / dotfiles.cli · GitLab

lra/mackup: Keep your application settings in sync (OS X/Linux)
tuning — command-line utility in Rust // Lib.rs
hm — command-line utility in Rust // Lib.rs
quickcfg — command-line utility in Rust // Lib.rs

How to make your Dotfile management a painless affair
ajmalsiddiqui/autodot: A dotfile management system that makes sharing your dotfiles easy while keeping you in the loop.
RCM(7)

Home | chezmoi.io
twpayne/chezmoi: Manage your dotfiles across multiple machines, securely.
Linux Fu: The Kitchen Sync | Hackaday

svetlyak40wt/dotfiler: Shell agnostic git based dotfiles package manager, written in Python.
Using Dropbox as a Settings Repository

Stow

stow: manage farms of symbolic links | stow System Administration | Man Pages | ManKier

Use Stow for configuration management of multiple machines | Opensource.com
Brandon Invergo - Using GNU Stow to manage your dotfiles
Managing dotfiles using GNU Stow
Managing dotfiles with stow - Apiumhub
Managing dotfiles with GNU Stow | Bastian Venthur's Blog
Manage dotfiles with GNU Stow | Deepak Ramani

GitHub - omerxx/dotfiles
GitHub - eeowaa/stow-dotfiles

switching OS and host

SYS_OS=`uname -a`   # linux or mac
SHORT_HOSTNAME=`hostname -s`

# system aliases
if [[ "$SYS_OS" == 'Linux' ]]; then
     PATH="..LINUX PATH..."
     source ~/dotfiles/aliases.lx
 else
     PATH="... MAC PATH..."
     source ~/dotfiles/aliases.mac
fi

# run host specific profile
if [[ -e ~/dotfiles/profile.$SHORT_HOSTNAME ]]; then
    source ~/dotfiles/profile.$SHORT_HOSTNAME
fi

Environment variable

Environment variable - Wikiwand
Environment variables - ArchWiki

Frameworks

niieani/bash-oo-framework: Bash Infinity is a modern boilerplate for bash
jmmv/shtk: Application toolkit for programmers writing POSIX-compliant shell scripts

`exec`

The Uses of the Exec Command in Shell Script | Baeldung on Linux
bash - What does an "exec" command do? - Ask Ubuntu

Subprocess

How to Use Multi-Threaded Processing in Bash Scripts – CloudSavvy IT
Bash Process Termination Hacks – CloudSavvy IT

Common tasks

Useless Use of Cat Award

Follow The Art of Unix Programming, implement filters.
Simple filters in Perl, Ruby, and Bourne shell - TechRepublic

Capturing output of find . -print0 into a bash array - Stack Overflow
Filenames and Pathnames in Shell (bash, dash, ash, ksh, and so on): How to do it Correctly
Writing Safe Shell Scripts

Ten Things I Wish I’d Known About bash – zwischenzugs
Ten More Things I Wish I'd Known About bash – zwischenzugs
How to debug bash scripts with two tools - TechRepublic

shebang line

The #! magic, details about the shebang/hash-bang mechanism
Sambal : Passing options to node on the shebang (#!) line
Executing Python Scripts With a Shebang – Real Python

envns:

#!/bin/bash

ARGS=( $1 )  # separate $1 into multiple space-delimited arguments.
shift # consume $1

PROG=`which ${ARGS[0]}`
unset ARGS[0] # discard executable name

ARGS+=( "$@" ) # remainder of arguments preserved "as-is".
exec $PROG "${ARGS[@]}"

#!/usr/local/bin/envns node --harmonic

node-es6: (alias won't work in shebang)

#!/bin/bash
node --harmonic "$@"

Use -s to pass extra arguments

#!/usr/bin/env -S python3 -i

print("Hello, World!")
# you will be drop to a interactive shell

args parsing

Argbash: Bash argument parsing made easy

SHELLdorado - cmdargs
Command Line Options: How To Parse In Bash Using “getopt” — BahmanM.com

#!/bin/bash
# http://www.bahmanm.com/blogs/command-line-options-how-to-parse-in-bash-using-getopt

# "a" and "arga" have optional arguments with default values.
# "b" and "argb" have no arguments, acting as sort of a flag.
# "c" and "argc" have required arguments.

# e.g.:
# ./getopt.sh  1 -a'a' -b 2 -c'c' 3 4
# ./getopt.sh  1 --arga='a' 2 --argc='c' 3 4

# set an initial value for the flag
ARG_B=0

# read the options
TEMP=`getopt -o a::bc: --long arga::,argb,argc: -n 'getopt.sh' -- "$@"`
if [[ $? -ne 0 ]]; then
    echo -e "\e[1;31m""getopt error""\e[0m"
    exit 2
fi

eval set -- "$TEMP"

# extract options and their arguments into variables.
while true ; do
    case "$1" in
        -a|--arga)
            case "$2" in
                "") ARG_A='some default value' ; shift 2 ;;
                *) ARG_A=$2 ; shift 2 ;;
            esac ;;
        -b|--argb) ARG_B=1 ; shift ;;
        -c|--argc)
            case "$2" in
                "") shift 2 ;;
                *) ARG_C=$2 ; shift 2 ;;
            esac ;;
        --) shift ; break ;;
        *) echo "Internal error!" ; exit 2 ;;
    esac
done

# do something with the variables -- in this case the lamest possible one :-)
echo "ARG_A = $ARG_A"
echo "ARG_B = $ARG_B"
echo "ARG_C = $ARG_C"

# positional args
args=("$@")
echo \#: [${#args[@]}]
echo @: [${args[@]}]
for ((i=0; i<${#args[@]}; i++)); do
    echo ${args[i]};
done

Redirection

Redirections
Bash One-Liners Explained, Part III: All about redirections - good coders code, great coders reuse

All these redirections are equal:

$ > file command arg1 arg2 arg3
$ command > file arg1 arg2 arg3
$ command arg1 > file arg2 arg3
$ command arg1 arg2 > file arg3
$ command arg1 arg2 arg3 > file

Process Substitution

Takes output of process as file

diff <(ls -a folder1) <(ls -a folder2)
diff <(grep -r -e foo src1/) <(grep -r -e foo src1/)

Command Substitution

Takes output of process as text

vim `which a.sh`
vim $(which a.sh)

Here Doc

linux - How can I write a here doc to a file in Bash script? - Stack Overflow
Here Documents

This is a technique to embed string block in scripts and feed it to an interactive command.

interactive-program <<LimitString
command #1
command #2
...
LimitString

Single quote (') the commands to prevent string interpolation.

cat << 'EOF'
I want to print ${FOO} literally
EOF

<<- will suppress leading tabs (not spaces).

Create file with Here Doc

cat << EOF > ${FILE}
This is the file content.
Second line.
EOF

detect undefined variables

it's more meaningful in scripts to report error then to continue with empty string

set -u  # throw error on uninitialized variable, `set -o nounset`
set -e  # exit on statement error, `set -o errexit`

A color escape sequence: <Esc>[<FormatCode>m, ``` can be:

\e
\033
\x1B

zsh have color macros predefined, see themes in oh-my-zsh

scripts/ansi2html.sh at master · pixelb/scripts
ralphbean/ansi2html: Convert text with ansi color codes to HTML

eliukblau/pixterm: Draw images in your ANSI terminal with true color

power tools

mbrubeck/compleat

zstyle -L ":completion:*"

clvv/fasd

rupa/z

calculation

# do math calculation in $(( )), it is faster then `expr`
# and we can specify the number system with 8#, 10#, 16#
# useful for doing operation on variables with leading zeros
a=005
$((10#${a} + 1))
$(expr ${a} + 1)  # slower

or use bc, or use math (inspired by fish)

#!/bin/sh
# thin wrapper around bc for easier math expression in shell
# inspired by fish
echo "$@" | bc -l

trap

bash specific?

SignalTrap - Greg's Wiki
Bourne Shell Builtins (Bash Reference Manual)
How "Exit Traps" Can Make Your Bash Scripts Way More Robust And Reliable
Writing shell scripts - Lesson 15: Errors and Signals and Traps (Oh My!) - Part 1
Writing shell scripts - Lesson 16: Errors and Signals and Traps (Oh, My!) - Part 2

Linux Fu: It’s A Trap! | Hackaday
wd5gnr/trappist: A Bash function for managing signal traps easily

# clean up
trap cleanUp INT TERM EXIT
# cannot be killed
trap - INT TERM EXIT

Number	SIG	Meaning
0	0	On exit from shell
1	SIGHUP	Clean tidyup
2	SIGINT	Interrupt (usually Ctrl-C)
3	SIGQUIT	Quit (usually Ctrl-\)
6	SIGABRT	Abort
9	SIGKILL	Die Now (cannot be trap'ped)
14	SIGALRM	Alarm Clock
15	SIGTERM	Terminate

proper critical section

bash specific?

if [ ! -e $lockfile ]; then
    trap "rm -f $lockfile; exit" INT TERM EXIT
    touch $lockfile
    critical-section
    rm $lockfile
    trap - INT TERM EXIT
else
    echo "critical-section is already running"
fi

Shell Script Constructs

http://zsh.sourceforge.net/Doc/Release/Shell-Grammar.html

`test` command

The classic test command [Bash Hackers Wiki]

`if`

# if construct
if [[ -z "$1" || -z "$2" ]]; then
    echo "Usage: $0 <command> <arg>"
    exit 1
fi

# test existence of binary
if [[ -z "$(which wget)" ]]; then
  doSomething
fi

[[ $(which colordiff) ]] && alias diff='colordiff '

`case`

case '$param' in
A)
    ;;
B)
    ;;
*)
    ;;
esac

# choose 'open' command
case $(uname) in
    Darwin) OPEN_CMD=open ;;
    Linux) OPEN_CMD=xdg-open ;; # assuming X is present
    CYGWIN*) OPEN_CMD=cygstart ;;
    *) echo "Unknown OS"; exit 1 ;;
esac

`for`

for file in `ls -v` ; do cat $file >> concat.file ; done

for file in `ls -v` ; do
    cat $file >> concat.file ;
done

# {START..END..INCREMENT}
for i in {1..5}; do
   echo "Welcome $i times"
done

`while`

while [ -z $1 ] ; do
    echo $1
    shift
done

`until`

until [ -z $1 ] ; do
    echo $1
    shift
done

array

Bash associative array examples | Andy Balaam's Blog
Bash arrays | Andy Balaam's Blog
The Ultimate Bash Array Tutorial with 15 Examples

cherries+=(54758)
cherries+=(54761)
cherries+=(55075)
# array is 1-based
echo ${cherries[1]}  # 54758
# stringify array
echo ${cherries[@]}

# construct array form string
FILES=("${@}")
NUMFILES=${#FILES[@]}
for (( i = 0; i < ${NUMFILES}; i++ )); do
    echo ${FILES[${i}]}
done
for i in ${!FILES[@]}; do
    echo ${FILES[$i]}
done
# "${!array[*]}" "${array[*]}"?
# keys: ${!array[@]}
# values ${array[@]}

declare -A CAPITAL
CAPITAL[Alabama]=Montgomery
CAPITAL[Alaska]=Juneau
CAPITAL[Arizona]=Phoenix
CAPITAL[New Mexico]="Santa Fe"

# ${!ARRAY[@]} get the keys
for STATE in "${!CAPITAL[@]}"; do
    echo "The capital of $STATE is ${CAPITAL[$STATE]}."
done

passing array to function

bash how to pass array as an argument to a function - Stack Overflow
Passing arrays as parameters in bash - Stack Overflow

# stringify and reconstruct
foo()
{
    array=("$@")
    echo "array is ${array[@]}"
}

array=(one two three)
foo "${array[@]}"

# by reference
foo()
{
    # note: must use a name different from "array"
    local -n a=$1
    echo "array is ${a[@]}"
}

array=(one two three)
foo array

# by reference
foo()
{
    # note: must use a name different from "array1", "array2"
    declare -a argAry1=("${!1}")
    declare -a argAry2=("${!2}")
    declare -p argAry1
    declare -p argAry2
}

array1=(one two three)
array2=(a b c)
foo array1[@] array2[@]

returning values from function

Returning Values from Bash Functions | Linux Journal

Libraries

jwerle/q: Simple message queuing
jmcantrell/bashful: A collection of modules to simplify writing bash scripts.
dominictarr/JSON.sh: a pipeable JSON parser written in Bash
ShellCheck – Shell script analyzer

linux - Unit testing for shell scripts - Stack Overflow
kward/shunit2
Wiki Pages - shunit2 - shUnit2 - xUnit based unit testing for Unix shell scripts - Google Project Hosting
sstephenson/bats

Reference

Bash Shell Quick Reference, by Aurelio Jargas
The Linux Command Line by William E. Shotts, Jr.
Shell / Command Line | Linux.org
GNU/Linux Command-Line Tools Summary
Heiner's SHELLdorado
An Introduction to Shell Scripting | DigitalOcean
Portable Shell - Autoconf
Shell 特殊变量： $0, $#, $*, $@, $?, $$, $!, $_和命令行参数$n - Gikor - CSDN 博客

Writing Secure Shell Scripts | Linux Journal

Bash:
denysdovhan/bash-handbook: For those who wanna learn Bash
BASH Programming - Introduction HOW-TO
Bash Guide for Beginners
Advanced Bash-Scripting Guide
Bash Hackers Wiki Frontpage [Bash Hackers Wiki]
Bash Scripting Tutorial - Ryans Tutorials
http://justinlilly.com/bash/forgotten_friend_1.html
http://justinlilly.com/bash/forgotten_friend_2.html
http://justinlilly.com/bash/forgotten_friend_3.html
http://www.dsj.net/compedge/shellbasics.html
http://www.dsj.net/compedge/shellbasics1.html
http://www.dsj.net/compedge/regex.html
http://www.dsj.net/compedge/morebashfun2.html
http://www.dsj.net/compedge/bashit.html

Zsh:
http://zsh.sourceforge.net/Doc/Release/zsh_toc.html
http://grml.org/zsh/zsh-lovers.html
http://www.zzapper.co.uk/zshtips.html
http://fendrich.se/blog/2012/09/28/no/
http://www.tuxradar.com/content/z-shell-made-easy

Fish:
https://wiki.archlinux.org/title/Fish