Shell Tools

# print user info
id $USER

# new user with secondary group
useradd -G group[,group] user

# add secondary group to existing user
usermod -a -G group[,group] user

# delete user
userdel user
# delete user and home folder
userdel -r user

# relogin to take effect
su - $USER

# password expiry policy can be set with `passwd` and `chage`
passwd -n 0 user  # 0 day between password changes
passwd -x 90 user # days for password to be valid
passwd -w 7 user  # days of warning before expiration
chage -l user

# forces user to change at next login
passwd -l user    # lock user password

shell - Reload a Linux user's group assignments without logging out - Super User

Term recorder/sharing

shell-sharing

Pipes

All about pipes, by The Linux Information Project (LINFO)
Pipecut Project Home

akavel/up: Ultimate Plumber is a tool for writing Linux pipes with instant live preview
Linux Fu: Up -- A Tool for Interactive Linux Pipe - YouTube

Command palette

denisidoro/navi: An interactive cheatsheet tool for the command-line

pindexis/marker: The terminal command palette
Linux Fu: Marker Is A Command Line Menu | Hackaday

nvbn/thefuck: Magnificent app which corrects your previous console command.

openRTSP

http://www.live555.com/openRTSP/#option-summary

tar

# list content
tar -tf tarball.tar.gz | head

# skip 1 level (container folder)
tar -xvf ZendFramework-1.7.2.tar.gz --strip 1

# use multi-core compress tool
# https://stackoverflow.com/questions/12313242/utilizing-multi-core-for-targzip-bzip-compression-decompression
tar -I pbzip2 -cf OUTPUT_FILE.tar.bz2 paths_to_archive
tar --use-compress-program=pigz -cf OUTPUT_FILE.tar.gz paths_to_archive

Performing Incremental backups using tar | Unixmen
How to Create and Restore Incremental Backups with Tar on Linux - SnapShooter Documentation

tar --listed-incremental=tarfiles.list cvf images.tar images/ > tar.log
tar --listed-incremental=tarfiles.list cvf images.1.tar images/ > tar.log

grep

grep - Wikiwand
Grep
grep is a beautiful tool
Cover - GNU GREP and RIPGREP

Where GREP Came From - Computerphile - YouTube

# list file name only
grep -l pattern file

# grep | xarg combo
grep -lZ pattern file | xargs -0 ls -l

# find files without word "foo"
grep -L "foo" *
grep -c "foo" * | grep ":0" | awk -F: '{ print $1 }'

uniq

cat a b | sort | uniq > c   # c is a union b
cat a b | sort | uniq -d > c   # c is a intersect b
cat a b b | sort | uniq -u > c   # c is set difference a \ b

date

Doing Date Math on the Command Line, Part I | Linux Journal
Doing Date Math on the Command Line - Part II | Linux Journal

find

grep - Wikiwand
Find
Find is a beautiful tool
Mommy, I found it! — 15 Practical Linux Find Command Examples
Unix Find Tutorial
UsingFind - Greg's Wiki

Capturing output of find . -print0 into a bash array - Stack Overflow
linux - exclude directory from find . command - Stack Overflow
regex - How to use '-prune' option of 'find' in sh? - Stack Overflow
How to rsync files by date or by size (actually find trickery)

# find | xarg combo
find . -print0 | xargs -0 ls -l

# all files greater than 1mb
find $HOME/. -size +1024k

# all .js files inside current directory greater than 500k
find . -name '*.js' -size +500k

# find all files larger than zero but less than 500bytes
find . -size +0 -a -size -500c # (-a is AND, -c is bytes)

# find all files larger than zero OR (-o) any that haven't been accessed in over a year
find . -size 0 -o -atime +365

# find all files modified more than 5 days ago
find . -type f -mtime +5
# find all files modified within last 60 minutes
find . -type f -mmin -60

# find all files of size between 50-100MB
find . -size +50M -size -100M

# concat all Java files and print to console, note the space before '\'
find . -type f -name "*.java"-exec cat {} \;

# find executables
find . -perm /a=x

# find empty directories
find . -type d -empty

# excluding folders
# prune is faster than `-not` for files are skipped
find build -not \(-type d -path build/external -prune \) -name \*.js

operate files using inum

This is useful when the file name is particularly long or contains weird characters.

# list inum of files
ls -i

cd "$(find -inum 123456)"
mv "$(find -inum 123456)" ../some/where/

locate

Linux ‘locate’ command examples | alvinalexander.com
10 Useful 'locate' Command Practical Examples for Linux Newbies

fzf

junegunn/fzf: A command-line fuzzy finder
Examples · junegunn/fzf Wiki

Vim universe. fzf - command line fuzzy finder - YouTube
Why you should be using fzf, the command line fuzzy finder

Tab to select multiple items

Pipe with other tools to provide fuzzy filtering interface
Fussy match load-session · Issue #59 · jimeh/tmuxifier

peco

peco/peco: Simplistic interactive filtering tool

skim

lotabout/skim: Fuzzy Finder in rust!

directory jumping

ajeetdsouza/zoxide: A smarter cd command. Supports all major shells. reimplementation of z in Rust
rupa/z: z - jump around
b4b4r07/enhancd: A next-generation cd command with your interactive filter
wting/autojump: A cd command that learns - easily navigate directories from the command line

rsync

rsync - man page
又一枚瑞士军刀：rsync – Canvas
Replace Storage Drives with Rsync in Arch Linux | DominicMDominicM
How to use advanced rsync for large Linux backups | Opensource.com

rsync -avihXP --info=progress2 --stats <SRC> <DEST>

rsync -r -t -v --progress -s -e "ssh -p 1234" /mnt/media/coding user@backup:/mnt/media

# use rsync to copy files while retaining folder structure
# an alternative to `cp --parents`
rsync -avR --progress dir1/file1 dir2/file2 target/
# target/dir1/file1, target/dir2/file2 will be created

rsync 的核心算法 | 酷壳 - CoolShell.cn

rsnapshot

rsnapshot | rsnapshot
Guide to rsnapshot and incremental backups on Linux
Easy Automated Snapshot-Style Backups with Rsync inspiration for rsnapshot

rename

# to lower (y = transliterating, character substitution)
rename 'y/A-Z/a-z/' *
# remove '.txt' suffix
rename 's/.txt$//' *
# add '.txt' suffix
rename 's/$/.txt/' *

laurent22/massren: massren - easily rename multiple files using your text editor
whyboris/Simplest-File-Renamer: Simplest file renamer - rename your files quickly and easily

ismaelgv/rnr: A command-line tool to batch rename files and directories

75lb/renamer: Rename files in bulk. custom function
jhotmann/node-rename-cli: A tool for renaming files quickly, especially multiple files at once. Nunjucks template

nomino — command-line utility in Rust // Lib.rs

wernight/renameit: Rename-It! is a Windows software to safely rename of thousands of files and folders at once via regex and all kind of other renaming filters. See the wiki section for more details.

sort

sort (Unix) - Wikiwand
sort - man page

xargs

xargs - Wikiwand
xargs - man page
Xargs - Charles Martin Reid
Things you (probably) didn't know about xargs
8 Practical Examples of Linux Xargs Command for Beginners
find -exec vs find | xargs

-n1 vs -L1

# simple use case
echo "U R L" | xargs -n1 echo

# `-I` may be needed for more complex use case (`mv @ @.bak`)
# however `-I` implies `-L` (takes input as line)
# so split before call
echo -e "U R L\na b c" | sed 's/ /\n/g' | xargs -I@ echo @

# or do with temp file
echo "U\nR\nL" > file
cat file | xargs -I@ echo @

# custom delimiter, custom replace-str
echo -n 4500 10500 30000 | xargs -n1 -d" " -I@ math @*200/1024

diff

# left only
diff --changed-group-format='%<' --unchanged-group-format='' FILE1 FILE2
# right only
diff --changed-group-format='%>' --unchanged-group-format='' FILE1 FILE2

https://superuser.com/questions/125376/how-do-i-compare-binary-files-in-linux

vimdiff <(xxd file1) <(xxd file2)
code -diff file1 file2

so-fancy/diff-so-fancy: Good-lookin' diffs. Actually… nah… The best-lookin' diffs.
diffsitter — command-line utility in Rust // Lib.rs
Wilfred/difftastic: a diff that understands syntax 🟥🟩

madsen/vbindiff: Visual Binary Diff (VBinDiff) displays files in hex & ASCII and can highlight the differences between 2 files

Parallel

GNU Parallel Tutorial — GNU Parallel 20230822 documentation
GNU Parallel videos - YouTube

GNU Parallel, where have you been all my life? | Alex Plescan
This CLI Tool is AMAZING | Prime Reacts - YouTube

Text Searching

The Platinum Searcher: search source code, faster than ack - Progville
Releases · monochromegane/the_platinum_searcher

ripgrep is faster than {grep, ag, git grep, ucg, pt, sift} - Andrew Gallant's Blog
Releases · BurntSushi/ripgrep

Tutorial – ugrep » Linux Magazine

Split files

csplit invocation (GNU Coreutils 9.0)
csplit - Split text files - IBM Documentation

split invocation (GNU Coreutils 9.0)
split - Split a file into manageable pieces - IBM Documentation

csplit -f"bufferedsource." -b"%02d.log" bufferedsource.log "/.*Clicked NextStep/" "{*}"

split -l 1000000 --additional-suffix=".log" -d bufferedsource.log "bufferedsource."

bash - Using regex to tell csplit where to split the file - Stack Overflow

Trim lines in place

Remove the last line from a file in Bash - Stack Overflow

filename="bufferedsource.log"
lines=2382577

file_size="$(stat --format=%s "$filename")"
trim_count="$(tail -n"$lines" "$filename" | wc -c)"
end_position="$(echo "$file_size - $trim_count" | bc)"

dd if=/dev/null of="$filename" bs=1 seek="$end_position"

File watching

rvoicilas/inotify-tools: inotify-tools is a C library and a set of command-line programs for Linux providing a simple interface to inotify.
Linux Fu: Watch That Filesystem | Hackaday

entr(1)

cpio

cpio - man page
Linux cpio Examples: How to Create and Extract cpio Archives (and tar archives)
cpio - Wikiwand

copy files to destination folder keeping the tree structure of the path specified

# in ~/source
# these files are reachable in ~/source with their relative paths
cat ~/filelist
frameworks/av/media/libstagefright/foundation/ANetworkSession.cpp
frameworks/av/media/libstagefright/wifi-display/Android.mk
frameworks/av/media/libstagefright/wifi-display/MediaSender.cpp
frameworks/av/media/libstagefright/wifi-display/source/Converter.cpp
frameworks/av/media/libstagefright/wifi-display/source/PlaybackSession.cpp
frameworks/av/media/libstagefright/wifi-display/source/TSPacketizer.cpp
frameworks/av/media/libstagefright/wifi-display/source/WifiDisplaySource.cpp
frameworks/av/media/libstagefright/wifi-display/source/WifiDisplaySource.h
frameworks/av/media/libstagefright/wifi-display/VideoFormats.cpp
frameworks/av/media/libstagefright/wifi-display/wfd.cpp

cpio -pud ~/dest < ~/filelist

# cpio single file
echo "frameworks/base/cmds/pm/src/com/android/commands/pm/Pm.java" \
| cpio -pud ${ANDROID_PATCH}/board/common/

cut

# get n-th to m-th word in string, using ':' as separator
echo ${LINE} | cat -d: -f n,m

tr

replace character in SET1 with characters in SET2

echo "foo\nbar\nbaz" | tr "bf" "\!"
# negates SET1
echo "foo\nbar\nbaz" | tr -c "bf\n" "\!"
# ROT13
alias rot13="tr '[A-Za-z]' '[N-ZA-Mn-za-m]'"

lsof

akme/lsofgraph-python: python version of lsof to graphviz parser
zevv/lsofgraph: lsof to graphviz
10 lsof Command Examples in Linux

funny

20 amusing Linux commands to have fun with the terminal
cowsay - Wikiwand

cowsay -l
fortune | cowsay -f stegosaurus

Yaksay — command-line utility in Rust // Lib.rs
dustinkirkland/hollywood multiple panels of noisy apps
genact — command-line utility in Rust // Lib.rs
bartobri/no-more-secrets: A command line tool that recreates the famous data decryption effect seen in the 1992 movie Sneakers.
rusty-rain — command-line utility in Rust // Lib.rs

JSON manipulation

List of JSON tools for command line – Ilya's blog

json(1) - JSON love for your command line
maxogden/jsonmap: CLI JSON mapping/transformation utility
FGRibreau/jq.node: jq.node - like jq but WAY MORE powerful

JMESPath — JMESPath libray for multiple languages

jzelinskie/faq: Format Agnostic jQ

antonmedv/fx: Terminal JSON viewer viewer and run JavaScript code on JSON
FX: Powerful Terminal JSON Visualizer And Processor - YouTube

jless - A Command-Line JSON Viewer node folding
PaulJuliusMartinez/jless: jless is a command-line JSON viewer designed for reading, exploring, and searching through JSON data.

simeji/jid: json incremental digger interactive filter with autocomplete

jpmens/jo: JSON output from a shell

lloyd/JSONSelect: CSS-like selectors for JSON 😴inactive
dtao/gquery: Generic jQuery 😴inactive

jpmens/jo: JSON output from a shell
Jan-Piet Mens :: A shell command to create JSON: jo

Python

jmespath/jp: Command line interface to JMESPath - http://jmespath.org

mahmoud/glom: ☄️ Python's nested data operator (and CLI), for all your declarative restructuring needs. Got data? Glom it! ☄️

asq — asq 1.3a documentation

Welcome to Flupy — flupy 1.0.2 documentation

jq

jq
jq Manual
jqterm: jq as a service
jq play
object { 5 props } — jqTerm

jiq - JSON Incremental jq-filterer
fiatjaf/jiq: jid on jq - interactive JSON query tool using jq expressions
noahgorstein/jqp: A TUI playground to experiment with jq

joelpurra/jqnpm: A package manager built for the command-line JSON processor jq. 😴inactive

jq/builtin.jq at master · stedolan/jq
FAQ · stedolan/jq Wiki
Advanced Topics · stedolan/jq Wiki
Cookbook · stedolan/jq Wiki
For JSONPath users · stedolan/jq Wiki
Docs for Oniguruma Regular Expressions (RE.txt) · stedolan/jq Wiki
How to: Avoid Pitfalls · stedolan/jq Wiki
jq Cheet Sheet
jq recipes
rjz/jq-tutorial: Interactive exercises for learning jq

JSON Tools: Jq - Hyperpolyglot
jq Primer: Munging JSON Data - Andrew Gibiansky
JQ Select Explained: Selecting elements from JSON with Examples - Earthly Blog
jq is sed for JSON
Reshaping JSON with jq | Programming Historian
Parsing JSON with jq
Wrestling JSON with jq by Arjan van der Gaag

# pick field from array
cat JSON | jq  '[ .[] | { a: .field1, } ]'
cat JSON | jq  'map({ a: .field1, })'

# pick field from files, shorthand for `slideId`
find . -maxdepth 2 -name *json | xargs -I@ jq -c '{slideId, w: .width, h: .height}' @
# collect NDJSON as array and convert to CSV
find . -maxdepth 2 -name *json | xargs -I@ jq -c '{slideId, w: .width, h: .height}' @ | jq -s -f ~/tocsv.jq

# match field
cat JSON | jq -c 'select(.row === 33089)'
cat JSON | jq -c 'select(.list | contains(["foo", "bar"]) | select(.access | test("(deny|disallow)") | not)'

tocsv.jq:

def tocsv($x):
    $x
    |(map(keys)
        |add
        |unique
        |sort
    ) as $cols
    |map(. as $row
        |$cols
        |map($row[.]|tostring)
    ) as $rows
    |$cols,$rows[]
    | @csv;

tocsv(.)