awk notes

AWK is an interpreted programming language designed for text processing and typically used as a data extraction and reporting tool.

The GNU Awk User’s Guide
Cover - GNU AWK
How To Use awk In Bash Scripting - nixCraft
The many faces of awk | Network World
Doing math with awk | Network World
How to Use the awk Command on Linux
EVERYONE Needs to Learn a Little Bit of AWK! - YouTube
Awk Master Class - YouTube

Why Every Linux User Needs To Learn Awk - YouTube
Awk Isn't Just A Command: Let's Learn The Basics - YouTube
Every Linux User Should Know Regex and Awk - YouTube

awk '{PROGRAM}' file
awk -f <PROGRAM.awk> file

# passing variable to AWK
n1=5; n2=10;
echo | awk -v x=$n1 -v y=$n2 -f program.awk

BEGIN {...}
CONDITION {action}
CONDITION {action}
END {...}

A Awk program is executed on a row basis, with optional rules to switch the statements (block) to execute
each row is parsed into fields by the separator.
The default action is { print $0 } (print the whole row).
We can add condition before statement to serve as filter, condition can use field, regular expression and comparisons.

Awk program:

#!/usr/bin/awk -f

# This line is a comment

BEGIN {
    printf "%s\n","User accounts:"
    print "=============="
    FS=":"
    n=0
}

# Now we'll run through the data
{
    if ($3 >= 1000) {
        print $1
        n ++
    }
}

END {
    print "=============="
    print n " accounts"
}

NR      Number of Row
NF      Number of Fields
$()     field substitution

filtering

# print first field of first row
awk 'NR==1 { print $1 }'

# print rows with third field >= 1000
awk -F":" 'BEGIN {print "user accounts:"} $3 >= 1000 ' /etc/passwd

# print rows starts with systemd (regular expression)
awk '/^systemd/' /etc/group

# print rows not ending with ":" (having member) (regular expression)
awk '! /:$/' /etc/group

field separator

$N is the N-th field, $NF is the the last field

# get second field with ":" as separator
awk 'BEGIN { FS = ":" } ; { print $2 }'
awk -F: '{ print $2 }'

# comma separate the output fields
# otherwise they will be concatenated to a single string
date -R | awk '{OFS="-"; print $4 $3 $2}'   # 2019Dec05, OFS ignored
# specify output field separator, `$4-$3-$2` does a mathematical subtraction
date -R | awk '{OFS="-"; print $4,$3,$2}'

print number of lines

awk 'BEGIN {sum=0} {sum=sum+1} END {print sum}' filename
awk 'END{print NR}' filename

print line number and second last field

awk '{ print "Line " NR ": " $(NF-1) }' test.txt

print first 16K lines

awk -F '' 'NF < 16384' infile >outfile
awk 'length($0) < 16384' infile >outfile
# which is equivalent to
awk '{ if (length($0) < 16384) print }' infile >outfile

Reference

The GNU Awk User’s Guide
AWK Programming
awk is a beautiful tool
awk 使用手册 - fanqiang.com
AWK 简明教程 | 酷 壳 - CoolShell.cn
AWK Tutorial

Awk - A useful little language

Famous Awk One-Liners Explained, Part I: File Spacing, Numbering and Calculations - good coders code, great reuse
Famous Awk One-Liners Explained, Part II: Text Conversion and Substitution - good coders code, great reuse
Famous Awk One-Liners Explained, Part III: Selective Printing and Deleting of Certain Lines - good coders code, great reuse
Update on Famous Awk One-Liners Explained: String and Array Creation - good coders code, great reuse
http://www.pement.org/awk/awk1line.txt

AWK GTF! How to Analyze a Transcriptome Like a Pro - Part 1 - Blog - Reason I Am Here - Nacho Caballero
AWK GTF! How to Analyze a Transcriptome Like a Pro - Part 2 - Blog - Reason I Am Here - Nacho Caballero
AWK GTF! How to Analyze a Transcriptome Like a Pro - Part 3 - Blog - Reason I Am Here - Nacho Caballero

Common threads: Awk by example, Part 1
Common threads: Awk by example, Part 2

Awk by Example, Part 1 : An intro to the great language with the strange name
Awk by Example, Part 2 : Records, loops, and arrays
Awk by Example, Part 3 : String functions and ... checkbooks?

Awk, Nawk and GNU Awk Cheat Sheet - good coders code, great reuse

Awk - A Tutorial and Introduction - by Bruce Barnett
Awk Quick Reference - Bruce Barnett
Top Examples of Awk Command in Unix
8 Powerful Awk Built-in Variables – FS, OFS, RS, ORS, NR, NF, FILENAME, FNR

sed & awk, 2nd Edition