S003 · The Command Line — bash, pipes, grep, jq
Live in the terminal without fear. The pipes, filters, and text-crunching muscle memory that separates an engineer from a button-clicker — and the 20 commands you'll use every single day for the rest of your career.
🎯 Chain shell commands with pipes, filter text with grep/awk/sed, and slice JSON with jq — well enough to answer real questions from real log files in under 60 seconds.
Why this session exists
The terminal is where every backend engineer, SRE, data engineer, and ML platform person lives. A junior dev double-clicks through a file manager to find a log; a senior dev pipes it through grep | awk | sort | uniq -c | sort -rn | head and has an answer in eight seconds. That gap is not talent. It's a handful of commands, learned once, wired into muscle memory. This session installs that muscle memory.
- Navigate the filesystem, find files, and inspect processes without touching a GUI.
- Chain commands with pipes to answer questions like ‘which endpoint had the most 500s in the last hour?’ from a raw log file.
- Use `grep`, `sed`, and `awk` well enough that you stop reaching for Python for one-off text munging.
- Slice, filter, and reshape JSON with `jq` — the tool every API and Kubernetes user needs.
- Read a stranger's bash script and predict what it does before running it.
Prerequisites
- S001 — Dev Environment — a working Linux/WSL/macOS shell.
- S002 — Git & GitHub — you'll practise these commands inside a Git repo.
(a) Intuition · 5 min
Imagine a factory floor. Every station does one job — cut, weld, paint, box. Parts flow along a conveyor. To make a new product you rearrange the stations, you don't build a new factory.
The Unix shell is that factory. Each command is a station. The pipe (|) is the conveyor. To answer a new question you snap together commands you already know — you don't write a new program.
Technically: every Unix command reads a stream of bytes from stdin and writes a stream of bytes to stdout. A pipe wires one command's stdout to the next command's stdin — in memory, streaming, zero disk I/O.
That is the entire Unix philosophy: do one thing well, and communicate via text streams. Fifty years later, no other OS design has aged as gracefully.
The three streams that everything runs on
- stdin (fd 0) — where input comes from. Default: your keyboard. Redirect with `<` or a pipe.
- stdout (fd 1) — where normal output goes. Default: your terminal. Redirect with `>` or a pipe.
- stderr (fd 2) — where errors go. Default: your terminal too, but a SEPARATE stream. Redirect with `2>`.
A quick history so you know why the world looks like this
- 1971First Unix shell (Thompson shell)Ken Thompson writes `sh` for the PDP-11 — the ancestor of every shell you'll ever use.
- 1979Bourne shell (sh)Stephen Bourne's shell becomes the Unix standard. Its syntax survives untouched in every POSIX shell today.
- 1989Bash 1.0GNU's ‘Bourne Again SHell’ — a free, feature-richer sh. Ships with Linux. Becomes the default everywhere except macOS.
- 2007jq releasedStephen Dolan writes a ‘sed for JSON’. Becomes essential the moment REST APIs eat the world.
- 2019macOS switches default to zshApple's licensing dance ends bash 3.2 as macOS default. zsh + oh-my-zsh becomes the new normal.
(b) Visual walkthrough · 15 min
How a pipe actually works
Six processes running in parallel, connected by kernel-managed in-memory buffers. No temp files, no waiting. When head has 10 lines it closes its stdin; sort -rn gets SIGPIPE and terminates; a cascade of shutdowns walks back to cat. This is why pipes are fast — the OS does the scheduling for you.
The 20 commands that do 90% of the work
Your daily toolbox
The pipeline that answers a real question
You've got an nginx access log with 4 million lines. Question: which 10 URLs got the most HTTP 500 errors in the last hour? In one line:
grep " 500 " access.log \
| awk '$4 > "[04/Jul/2024:14:00:00" {print $7}' \
| sort | uniq -c | sort -rn | head -10Keep only lines with a 500 status code (spaces prevent matching timestamps that contain 500).
Keep lines after 14:00, print field 7 (the URL). awk splits on whitespace by default.
Groups identical URLs next to each other — required for uniq to work.
Collapse runs of identical lines into a single line prefixed with the count.
Sort reverse-numeric — largest counts first.
Just show the top 10 offenders.
That is the shape of every log-analysis pipeline you'll ever write: filter → extract → aggregate → sort → limit. Learn the shape; the tools you already have handle the rest.
The tools most beginners haven't heard of (but should use)
Text search
- grep — POSIX standard, everywhere
- rg — 5-100× faster, respects .gitignore
- rg is smart about UTF-8 and binary files
- Use rg for interactive search, grep in scripts (for portability)
File search
- find — POSIX standard, byzantine syntax
- fd — sane defaults, colored output
- fd 'session-\d+' vs find . -regex './session-[0-9]+.*'
- Use fd interactively, find in scripts
File viewing
- cat — dump raw bytes
- bat — cat with syntax highlighting + git diff markers
- bat is the ‘why haven't I always used this?’ tool
- Alias `cat=bat` if you like
Process monitor
- top — everywhere, ugly, keyboard-driven
- htop — colorful, mouse-clickable, sane
- btop — pretty graphs, resource-heavy
- Use htop day-to-day
The mental model to hold
"Commands like ls *.log receive the pattern *.log and expand it themselves. The shell just launches programs."
The shell expands globs before the program ever starts. ls receives an already-expanded list of filenames — it has no idea a * was typed. Same for ~, $VAR, {a,b}, and command substitution.
Because the illusion is perfect until it breaks in exactly two ways, and both look like bugs in the program. First: find . -name *.log fails mysteriously — the shell expanded the glob against the current directory before find ran, so find got a filename instead of a pattern. Second: rm * in a huge directory dies with "Argument list too long" — the shell built a multi-megabyte argv that exceeded the kernel's execve limit. Neither makes sense unless you know expansion happens one layer earlier than you thought.
Make the expansion visible — echo shows you exactly what the program would have received:
mkdir -p /tmp/g && cd /tmp/g && touch a.log b.log
echo *.log # a.log b.log <- shell already expanded it
echo '*.log' # *.log <- quotes suppress expansion
# the classic failure:
find . -name *.log # breaks: find sees 'a.log'
find . -name '*.log' # correct: find does its own matchingWhy can a Unix pipeline of five tiny programs beat a purpose-built script on a 200 GB file — and why does adding sort in the middle sometimes destroy that advantage entirely?
- 1A pipe is a fixed-size in-kernel ring buffer between two processes, not a file and not a copy of the data.forced by · the kernel needs bounded memory regardless of how much data flows through
- 2Because the buffer is bounded, a fast producer blocks when it fills and a fast consumer blocks when it empties. Backpressure is automatic and requires no cooperation between the programs.forced by · a blocking write on a full pipe is the only sane behaviour when you cannot grow the buffer
- 3Therefore every stage runs concurrently on a different chunk of the stream, and peak memory is O(buffer × stages) — independent of input size.forced by · no stage ever needs to hold the whole input to produce its next byte of output
- 4This works only for streaming operators — ones whose next output depends on a bounded window of input.
grep,cut,sed,awk,headall qualify.forced by · if output depends on unseen future input, you cannot emit anything yet - 5But
sortcannot emit its first line until it has seen the last one. It is a blocking barrier: it must buffer everything (spilling to disk when it exceeds memory).forced by · the minimum element could be the final line of the file
Therefore a pure streaming pipeline processes arbitrarily large inputs in constant memory with N-way parallelism for free, while any blocking stage collapses that into "materialise everything, then continue".
And note what this predicts: put grep before sort, never after — filtering early shrinks what the barrier must hold. It also predicts why head -1 on a huge file returns instantly while sort | head -1 does not, and why yes | head -5 terminates at all: head exits, the pipe closes, and yes dies of SIGPIPE. This is exactly the same streaming-vs-blocking distinction that governs map-side operations versus shuffles in Spark.
Every process is born holding three wires: stdin (0), stdout (1), stderr (2). It does not know and cannot ask whether a wire leads to a terminal, a file, another process, or nowhere. The shell's entire job before exec is to solder those wires to the right endpoints.
So >, <, |, 2>&1 are not features of the program — they are wiring instructions to the shell, applied left to right, before the program exists.
- Order matters because redirections apply sequentially:
cmd > f 2>&1sends both to the file;cmd 2>&1 > fsends stderr to the old stdout (your terminal) and only stdout to the file. - Errors go to stderr specifically so they survive being piped — that is why
cmd | grep xstill shows error messages. - Exit status is the other channel: 0 means success, non-zero means failure, and
&&/||/set -eall read it. In a pipeline only the last command's status counts unless you setpipefail. - Text with one record per line is the universal interchange format — which is why
xargs,-print0, andIFSexist to handle the case where filenames contain spaces or newlines.
Fire this model the moment you see: output missing from a log file · 2>&1 that didn't work · a pipeline that reports success despite a failed middle stage · "Argument list too long" · a script that behaves differently under cron than in your terminal.
You have a data-munging task. Do you write a shell pipeline, a Python script, or push it into a query engine?
grep/awk on a local file routinely beat naive Python by a wide marginPrototype in the shell, ship in Python, scale in the engine — and be honest about which stage you are in. The shell's real value is the feedback loop: you can inspect 200 GB with head, wc -l, and awk in seconds and learn the shape of the data before writing any code at all.
The mistake seniors avoid in both directions: don't grow a 200-line bash script that should have been Python an hour ago, and don't spin up a cluster for a file that sort -u would have handled on a laptop.
(c) Hands-on · 25 min
Run this end-to-end. It creates a fake nginx log, then walks you through progressively harder questions about it. Save as shell-workshop.sh, chmod +x, run.
What each block does
Anatomy of the script
Use the public GitHub API (no auth needed for low volume) and jq:
# Q1: names of the last 10 repos user "torvalds" created
curl -s "https://api.github.com/users/torvalds/repos?sort=created&per_page=10" \
| jq -r '.[].name'
# Q2: top 5 most-starred of those repos
curl -s "https://api.github.com/users/torvalds/repos?per_page=100" \
| jq -r 'sort_by(-.stargazers_count) | .[0:5] | .[] | "\(.stargazers_count) \(.name)"'
# Q3: how many are forks?
curl -s "https://api.github.com/users/torvalds/repos?per_page=100" \
| jq '[.[] | select(.fork)] | length'Six aliases every senior dev has in their .bashrc
# Add to ~/.bashrc — restart shell or `source ~/.bashrc`
alias ll='ls -alF --color=auto'
alias ..='cd ..'
alias ...='cd ../..'
alias grep='grep --color=auto'
alias gs='git status -sb'
alias gl='git log --oneline --graph --all --decorate -20'These aren't optional. They're the reason a senior dev can gs twenty times a day without their wrists hurting.
(d) Production reality · 15 min
The Steam Linux client shipped an update with an unquoted variable in a shell script:
rm -rf "$STEAMROOT/"*When STEAMROOT was unset (a rare install path), the line became rm -rf /* — recursively delete every file on the system.
set -euo pipefail catches unset variables (-u), and ${'{STEAMROOT:?not set}'} would have aborted with a message instead of expanding to nothing."$var", never bare $var; (2) always start scripts with set -euo pipefail. These two rules would have prevented this incident entirely.3 am pager. Junior SRE runs grep ERROR huge.log > errors.txt. Two minutes later disk full, half of production is choking on its own logs.
The problem: huge.log was 200 GB, and most lines matched. errors.txt was on the way to 190 GB.
Kill the grep. Free disk. Retry with awareness: grep -c ERROR huge.log first to see the count, then grep ERROR huge.log | head -1000 > sample.txt.
Better: use rg (ripgrep), which streams and is faster. Best: ship logs to a real search backend (Loki, Elasticsearch) and grep the index, not the raw file.
wc -l and du -h first. On any pipeline that writes to disk, always know the upper bound. On production hosts, always df -h before starting anything that grows.A nightly cleanup cron ran find /tmp -mtime +7 -exec rm {'{}'} \;. Innocent-looking. But find included /tmp/.mysql.sock, the Unix socket for MySQL. Killed every DB connection at 2 am.
Rewrite with a type filter: find /tmp -type f -mtime +7 -delete. Prefer -delete to -exec rm (safer, faster, no per-file fork).
Add a dry-run in every destructive cron: find ... -print for a week before switching to -delete.
find ... -exec rm or find ... -delete in production should be paired with (a) a type filter (-type f), (b) a specific name pattern, and (c) a prior week of -print logging. Cleanup crons cause more incidents than actual bugs.Where this shows up in the rest of the plan
(e) Recall + stretch · 10 min
Explain-out-loud test
If you can't teach these three without notes, redo the session:
- What is a pipe, and why is it fast? (mention streams)
- What does
set -euo pipefailprevent? (name each flag) - When would you use
jqinstead of Python? (one concrete example)
What comes next
Hub: The 6-Month Learning Plan
Part of a 130-session evergreen learning series. Session structure: intuition → visual → hands-on → production war stories → recall. Duration: 90 minutes.