S003 · The Command Line — bash, pipes, grep, jq

Module M00: Setup & Tools · Session 3 of 130 · Track: Setup ⌨️

What you'll be able to do after this session

• Explain The Command Line to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S001 — Dev Environment — Linux/WSL, Terminal, VS Code

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · Bash in 100 Seconds — Fireship's whistle-stop tour of what the shell is.
• 🎥 Deep dive (30–60 min) · MIT Missing Semester — Shell Tools and Scripting — The definitive 'pipes, redirection, find, grep' lecture.
• 🎥 Hands-on demo (10–20 min) · jq Tutorial — Parse JSON on the Command Line — Cameron Nokes shows why jq is the missing piece for API work.
• 📖 Canonical article · GNU Bash Reference Manual — Shell Expansions — The authoritative reference; skim to know it exists.
• 📖 Book chapter / tutorial · MIT Missing Semester — Shell Tools and Scripting (notes) — Written companion with exercises — do them all.
• 📖 Engineering blog · Julia Evans — How to use grep — Julia's short posts on grep/find/ripgrep are the internet's best CLI cheatsheets.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: Live in the terminal and never fear it again.

The command line is a text-mode conversation with your computer. You type a verb ("show me these files"), the computer does it, prints an answer, and waits for the next verb. That's it. It feels intimidating because there's no menu of options — but that's also the superpower: every action is a word you can save into a script, share with a friend, or run on a server 8,000 km away that has no screen.

Every command follows one pattern: verb noun --flags. ls -la ~/projects means "list (verb) my projects folder (noun), in long format and including hidden files (flags)." Once you see it, you see it everywhere.

The magic ingredient is pipes (|). Each command reads text from its input and writes text to its output. A pipe glues one command's output to the next's input, so you can build a Lego tower of tiny tools: cat log.txt | grep ERROR | sort | uniq -c | sort -rn | head — read log, keep error lines, sort, count duplicates, sort by count descending, show top 10. That's a one-line log analyzer.

Gotcha to carry forever: spaces matter and quoting matters. rm -rf /home/you/tmp and rm -rf / home/you/tmp differ by one space; the second deletes your entire disk. Quote paths with spaces (rm -rf "my folder") and prefer trash over rm while you're learning.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

The shell is a text pipeline. Here's the mental picture:

Every command has three streams: input, normal output, and error output. You can redirect any of them with <, >, 2>, &>.

The 15 commands that cover 95 % of daily work:

Worked example 1 — find the 5 biggest files under a folder:

find ~/projects -type f -printf '%s %p\n' | sort -rn | head -5

Read right-to-left as words: "keep the top 5 (head -5) of the reverse-numerically-sorted (sort -rn) list of (size, path) pairs produced by find."

Worked example 2 — count HTTP status codes in an nginx log:

awk '{print $9}' access.log | sort | uniq -c | sort -rn | head

Extract column 9 (the status), sort so duplicates are adjacent, uniq -c collapses and counts them, sort by count descending, show top. Every SRE writes some variation of this five times a week.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

Run this end-to-end. You'll build a real "which errors happened most today?" log analyzer.

#!/usr/bin/env bash
set -euo pipefail
 
mkdir -p ~/projects/learning/s003 && cd ~/projects/learning/s003
 
# 1. Generate a fake log so we have something to grep.
python3 - <<'PY'
import random, datetime as dt
levels = ["INFO"]*80 + ["WARN"]*15 + ["ERROR"]*5
msgs = {
  "INFO":  ["user login", "cache hit", "request served"],
  "WARN":  ["slow query", "retrying", "cache miss"],
  "ERROR": ["db timeout", "null pointer", "429 rate limit"],
}
now = dt.datetime.utcnow()
with open("app.log", "w") as f:
    for i in range(1000):
        lvl = random.choice(levels)
        ts  = (now - dt.timedelta(seconds=random.randint(0, 86400))).isoformat()
        f.write(f"{ts} {lvl} {random.choice(msgs[lvl])}\n")
PY
 
echo "--- 5 sample lines ---"
head -5 app.log
 
# 2. How many total lines?
wc -l app.log
 
# 3. Count each log level.
echo "--- count by level ---"
awk '{print $2}' app.log | sort | uniq -c | sort -rn
 
# 4. Show only the top-3 most common ERROR messages.
echo "--- top errors ---"
grep ' ERROR ' app.log | awk '{$1=""; $2=""; print}' | sort | uniq -c | sort -rn | head -3
 
# 5. Save just the errors to a new file, then follow it live.
grep ' ERROR ' app.log > errors.log
wc -l errors.log
 
# 6. Use jq on a real API (needs internet).
curl -s https://api.github.com/repos/torvalds/linux | jq '{name, stars: .stargazers_count, forks}'

Checklist — what to observe:

1. head -5 app.log shows ISO timestamps + a level + a message.
2. Step 3 prints something like 800 INFO, 150 WARN, 50 ERROR — the ratios you baked in.
3. Step 4's grep ' ERROR ' uses spaces to avoid matching "ERROR" appearing inside a message.
4. The curl … | jq call returns a compact JSON object with just 3 fields.
5. wc -l errors.log matches the ERROR count from step 3.

Try this modification: rewrite the analyzer as a single one-liner with no intermediate files, then pipe it through tee summary.txt so you can see the output AND save it to a file at the same time. Bonus: run time in front to see how many milliseconds the whole pipeline takes.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

War story — the accidental rm -rf. A Steam client bug once had rm -rf "$STEAMROOT/"* where $STEAMROOT was empty. Result: rm -rf /* — the script cheerfully deleted the user's home directory. Rule burned in from that day: set -u at the top of every script (errors on undefined vars) and quote every expansion.

War story — the "quick production fix." A dev SSHs into prod, edits /etc/nginx/nginx.conf with vim, and reloads. Server dies at 3 a.m. two weeks later when auto-scaling spins up a new machine with the old config. Lesson: never edit prod servers directly. Everything through Git and config management (Ansible, Terraform, Kubernetes). This is why Session S056+ hammers on containers and infrastructure-as-code.

The tools real SREs live in:

• rg (ripgrep) replaces grep — 10× faster, respects .gitignore.
• fd replaces find — friendlier syntax.
• bat replaces cat — syntax highlighting + line numbers.
• jq for JSON, yq for YAML.
• tmux for persistent SSH sessions that survive network drops.
• htop / btop instead of top for a readable process view.
• fzf for fuzzy history search (Ctrl+R becomes magical).

Shell scripting hygiene at Google / Netflix scale:

1. Every non-trivial script starts with set -euo pipefail — die on error, undefined var, or pipe failure.
2. Quote every variable: "$var", not $var.
3. Prefer [[ … ]] over [ … ] for tests.
4. Use shellcheck in CI to lint every .sh file.
5. If a script exceeds 100 lines, rewrite it in Python. Shell is glue, not an application language.

The one habit that will save you the most: history | grep <command> to find the exact command you ran yesterday. Combined with Ctrl+R for reverse-search, you'll rarely retype anything.

(e) Quiz + exercise · 10 min

1. What are stdin, stdout, and stderr, and how do you redirect each to a file?
2. What does the pipe | actually do between two commands?
3. Why do experienced devs write set -euo pipefail at the top of every bash script?
4. What is the difference between grep -R 'foo' . and find . -name 'foo'?
5. Given a JSON file data.json with {"users": [{"name":"A"}, {"name":"B"}]}, what jq expression prints just the names, one per line?

Stretch problem (connects to S002): write a one-line pipeline that lists every unique author name across the last 100 commits of a Git repo, sorted by commit count descending. Hint: git log --pretty=format:'%an' -n 100 | ….

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is The Command Line? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S004): Reading Docs & Effective Googling — the Meta-Skill
• Previous (S002): Git & GitHub — Commits, Branches, PRs
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.