The Command Line — bash, pipes, grep, jq

Review boundary and corrected contracts

Pipelines connect stdout to stdin but stderr remains separate unless redirected. Pipeline processes can overlap; sort and aggregating awk/jq can materialize unbounded input, so constant-memory claims are conditional. Quoting protects arguments from splitting/globbing; shell commands are documented exercises, not executed host changes.

The examples below teach the stated input contracts; fragments and deliberately incorrect counterexamples are exercises, not production implementations. Complexity counts elementary operations unless a bit-cost or allocation caveat is stated. Practice times are suggestions, not measured performance or hiring guarantees.

Learning objective: Chain shell commands, inspect exit status and parse a declared log/JSON format correctly. Practice speed only after checking field positions, time bounds and negative cases.

Why this session exists

A shell is useful because small programs can be composed, inspected and rerun. A pipeline is not automatically correct just because it is short or fast: substring matching, quoting, exit status and record formats all affect the result. This lesson works from explicit fixtures rather than a senior-versus-junior speed story.

You will be able to

• Navigate the filesystem, find files, and inspect processes without touching a GUI.

• Chain commands with pipes to answer questions like ‘which endpoint had the most 500s in the last hour?’ from a raw log file.

• Use grep, sed, and awk well enough that you stop reaching for Python for one-off text munging.

• Slice, filter, and reshape JSON with jq — the tool every API and Kubernetes user needs.

• Read a stranger's bash script and predict what it does before running it.

Prerequisites

• S001 — Dev Environment — a working Linux/WSL/macOS shell.
• S002 — Git & GitHub — you'll practise these commands inside a Git repo.

Pre-read · skim before the session (35 min total)

Further reading: Missing Semester · Shell Tools and Scripting

Further reading: Bash Pitfalls

Further reading: jq Tutorial (official)

(a) Intuition · 5 min

Why Unix people love pipes

Real: Imagine a factory floor. Every station does one job — cut, weld, paint, box. Parts flow along a conveyor. To make a new product you rearrange the stations, you don't build a new factory. The Unix shell is that factory. Each command is a station. The pipe (|) is the conveyor. To answer a new question you snap together commands you already know — you don't write a new program.

 
    Technically: many Unix filters read stdin and write stdout; other commands use arguments, files or devices. A pipe connects file descriptors through kernel buffers. The commands may still read/write disk, accumulate state or spill temporary files.
    That is the entire Unix philosophy: do one thing well, and communicate via text streams. Fifty years later, no other OS design has aged as gracefully.
  

The three streams that everything runs on

The usual shell-launch convention: three standard descriptors

• stdin (fd 0) — where input comes from. Default: your keyboard. Redirect with < or a pipe.

• stdout (fd 1) — where normal output goes. Default: your terminal. Redirect with > or a pipe.

• stderr (fd 2) — where errors go. Default: your terminal too, but a SEPARATE stream. Redirect with 2>.

The distinction that saves your logs

If stdout is redirected but stderr is not, errors can remain on the terminal or another inherited destination instead of the output log. cmd > out.log catches stdout only. cmd > out.log 2>&1 also captures stderr into the same file. cmd &> out.log is Bash shorthand, not portable POSIX sh syntax. Learn this now, before you spend an hour debugging a "silent" cron job.

A quick history so you know why the world looks like this

Unix-style tools share file-descriptor and byte-stream conventions; later shells and structured-data tools build on those conventions rather than replacing them. Bash has features not required by POSIX sh, and macOS commonly starts zsh with BSD versions of utilities. jq brings a JSON value model rather than treating every record as whitespace-separated text. Choose syntax for the actual shell/tool versions, not a universal Unix timeline.

(b) Following data through a shell pipeline · 15 min

How a pipe actually works

The shell tools have changed over time, but the stream interface lets old and new commands cooperate. Follow an access-log line through filtering, URL extraction, counting, and ranking; each arrow carries the previous command’s output into the next command’s input.

The diagram shows seven command stages, with a terminal as the destination. Stages can overlap, but sort must inspect all input before giving a globally sorted result and may spill temporary files. Pipe buffers provide backpressure: readers/writers can block. If head exits early, an upstream process that later writes may receive SIGPIPE; that outcome depends on buffering and is not a guaranteed cascade. With pipefail, an intentional early close can make a pipeline nonzero.

A practical command toolbox

Your daily toolbox

Name: Navigation

Desc: cd · ls · pwd · tree · pushd/popd — move around the filesystem.

Name: Inspect files

Desc: cat · less · head · tail · wc · file · stat — look at file contents and metadata.

Name: Find things

Desc: find (name/mtime/size) · fd (modern find) · locate — search for files by attributes.

Name: Search inside files

Desc: grep · rg (ripgrep) · ack — search inside files; defaults, ignore rules and availability differ.

Name: Transform text

Desc: sed (stream editor) · awk (mini language) · tr · cut · sort · uniq · wc — the classic pipeline toolkit.

Name: JSON / YAML / CSV

Desc: jq · yq · mlr (Miller) · csvkit — structured-data equivalents of the classic tools.

Name: Processes

Desc: ps · top · htop · kill · pgrep · pkill · jobs · fg/bg · nohup — see and control what's running.

Name: Network

Desc: curl · wget · ss · dig · nslookup · nc — hit URLs, resolve DNS, inspect sockets.

Name: Composition

Desc: | (pipe) · > < >> 2>&1 · && || · $() (command substitution) · xargs — glue the above together.

The pipeline that answers a real question

For this exercise, use a common-log-style record with a whitespace-free request target: IP is $1, bracketed local timestamp starts in $4, URL is $7, status is $9, and bytes is $10. nginx log formats are configurable; combined format appends referrer and user agent, so $NF is not its byte count. Escaped quotes, custom fields or a different format require a real parser.

Question: which 10 URLs had the most status-500 responses in the UTC hour beginning 14:00 on 4 July 2024? The following fixture-specific comparison assumes every line is from that same date with the same +0000 offset. It is not a general “last hour” parser across dates/time zones.

LC_ALL=C awk '$9 == 500 && $4 >= "[04/Jul/2024:14:00:00" && $4 < "[04/Jul/2024:15:00:00" {print $7}' access.log \
  | sort | uniq -c | sort -rn | head -10

Status and time filter

$9 == 500 tests the declared status field. In combined logs, grep " 500 " can also match a byte count of 500 on a status-200 line because more fields follow it. The synthetic common-style fixture ends at bytes, so that exact false match does not occur there; testing the declared status field still expresses the intended contract. The half-open time interval includes 14:00:00 and excludes 15:00:00. For mixed dates/offsets, parse timestamps and compare normalized instants; lexicographic day/month strings do not order arbitrary dates correctly.

awk extraction

After filtering, print $7. The fixture has ten possible URLs, so the count/rank result has at most ten lines; a larger real pipeline needs deliberate handling of head and SIGPIPE.

sort

Desc: Groups identical URLs next to each other — required for uniq to work.

uniq -c

Desc: Collapse runs of identical lines into a single line prefixed with the count.

sort -rn

Desc: Sort reverse-numeric — largest counts first.

head -10

Desc: Just show the top 10 offenders.

That is the shape of every log-analysis pipeline you'll ever write: filter → extract → aggregate → sort → limit. Learn the shape; the tools you already have handle the rest.

The tools most beginners haven't heard of (but should use)

Name: grep vs ripgrep (rg)

Tagline: Text search

• grep — POSIX standard, everywhere

• rg — recursive search with ignore rules by default; performance depends on workload and options

• rg is smart about UTF-8 and binary files

• Use rg for interactive search, grep in scripts (for portability)

Name: find vs fd

Tagline: File search

• find — POSIX standard, byzantine syntax

• fd — sane defaults, colored output

• fd 'session-[0-9]+' uses a regex; a portable simple alternative is find . -type f -name 'session-[0-9]*' (a glob, not the same language)

• Use fd interactively, find in scripts

Name: cat vs bat

Tagline: File viewing

• cat — dump raw bytes

• bat — cat with syntax highlighting + git diff markers

• bat is the ‘why haven't I always used this?’ tool

• Use bat explicitly for interactive viewing; do not silently change script semantics by treating it as raw cat

Name: top vs htop vs btop

Tagline: Process monitor

• top — everywhere, ugly, keyboard-driven

• htop — colorful, mouse-clickable, sane

• btop — pretty graphs, resource-heavy

• Use htop day-to-day

The mental model to hold

Text is the universal interface

Many Unix filters speak text; programs can also emit binary data or structured formats. That's a limitation (you spend time parsing) and a superpower (nothing is a closed black box). When you learn a new tool — kubectl, terraform, docker — its output is text that you already know how to slice with grep, awk, and jq. The 20 commands above are a lifetime investment.

Myth: "Commands like ls *.log receive the pattern *.log and expand it themselves. The shell just launches programs."

Truth: The shell expands globs before the program ever starts. ls receives an already-expanded list of filenames — it has no idea a * was typed. Other expansions include tilde expansion, parameter expansion and command substitution; brace expansion is a Bash feature, not required by POSIX sh. Quoting and context decide which expansions occur.

Why: Because the illusion is perfect until it breaks in exactly two ways, and both look like bugs in the program. First: find . -name *.log fails mysteriously — the shell expanded the glob against the current directory before find ran, so find got a filename instead of a pattern. Second: rm * in a huge directory dies with "Argument list too long" — the shell built a multi-megabyte argv that exceeded the kernel's execve limit. Neither makes sense unless you know expansion happens one layer earlier than you thought.

Test: Use a new directory and print argument boundaries rather than trusting echo:

WORK_DIR=$(mktemp -d "${TMPDIR:-/tmp}/glob-workshop.XXXXXX")
cd "$WORK_DIR"
touch a.log b.log
printf '<%s>\n' *.log
printf '<%s>\n' '*.log'
find . -name '*.log' -print

The first printf gets two filename arguments, the second gets the literal pattern, and find receives a pattern it can match in each visited directory. A no-match Bash glob is normally left literal, unless options such as nullglob/failglob change that policy.

Question: When can a streaming pipeline use bounded memory, and what changes when sort needs the whole input?

Premise: A pipe is a kernel-managed byte-stream channel with bounded buffering; its capacity and implementation depend on the system. It is not a complete in-memory copy of the input.

Because: the kernel needs bounded memory regardless of how much data flows through

Premise: Because the buffer is bounded, a fast producer blocks when it fills and a fast consumer blocks when it empties. Backpressure is automatic and requires no cooperation between the programs.

Because: in ordinary blocking mode, writers wait for capacity; nonblocking I/O instead needs explicit retry/readiness handling

Premise: Stages may overlap on different chunks; total memory includes the bounded pipe buffers and each program’s state, so it is not generally independent of input size.

A pipeline has bounded pipe buffers, but each process can allocate additional memory. Streaming filters may use bounded state; sort, arbitrary awk programs and jq -s can materialize input.

Premise: This works only for streaming operators — ones whose next output depends on a bounded window of input. A simple grep/cut/awk line filter can qualify subject to record size; an awk associative array grows with distinct keys, and sed hold-space or multiline programs can also accumulate input.

Because: if output depends on unseen future input, you cannot emit anything yet

Premise: But sort cannot emit its first line until it has seen the last one. It is a blocking barrier: it must buffer everything (spilling to disk when it exceeds memory).

Because: the minimum element could be the final line of the file

Push a filter before a blocking sort when that preserves semantics. Some filters depend on sorted order and cannot be moved without changing the answer.

Name: Everything is a byte stream through three wires

Picture: A shell-launched process normally inherits three standard descriptors: stdin (0), stdout (1), stderr (2). A program can inspect descriptors (for example with isatty/fstat) and may change buffering or formatting for a terminal. One shell responsibility before executing a command is connecting those descriptors. So >, <, |, 2>&1 are not features of the program — they are wiring instructions to the shell, applied left to right, before the program exists.

• Order matters because redirections apply sequentially: cmd > f 2>&1 sends both to the file; cmd 2>&1 > f sends stderr to the old stdout (your terminal) and only stdout to the file.

• Errors go to stderr specifically so they survive being piped — that is why cmd | grep x still shows error messages.

• Exit status is the other channel: 0 means success, non-zero means failure, and &&/||/set -e all read it. In a pipeline only the last command's status counts unless you set pipefail.

• Newline-delimited text is convenient only when records cannot contain unescaped newlines. Use NUL-delimited filename interfaces (-print0, xargs -0) where supported; changing IFS alone is not a general filename parser.

Trigger: Fire this model the moment you see: output missing from a log file · 2>&1 that didn't work · a pipeline that reports success despite a failed middle stage · "Argument list too long" · a script that behaves differently under cron than in your terminal.

Decision: You have a data-munging task. Do you write a shell pipeline, a Python script, or push it into a query engine?

Name: Shell pipeline

Gain: quick composition with explicit stage boundaries; bounded-state filters can stream. Startup, scheduling, I/O and per-stage memory still cost resources; benchmark the actual alternatives.

Cost: byte/text interfaces, quoting and whitespace hazards, and explicit error-handling work. Shell scripts can and should have fixture tests; complexity, not a fixed line count, determines maintainability.

Pick: the task is a one-off or a filter, the data is line-oriented text, and you can write it in under ~15 lines

Name: Python script

Gain: testable, typed, debuggable, real data structures, and libraries for JSON/Parquet/HTTP that shell handles badly

Cost: interpreter startup, higher per-row cost, and you must implement streaming deliberately (generators) or you will load the file into RAM

Pick: the logic has branches, will be re-run by other people, needs tests, or touches formats that are not newline-delimited text

Name: Query engine (SQL / Spark)

Gain: declarative, a planner that reorders your filters and picks join strategies, and horizontal scale past a single machine

Cost: a distributed engine adds scheduling/infrastructure overhead; a local SQL engine need not start a cluster. Debugging includes query plans and data movement.

Pick: the data no longer fits one machine, or already lives in a table/lake and moving it out would cost more than querying it in place

Verdict: Prototype in the shell, ship in Python, scale in the engine — and be honest about which stage you are in. The shell's real value is the feedback loop: head can sample a file without scanning it all, while wc -l and full aggregations read their full input and are bounded by its size and storage throughput. Use samples to learn the data shape, and full scans when the question needs every record. Avoid both extremes: don't grow a 200-line bash script that should have been Python an hour ago, and don't spin up a cluster for a file that sort -u would have handled on a laptop.

(c) Hands-on · 25 min

Run this end-to-end. It creates a fake nginx log, then walks you through progressively harder questions about it. Save as shell-workshop.sh, chmod +x, run.

#!/usr/bin/env bash
# shell-workshop.sh — a hands-on tour of pipes, grep, awk, sed, jq.
set -euo pipefail
export LC_ALL=C
 
log() { printf "\n\033[1;36m▶ %s\033[0m\n" "$*"; }
 
log "1/10 Fresh workspace"
WORK_DIR=$(mktemp -d "${TMPDIR:-/tmp}/shell-workshop.XXXXXX")
cd "$WORK_DIR"
 
log "2/10 Generate a synthetic nginx log (10k lines)"
python3 - <<'PY' > access.log
import random, datetime as dt
random.seed(42)
urls = ["/","/api/login","/api/users","/api/orders","/static/main.css",
        "/api/search","/api/checkout","/health","/api/users/42","/robots.txt"]
statuses = [200]*80 + [301]*5 + [404]*10 + [500]*5
start = dt.datetime(2024, 7, 4, 13, 0, 0)
for i in range(10_000):
    ts = start + dt.timedelta(seconds=random.randint(0, 7200))
    ip = f"10.0.{random.randint(0,255)}.{random.randint(0,255)}"
    method = random.choice(["GET","GET","GET","POST"])
    url = random.choice(urls)
    status = random.choice(statuses)
    size = random.randint(100, 8000)
    print(f'{ip} - - [{ts.strftime("%d/%b/%Y:%H:%M:%S +0000")}] '
          f'"{method} {url} HTTP/1.1" {status} {size}')
PY
wc -l access.log
 
log "3/10 First 3 lines, last 3 lines"
head -3 access.log
echo "..."
tail -3 access.log
 
log "4/10 How many 500s?"
awk '$9 == 500 {n++} END {print n+0}' access.log
 
log "5/10 Top 10 URLs by request count"
awk '{print $7}' access.log | sort | uniq -c | sort -rn | head -10
 
log "6/10 Top 10 URLs by 500-error count"
awk '$9 == 500 {print $7}' access.log | sort | uniq -c | sort -rn | head -10
 
log "7/10 Requests per hour (bucket by HH)"
awk '{split($4,t,":"); print t[2]}' access.log | sort | uniq -c
 
log "8/10 Total bytes served, per URL, sorted"
awk '{sums[$7]+=$10} END {for (u in sums) printf "%12d  %s\n", sums[u], u}' access.log \
  | sort -rn | head
 
log "9/10 Turn the log into JSON with sed + jq"
head -3 access.log \
  | sed -E 's/^([^ ]+) - - \[([^]]+)\] "([A-Z]+) ([^ ]+) [^"]+" ([0-9]+) ([0-9]+)$/\{"ip":"\1","time":"\2","method":"\3","url":"\4","status":\5,"bytes":\6\}/' \
  | jq -c '.'
 
log "10/10 Practise jq — group full log by status and count"
sed -E 's/^([^ ]+) - - \[([^]]+)\] "([A-Z]+) ([^ ]+) [^"]+" ([0-9]+) ([0-9]+)$/\{"ip":"\1","time":"\2","method":"\3","url":"\4","status":\5,"bytes":\6\}/' access.log \
  | jq -s 'group_by(.status) | map({status: .[0].status, count: length})'
 
echo "--- done ---"

What each block does

Anatomy of the script

Name: Block · set -euo pipefail

Desc: Bash -e exits on some unhandled failures, with exceptions in conditionals, lists and other contexts; -u rejects many unset expansions; pipefail reports the rightmost failing pipeline stage. This is not comprehensive error handling. Expected grep no-match status and intentional SIGPIPE need a chosen policy.

Name: Block · python3 - <<'PY' > access.log

Desc: Heredoc with quoted delimiter (no shell expansion inside) + output redirect. A common trick to embed a small program inside a bash script.

Name: Block · awk '{print $7}'

Desc: With its default field separator, awk splits whitespace-delimited fields $1 through $NF. Under this lesson’s declared request format, field 7 is the URL.

Name: Block · sort | uniq -c | sort -rn

Desc: The canonical ‘count occurrences of each unique line’ pipeline. uniq groups adjacent equal lines; sorting makes all identical values adjacent for a global count.

Name: Hour bucket · awk split()

Desc: split($4,t,":") separates the timestamp’s date and clock parts; t[2] is the hour for this fixture. This is not a complete timestamp parser.

Name: Block · awk with associative array

Desc: sums[$7]+=$10 maintains one sum per distinct URL. END emits the totals. Memory grows with the number of distinct URLs, and numeric range/precision depends on the awk implementation.

Name: Block · sed -E 's/regex/replacement/'

Desc: sed -E uses extended regex and capture groups to rearrange this controlled synthetic format. It does not JSON-escape arbitrary quotes/backslashes in real logs. Use structured logging or a proper parser plus JSON serializer for untrusted records.

Name: Block · jq -s 'group_by(.status)'

Desc: -s slurps stdin into a single array first. group\_by clusters by a key; map transforms each group. jq is a full language too — invest one afternoon.

What to observe

Step 5 and step 6 use the SAME pipeline shape — only the filter differs. That is the pattern. Step 8 sums field 10 by URL. Compare the total with an independent sum over all records; timing is machine-dependent. In step 9, the sed regex is scary-looking but it's just capture-and-rearrange. You'll write your own by session S010. Step 9 previews three records; step 10 reads all 10,000. Its group counts must sum to 10,000. jq -s loads the full input into an array; this demonstration is not a bounded-memory solution for arbitrary log size.

Goal: Interpret one API response correctly, including its page and error boundaries

Hint: jq -r '.[] | .name' prints raw strings (no quotes). Chain head -5 at the end to preview. If a field is missing, // gives a default: .company // "none".

The following optional network examples are not run in this review. Public endpoints may permit unauthenticated requests but still rate-limit them. Inspect HTTP failures and pagination; never pipe an error object into a filter expecting an array. The three examples below describe their own page, not every repository the user owns.

# Q1: names of the last 10 repos user "torvalds" created
curl --fail --silent --show-error "https://api.github.com/users/torvalds/repos?sort=created&direction=desc&per_page=10" \
  | jq -r '.[].name'
 
# Q2: top 5 by stars within the first page of up to 100 returned repos
curl --fail --silent --show-error "https://api.github.com/users/torvalds/repos?per_page=100" \
  | jq -r 'sort_by(-.stargazers_count) | .[0:5] | .[] | "\(.stargazers_count)  \(.name)"'
 
# Q3: how many repos on that page are forks?
curl --fail --silent --show-error "https://api.github.com/users/torvalds/repos?per_page=100" \
  | jq '[.[] | select(.fork)] | length'

Six optional Bash aliases

# Add to ~/.bashrc — restart shell or `source ~/.bashrc`
alias ll='ls -alF --color=auto'
alias ..='cd ..'
alias ...='cd ../..'
alias grep='grep --color=auto'
alias gs='git status -sb'
alias gl='git log --oneline --graph --all --decorate -20'

Aliases are personal conveniences. Inspect/merge shell configuration deliberately; these lines are not executed here. GNU --color=auto is not a portable option for every platform utility. Scripts should name commands and options explicitly instead of relying on interactive aliases.

(d) Production reality · 15 min

Three failure cases to reason through. These are teaching scenarios, not attributed incident metrics.

1. An empty directory variable can make a destructive path target the wrong location even when the variable is quoted. Quoting prevents splitting/globbing of the expansion; it does not prove the value is nonempty or authorized. In a destructive maintenance script, require a nonempty value (for example ${ROOT:?set ROOT}), verify its resolved location and ownership, and preview the exact targets. This lesson does not run deletion commands.
2. Redirecting matches from a large log into a second file can exhaust disk when most lines match. grep -c counts matching lines without writing a copy; grep -m 20 bounds matching-line output. Decide whether sampling answers your question; it cannot replace a required full aggregate.
3. A cleanup selector can include sockets/directories unless constrained. Inspect with find <reviewed-directory> -type f -mtime +7 -print before any authorized deletion; verify the environment and races separately. A path/name match alone is not permission to remove it.

For navigation use pwd, ls, cd and find to establish the current directory and inspect targets before acting. For process inspection, ps -p <known-pid> -o pid,comm reads a selected process; jobs refers to the current shell’s jobs, not every process on the system. Sending signals changes process state and is not part of this read-only inspection exercise.

Where this shows up in the rest of the plan

Root: The shell is under every tool you'll use in the next six months

Label: S004 · Reading docs

Desc: man, --help, tldr, cheat — all shell tools you learn to grep through.

Label: S055 · CI/CD basics

Desc: GitHub Actions workflows are essentially remote bash scripts. Every skill here compounds.

Label: S056 · Docker

Desc: Dockerfile RUN has shell and exec forms; the default Linux shell form uses /bin/sh -c unless SHELL changes it, not necessarily Bash.

Label: S059 · Kubernetes basics

Desc: kubectl get pods -o json produces structured data for jq; cluster access and authorization are a separate exercise.

Label: Further practice · Observability & SRE

Desc: Log grep-fu, journalctl, tail -F | awk — all built on this session.

Label: Further practice · System design

Desc: A log-analysis design extends these contracts to ingestion, storage, indexing and failure handling.

(e) Recall + stretch · 10 min

Self-check (answer follows each question).

Recall — questions and worked answers

Q: What are stdin, stdout, and stderr — and why are they separate?

A: By convention shell-launched programs inherit stdin (fd 0), stdout (fd 1) and stderr (fd 2), although callers can close or redirect them. Separate descriptors let data flow through a pipe while diagnostics use another destination. Programs must actually follow that convention for the separation to work.

Q: Why does sort | uniq -c require the sort first?

A: uniq only collapses adjacent identical lines — it doesn't do global deduplication. Without a prior sort, duplicates scattered through the file are not collapsed.

Q: What does set -euo pipefail do, line by line?

A: In Bash, -e exits on failures only outside its documented exception contexts (for example tests in if/while and parts of &&/|| lists); -u treats many unset-variable expansions as errors; pipefail makes pipeline status the rightmost nonzero status, or zero if none fails. It is a useful policy, not proof that every error exits. Handle expected no-match/SIGPIPE outcomes deliberately.

Q: Given a JSON array of user objects with a .age field, print the names of users older than 30, using jq.

A: For numeric ages: jq -r '.[] | select(.age > 30) | .name'. For mixed/missing types, use jq -r '.[] | select((.age | type) == "number") | select(.age > 30) | .name'; jq’s cross-type ordering is not validation that an age is numeric.

Q: You want to redirect BOTH stdout and stderr to the same file. What's the modern shell syntax?

A: Bash supports cmd &> out.log; portable shell syntax is cmd > out.log 2>&1. Redirections apply left-to-right, so cmd 2>&1 > out.log leaves stderr pointing at the previous stdout destination. > truncates its output file; choose the path deliberately.

Q: What's the difference between find and grep?

A: find selects filesystem entries by attributes; grep selects matching text lines. A safe display-only composition is find . -type f -name '*.py' -exec grep -l 'TODO' {} +. It passes each filename as an argument, including names with spaces, rather than parsing ls output.

Q: Why should you almost always quote your variables in shell ("$var" not $var)?

A: In ordinary Bash command arguments, unquoted $var undergoes word splitting and filename expansion. If its value is foo bar, cmd $var passes two arguments, while cmd "$var" passes one. Assignment and other special contexts have different expansion rules; quote variable arguments unless you deliberately need those expansions.

Q: Explain what xargs does in one sentence, and give one common use.

A: xargs converts delimited input items into command arguments, possibly in several invocations. Use NUL-delimited filenames, for example find . -type f -name '*.log' -print0 | xargs -0 printf '<%s>\n', to display rather than delete selected names. Never parse ls output as a safe deletion list; embedded spaces/newlines are valid filename characters.

Q: What does awk '{print $NF}' do, and why is it useful?

A: It prints the last whitespace-delimited field of each line. For the synthetic common-log-style fixture here, the last field is bytes; in combined nginx logs the last field belongs to the user agent, and the URL is not generally last. Use the format’s declared field or a proper parser.

Q: Stretch — you have access.log (nginx combined format). Write a one-liner that outputs the top 5 IPs by total bytes served, sorted descending, formatted as bytes IP.

Stretch: true

A: For this fixture, bytes is field 10 (not $NF in general combined logs):

awk '{sums[$1]+=$10} END {for (ip in sums) printf "%.0f %s\n", sums[ip], ip}' access.log \
  | sort -rn | head -5

This keeps one sum per IP and requires numeric byte counts; real nginx can log - for absent values. Decide how to treat those records. If used under pipefail, choose an explicit policy for expected upstream SIGPIPE from head rather than ignoring all pipeline failures.

Explain-out-loud test

If you can't teach these three without notes, redo the session:

1. What is a pipe, and why is it fast? (mention streams)
2. What does set -euo pipefail prevent? (name each flag)
3. When would you use jq instead of Python? (one concrete example)

Model answers: (1) A pipe connects one process’s output descriptor to another’s input through bounded kernel buffering; this enables streaming overlap but not a universal speed advantage. (2) -e has exception contexts, -u checks many unset expansions, and pipefail exposes failed pipeline stages. (3) Use jq to select names from an already available JSON array; use a program with explicit validation/state when the transformation becomes more complex.

What comes next

Hub: The 6-Month Learning Plan

Part of a 130-session learning series. Suggested session: 70 minutes across the five labeled blocks, plus 20 minutes for setup, questions and retesting. Optional pre-reading is separate; these are pacing suggestions, not completion guarantees.

Corrective source notes

• Pipelines (Bash Reference Manual) — A pipeline connects one command’s standard output to the next command’s standard input. Checked 2026-10-07.

Governing references: Bash manual (pipelines, expansions, redirections and set), jq 1.7 manual, nginx access-log formats, and GitHub list-user-repositories API.

These citations support the named claims, not blanket certification of the lesson. Worked proofs are derived in the text; execution scope and finite-test limits are recorded separately.