S046 · Batch vs Streaming — Mental Model & Use Cases

Module M05: Data Engineering · Session 46 of 130 · Track: DE 🌊

What you'll be able to do after this session

• Explain Batch vs Streaming to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S045 — Data Modelling — Dimensional, Data Vault, OBT

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · Batch Processing vs Stream Processing Explained — plain-language contrast with concrete examples.
• 🎥 Deep dive (30–60 min) · Batch Processing vs Stream Processing | Data Engineering Explained — deeper walkthrough with architecture diagrams.
• 🎥 Hands-on demo (10–20 min) · Stream Processing System Design Explained | Kafka, Flink, Spark — end-to-end design of a streaming system so you feel the moving parts.
• 📖 Canonical article · Databricks — Batch vs Streaming Data Processing — vendor-neutral summary of when each fits.
• 📖 Book chapter / tutorial · Google — The Dataflow Model (paper) — Tyler Akidau's landmark paper that unified batch and streaming semantics. Read the intro.
• 📖 Engineering blog · Netflix — Keystone Real-time Stream Processing — how one of the world's biggest streaming platforms actually runs streaming.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: The two shapes of data pipelines.

Think of doing your laundry two ways. Batch is: dump all the week's dirty clothes into the machine every Saturday, run one big cycle, fold everything. High throughput per cycle, low overhead, but you always wear day-old socks. Streaming is: every time you take off a shirt, you drop it into a magic machine that washes and folds it 30 seconds later. Always-fresh laundry, but the machine runs 24/7, uses more electricity, and if it jams at 3am you have a real problem. That's the entire mental model — batch pipelines process big chunks of data on a schedule; streaming pipelines process events one-by-one (or in tiny micro-batches) as they arrive.

Every real company runs both. The batch world (nightly ETL, warehouse loads, monthly finance reports, daily ML training jobs) is mature, cheap, and forgiving of failure — a job that dies at 3am can just be re-run at 4am. The streaming world (fraud detection, real-time personalization, live dashboards, alerting) is younger, more expensive, and unforgiving — if a fraud model is 5 minutes behind, cards get charged. Choose based on how stale is too stale for this decision to still be useful? If the answer is "hours," use batch. If it's "seconds," use streaming. In between (~1-5 min), micro-batching (Spark Structured Streaming) is often the sweet spot.

The gotcha to carry forever: streaming does not make batch obsolete. Even at Netflix, Uber, and Airbnb, the vast majority of data volume still flows through batch. Streaming is for the tail — the small percentage of decisions that must be low-latency. And streaming is much harder to get right — late data, out-of-order events, exactly-once semantics, replays after bugs, state recovery — all of which S049 covers. Start batch, add streaming when the business needs prove it.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

Two architectural shapes, side by side:

Batch vs Streaming — decision table:

Worked example — one company's data flow. An e-commerce site handling 5M orders/day might use:

1. Streaming (Kafka → Flink → Redis): update recommendation features and detect card fraud in <1 sec per order.
2. Micro-batch (Kafka → Spark Structured Streaming, 30-sec triggers): populate a "sales in the last hour" dashboard.
3. Batch (Airflow-scheduled Spark, hourly): update daily/weekly aggregates in the warehouse.
4. Batch (Airflow, nightly): retrain the recommendation model on the last 90 days.

Same events (orders), four different latency tiers. Each tier costs 5–10x more per event than the next slower one — so you push only what needs to be fast into the fast tier. This layered pattern (originally called "Lambda architecture," now more commonly a single Kafka backbone feeding multiple sinks) is the modern default.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

Two tiny scripts side by side — batch vs streaming — over the same synthetic event stream.

# events.py — a shared generator: fake e-commerce orders
import json, random, time, sys
skus = ["A","B","C","D","E"]
def gen():
    return {"ts": time.time(),
            "sku": random.choice(skus),
            "qty": random.randint(1,3),
            "price": round(random.uniform(50,500),2)}
if __name__ == "__main__":
    while True:
        print(json.dumps(gen()), flush=True)
        time.sleep(0.05)   # 20 events/sec

# batch.py — read a file of 10k events, aggregate once, exit
import json, collections, sys
totals = collections.defaultdict(float)
for line in sys.stdin:
    e = json.loads(line)
    totals[e["sku"]] += e["qty"] * e["price"]
for sku, rev in sorted(totals.items(), key=lambda kv:-kv[1]):
    print(f"{sku}: {rev:>10.2f}")

# streaming.py — process events as they arrive, print running totals every 5 sec
import json, sys, time, collections
totals = collections.defaultdict(float)
last = time.time()
for line in sys.stdin:
    e = json.loads(line)
    totals[e["sku"]] += e["qty"] * e["price"]
    if time.time() - last > 5:
        print("
--- 5-sec snapshot ---")
        for sku, rev in sorted(totals.items(), key=lambda kv:-kv[1]):
            print(f"  {sku}: {rev:.2f}")
        last = time.time()

Run them:

# Batch: capture 10 seconds of events into a file, then process
python3 events.py | head -200 > events.jsonl
python3 batch.py < events.jsonl
 
# Streaming: pipe live events directly, get rolling snapshots
python3 events.py | python3 streaming.py
# Ctrl-C when bored

Observe (checklist):

1. batch.py gives the exact right answer once, over a fixed input, then exits. Deterministic.
2. streaming.py never finishes — it emits partial answers forever, and each 5-second snapshot is slightly wrong (it doesn't know about events yet to arrive).
3. Add a time.sleep(2) in the middle of events.py — the batch job still finishes fine; the streaming job just goes quiet for 2 seconds. Different failure modes.
4. Kill and restart streaming.py — you lose all in-memory state (that's what real stream engines add via checkpointing).
5. The streaming version is always running — cost is proportional to uptime, not to work done.

Try this modification: rewrite streaming.py to emit a snapshot only for events in the last 60 seconds (a "sliding window"). You'll have to keep a per-event queue and evict old entries. Congratulations, you just implemented what Flink calls a TumblingWindow — badly. This is why you use Flink.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

The most common streaming failure story: "we built a beautiful Flink pipeline for fraud detection, and one week later a schema change upstream broke it, and we had zero fraud alerts for 6 hours before ops noticed." Streaming systems fail silently by default — no rows produced looks the same as "everything's fine, no fraud today." Every streaming pipeline in production needs: (a) heartbeat metrics ("events processed per minute"), (b) SLA alerts ("if lag > 60 sec, page someone"), (c) schema-registry contracts on the topic so bad data is rejected at the boundary, not deep in the pipeline. Confluent's Schema Registry and Protobuf/Avro exist for exactly this reason.

Uber's public engineering blog documents the shift from Lambda architecture (parallel batch + streaming pipelines, reconciled nightly) to Kappa architecture (streaming is the source of truth; batch is just "streaming replayed over history"). Both patterns are valid; the Kappa version is simpler operationally but only works if Kafka retention is long enough to replay months of history (some teams pay for 30-day retention specifically for this).

Cost is the sleeper. A Kafka + Flink + monitoring + on-call setup costs a lot more than "cron + Spark." At Microsoft in ODSP we did the math: streaming was ~7x the per-event cost of batch. So we streamed only the ~5% of events that fed real-time features and dashboards, and batched the rest through nightly Spark. That single decision (correct tiering) saved millions of dollars per year.

Finally: backfills are the hardest part of streaming. When a bug ships that computed the wrong metric for the last 3 days, in batch you re-run the DAG for those 3 days and you're done. In streaming, you must rewind the Kafka offsets, replay everything, and pray downstream sinks are idempotent. Most teams eventually build a "replay tool" or move to Lakehouse patterns (Delta Lake / Iceberg) that make batch-style backfills of streaming output tractable. This is a big driver of the Iceberg boom in 2024–2026.

(e) Quiz + exercise · 10 min

1. In one sentence: what determines whether you should use batch or streaming for a given use case?
2. Give one concrete workload where batch is clearly correct, and one where streaming is clearly correct.
3. Why is a streaming pipeline typically 5-10x more expensive per event than a batch pipeline doing the same aggregation?
4. What is the difference between "true streaming" (Flink) and "micro-batch streaming" (Spark Structured Streaming)?
5. Explain in one paragraph why "backfills" are much harder to do in streaming than in batch.

Stretch (connects to S045 — Dimensional modelling): you have a star schema in a warehouse loaded by nightly batch. Business wants "today's sales dashboard refreshed every 5 minutes." Sketch three architectures (pure batch with 5-min schedule; micro-batch streaming into the warehouse; streaming into a fast store like Druid layered over batch). Which do you pick and why?

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is Batch vs Streaming? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S047): Spark — RDD, DataFrame, Jobs/Stages/Shuffles
• Previous (S045): Data Modelling — Dimensional, Data Vault, OBT
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.