Batch vs Streaming — Mental Model & Use Cases

Why this session exists

“Real time” is a requirement to quantify, not an engine to buy. The goal is to classify workloads by input boundedness, freshness and repair semantics, then justify batch, micro-batch or record-at-a-time processing with explicit assumptions. Streaming is not intrinsically correct or expensive; batch is not intrinsically complete or deterministic. Both need well-defined inputs, transformations and output publication.

You will place five workloads on a latency spectrum, ask three design questions, explain bounded/unbounded inputs, calculate a micro-batch budget, and handle late data, out-of-order records, backpressure and duplicate effects.

Prerequisites

NoSQL landscape introduces access patterns; data modelling supplies grain and historical attribution. Kafka is a retained log, not a required fifth NoSQL family or the only streaming substrate.

Mail versus notifications

A monthly bank statement resembles a bounded report; transaction notifications resemble incremental outputs. A statement can still need later corrections, and a notification can be delayed or duplicated. The useful distinction is not that one is always cheap and complete, but what the consumer may infer from each result.

A batch reads a finite input snapshot and exits. An unbounded pipeline keeps accepting data and may incrementally update state. Micro-batch is an execution technique that can implement streaming semantics, including event-time windows, persistent state and late-event handling. Spark Structured Streaming is the counterexample to the false claim that shrinking batches can never support stream semantics. Record-at-a-time processing is another execution technique, not the definition of correctness [Beam; Spark streaming].

The three questions that decide it

1. How fresh must the result be? Define event-to-visible-result p95/p99, not just trigger frequency. State separately when a result is provisional versus reconciled.
2. What is the input boundary? A file, manifest or snapshot can close a finite set. “Yesterday's events” is not complete simply because the calendar advanced; offline devices and source backfills can still add yesterday's records.
3. What is the cost of a late or wrong answer? A fraud-blocking decision, a dashboard and an audited statement have different action and correction requirements.

There is no five-minute threshold at which batch always wins, no universal tenfold engineering multiplier, and no rule that only trading/fraud systems need subsecond decisions. Measure the actual source delays, state size, startup overhead, update rate and sink behavior.

Historical context

MapReduce popularized scalable bounded computation; retained logs and stream processors support continuous ingestion and replay. The 2015 Dataflow model makes what is computed, where in event time, when outputs fire and how refinements accumulate distinct questions; Apache Beam exposes that model. Lambda and Kappa describe processing architectures, while Iceberg/Delta/Hudi are table-format ecosystems with connector-specific incremental support. A table format is not a new lateness or exactly-once theorem, and warehouse streaming ingestion is not itself a complete stream processor.

The latency and complexity spectrum

The diagram orders example freshness requirements, not guaranteed prices or engine latency. Place five workloads deliberately:

• Monthly invoice: bounded account-period inputs, audit/reconciliation before finalization.
• Hourly inventory dashboard: incremental batch may suffice if hour-old data is acceptable.
• Two-minute product dashboard: micro-batch can publish provisional counts with later corrections.
• Device safety alert: event-to-action deadline may require continuous processing and an independent fail-safe.
• Fraud authorization: the request deadline and available features may demand an online service plus streaming feature updates; waiting for a window to close may be too late.

A numerical budget, not a benchmark

Assume source delay 20s, scheduling wait at most 60s, compute 15s and sink visibility 5s: a no-backlog worst-case budget is 100s, meeting a 120s objective. Average schedule wait for uniformly distributed arrivals would be 30s, but that assumption is not a p99 guarantee. If the source alone delays 180s, reducing compute from 15s to 1s cannot meet 120s.

For an illustrative per-run overhead of 2 CPU-seconds and per-event work of 0.001 CPU-seconds at 100 events/s, a 60s batch costs 2 + 6000*0.001 = 8 CPU-seconds; a 1s batch costs 2 + 100*0.001 = 2.1 CPU-seconds per second. This arithmetic describes the assumed model, not measured Spark/Flink costs. Persistent engines may have different overheads, parallelism and backpressure. If sustained arrival is 120/s and service is 100/s, backlog grows 20/s regardless of the marketing label.

Bounded versus unbounded data

A bounded snapshot can be sorted or aggregated to completion for that snapshot; reproducibility additionally needs stable input bytes, code, configuration, randomness and external lookups. A finite prefix of a Kafka topic is bounded only if its end offsets are fixed. An unbounded source has no known final record, so “all-time exact result now” is impossible when future events can change it. Stateless mapping need not use windows; a cumulative count can be incremental; keyed windows create retained state and cleanup decisions.

Event time is when the event happened according to the source clock; processing time is when an operator handles it. Event-time windows answer historical business questions. Processing-time windows answer operational questions such as requests received by a worker during the last minute. Neither is universally wrong. Timezone, clock skew and future timestamps need validation or a future-skewed record can advance progress incorrectly.

Lambda, Kappa and lakehouse choices

Lambda: a batch view and speed view serve different freshness/completeness roles; their merge must avoid double-counting overlap and reconcile corrections. Two paths increase consistency/testing obligations, not necessarily exactly twice the code.

Kappa: one processing path replays retained history to build a new result version. Replay starts at the earliest needed retained position, not always offset zero. Compaction or retention may have removed history. Budget replay throughput and isolate new output from live serving until validated.

Lakehouse: streams may write table snapshots that batch jobs query. Atomic table commit, connector replay behavior, small-file maintenance and incremental-read semantics differ by implementation. Keep source manifests and result versions; do not infer exactly-once external effects from a file-format name.

Four pitfalls and the completeness derivation

Late data: event 10:03 arriving 10:15 still belongs to its event-time bucket; decide whether to update, quarantine or later reconcile. Out of order: a count can update commutatively without sorting every event, but order-sensitive state may need sequence/version handling. Backpressure: bounded buffers and admission/rate limits prevent uncontrolled accumulation; a retained log delays rather than eliminates the retention/capacity limit. Duplicate effects: recovery can replay input, so state/offset recovery and sink publication must agree.

Derive why progress matters. A window aggregate needs retained information until no accepted event can change it. If arbitrary old events can arrive and every historical keyed window must be corrected exactly, state or reconstructible historical input cannot be discarded merely because wall-clock time passed. Bounded state requires a completeness signal, lateness assumption, approximation, or external reconstruction policy. This is not a universal “pick two” theorem for every stream: a stateless map and a fixed-cardinality cumulative sum have different state requirements.

A watermark expresses event-time progress under an engine policy. Spark 3.5.7 derives progress from observed event times and a configured delay; its one-sided guarantee protects sufficiently recent data from watermark drops, while older data may or may not be processed. Flink separates watermark-triggered window firing from allowed-lateness cleanup and late updates. A delay threshold is not “sleep this long after each event,” and a quiet source may not advance event-time progress by itself.

Closed ledger versus open tab: publish a snapshot or a provisional view with a version and a correction contract. Durable raw input plus deterministic transforms helps rebuild either. Stream replay can be reproducible if the relevant input order and side inputs are retained; batch can be nondeterministic if it reads “latest” from an external service.

Choosing the operating model

More frequent batch reuses code but can overlap runs and repeatedly scan history. Micro-batch amortizes work and can retain stream state, but trigger overhead/checkpointing still matter. Record processing can reduce per-event scheduling delay and offer rich timers, but sinks and source delays can dominate. Select based on the deadline, required operators and the team's recovery capability. A batch reconciliation path is useful for many systems, not a mandatory second implementation for every correct stream.

Hands-on: identical event buckets, different completion policies

This complete Python 3 lab replaces the open-ended socket server with a deterministic bounded replay. It executes both batch counting and an incremental algorithm, including out-of-order input, a late drop, timezone normalization, snapshot/restart and a crash-without-snapshot negative control. No ports are opened. The incremental policy below is explicitly an algorithm example, not an emulator of Spark/Flink's batch-specific watermark internals.

from collections import Counter
from datetime import datetime, timezone
import copy
 
BASE = int(datetime(2024,8,10,tzinfo=timezone.utc).timestamp())
def seconds(iso):
    d = datetime.fromisoformat(iso.replace('Z','+00:00'))
    if d.tzinfo is None: raise ValueError('timezone required')
    return int(d.astimezone(timezone.utc).timestamp()) - BASE
 
def batch(events):
    return Counter((user, (t//60)*60) for user,t in events)
 
class Incremental:
    def __init__(self, delay=30):
        if delay<0: raise ValueError('negative delay')
        self.delay, self.maximum = delay, None
        self.state, self.output, self.late = Counter(), {}, []
    @property
    def watermark(self):
        return float('-inf') if self.maximum is None else self.maximum-self.delay
    def accept(self, event):
        user,t = event
        start = (t//60)*60
        # This model accepts old timestamps while their window is still open.
        # It does not reopen a finalized window.
        if start+60 <= self.watermark:
            self.late.append(event)
            return
        self.state[(user,start)] += 1
        self.maximum = t if self.maximum is None else max(self.maximum,t)
        for key in sorted(list(self.state)):
            if key[1]+60 <= self.watermark:
                self.output[key] = self.state.pop(key)
    def snapshot(self):
        return copy.deepcopy(self)
 
assert seconds('2024-08-10T00:00:10Z') == seconds('2024-08-10T05:30:10+05:30') == 10
try: seconds('2024-08-10T00:00:10')
except ValueError: pass
else: raise AssertionError('naive time accepted')
events = [('u1',10),('u1',65),('u1',20),('u1',100),('u1',30)]
all_counts = batch(events)
s = Incremental()
for e in events: s.accept(e)
assert all_counts == {('u1',0):3,('u1',60):2}
assert s.output == {('u1',0):2}
assert s.state == {('u1',60):2}
assert s.late == [('u1',30)]
# End-of-fixture is not end-of-stream: the minute-60 window stays open.
assert s.watermark == 70
# Snapshot includes counts AND progress; resume at the next input position.
prefix = Incremental()
for e in events[:2]: prefix.accept(e)
restored = prefix.snapshot()
for e in events[2:]: restored.accept(e)
assert (restored.output,restored.state,restored.late)==(s.output,s.state,s.late)
# Crash without checkpoint loses u1@10; replaying only the suffix cannot restore it.
lost = Incremental()
for e in events[2:]: lost.accept(e)
assert lost.output != s.output
# Input replay without state/offset coordination duplicates effects.
dup = Incremental()
for e in events[:2] + events[:2]: dup.accept(e)
assert sum(dup.state.values()) == 4
# Wider delay retains this late fixture; it is not a tenfold promise.
wide = Incremental(delay=120)
for e in events: wide.accept(e)
assert not wide.late and wide.state == all_counts
assert 20+60+15+5 == 100 and 180+1+5 > 120
assert 2+6000*.001 == 8 and 120-100 == 20
print({'batch':sorted(all_counts.items()),'finalized':sorted(s.output.items()),'open':sorted(s.state.items()),'late':s.late,'watermark':s.watermark,'budget_seconds':100})

What each block does and the restart exercise answer

UTC normalization prevents equivalent instants in different offsets from creating different buckets; naive timestamps fail explicitly. Batch waits for end-of-fixture and returns 3 and 2. Incremental processing observes 10,65,20 before 100 advances the watermark to 70; minute [0,60) finalizes at count 2. The later timestamp 30 targets a finalized window and is recorded as late. Timestamp 20 was accepted while its window remained open even though event ordering regressed.

Snapshot/restart preserves the pending minute and progress, while resuming input after the corresponding snapshot position. The deliberate lost-state run produces a different answer. This is the answer to “kill the streaming process and restart”: memory alone loses unfinished windows; checkpoints help only when input is replayable and state/offset/sink protocols agree. A local deep copy is not a durable production checkpoint.

A finite local stream can be driven from JSONL with the same (user_id,ts) fields as the original socket exercise, but EOF in a test must not be mistaken for a universal streaming completion signal. Persist late records for correction, and use bounded validation/admission rules for malformed or implausibly future events. The original socket loop also needed cleanup, size bounds and timezone handling; networking is not needed to teach the state failure.

A recovery cut includes the input position

The deep-copy exercise resumes after two inputs by an explicit slice; it does not store that position in Incremental. A production checkpoint needs one consistent recovery cut: source positions, operator state and a sink publication protocol. Persisting a count without its input position is ambiguous: replay the prefix and double-count, or skip it with stale state and lose events. Test every cut, not only one convenient restart.

This companion runs after the preceding Python block in the same interpreter. It executes a finite replay policy, not a durable checkpoint implementation:

for cut in range(len(events)+1):
    running = Incremental()
    for event in events[:cut]: running.accept(event)
    checkpoint = (cut, running.snapshot())
    next_position, recovered = checkpoint
    for event in events[next_position:]: recovered.accept(event)
    assert (recovered.output, recovered.state, recovered.late, recovered.watermark) == (s.output, s.state, s.late, s.watermark)
# A quiet source has not announced event-time completion.
quiet = Incremental()
quiet.accept(('u', 10))
assert quiet.watermark == -20 and not quiet.output
# One far-future timestamp can close ordinary windows in this model.
quiet.accept(('u', 10000))
quiet.accept(('u', 20))
assert quiet.late == [('u', 20)]
print({'consistent_recovery_cuts':len(events)+1, 'future_timestamp_requires_validation':True})

Validate timestamps before admitting them to progress calculation; validation needs an application clock-skew/age policy, not a hard-coded universal bound. The example intentionally demonstrates the failure without pretending its class implements that policy. Its late list is unbounded across an infinite run: route rejected records to a separately capacity-managed correction store in a real design. A constant delay also does not bound the number of distinct keys or the event rate.

Production reality: three failure analyses

Analytics on a streaming platform: a daily report may prefer a fixed snapshot and cheap replay while a user-facing feature needs immediate increments. One retained source can serve both. The original LinkedIn anecdote about multi-hour restarts and a named internal slogan is not established by the cited page; this workload comparison does not rely on it.

Commit ambiguity: a sink commits, then the worker fails before checkpoint completion. On replay, append can duplicate the effect; deterministic replacement or a transactionally recorded batch identity can make it harmless. No invented Netflix/Iceberg latency relaxation or universal “exactly-once + latency + throughput pick two” is needed. Connector-specific commit protocols, retention and repair determine guarantees.

Per-event action: replaying a pricing/payment input can call an external API twice even if Kafka output is transactional. Use an effect identity that the receiver actually honors, or a transactional outbox and deduplicating receiver. Kafka transactions do not roll back arbitrary external effects. Cost and correctness depend on the boundary; idempotence is not inherently cheaper than every transaction protocol.

Where this connects next

Spark demonstrates batch execution and micro-batch support; Kafka provides partitioned replay under retention; stream processing develops window/trigger/sink details; Airflow coordinates finite runs and reconciliations. CDC exposes database changes but still needs snapshot/offset consistency. Distributed workflow steps can be event-driven without being interchangeable with windowed stream aggregation.

Recall and worked answers

1. Three questions? How fresh, which bounded input contract, and what cost of late/wrong output—including repair obligations?
2. Bounded/unbounded? A finite input with a known end versus an ongoing source with no known end; a bounded snapshot need not contain all business events for a date.
3. Watermark? Event-time progress under a stated policy used by stateful operators; not a wall-clock guarantee that old events cannot arrive.
4. Allowed lateness? In Flink windows, a post-firing retention/update policy; in Spark, use its distinct watermark/output-mode contract. More retention can cost state and delay final append results, not necessarily provisional updates.
5. Why micro-batch? Under the numerical assumptions above, a 60s trigger meets 120s freshness at acceptable overhead. The answer is conditional, not a claim that all such systems are tenfold cheaper.
6. Lambda/Kappa/lakehouse? Two coordinated views; one replayable computation path; transactional table storage with compatible batch/incremental readers. None eliminates correctness work.
7. Four pitfalls? Late data, out-of-order semantics, growing backlog and duplicate effects at recovery boundaries.
8. Why not exactly-once always on? A replayable source, recoverable processing state and cooperating sink/effect protocol are required. A file source need not have a Kafka transactional producer; a transactional Kafka input needs read-committed visibility if aborted records must be hidden.
9. Why retain batch analytics in a streaming organization? Fixed snapshots and backfills can simplify a particular reporting contract; that is a design argument, not a verified account of LinkedIn's internal workload decisions.

Dashboard failure answer: two-minute freshness with 5% of events an hour late means a provisional view and a separately stated correction horizon. Publish coverage/late-event metrics; never label the early count complete.

Explain out loud: distinguish input semantics from execution granularity, calculate the 100s budget, and trace why the 30-second timestamp is late in the exact model.

Sources and next step

Primary documents checked 2026-10-07: Apache Beam programming guide, Spark 3.5.7 Structured Streaming, Flink 1.18 windows, Flink fault tolerance. Exact passages/hashes and the Dataflow primary publication are in the evidence.

Next: Spark execution. Only the bounded CPU model ran here; the source-supported engine concepts do not imply cluster execution or performance measurements.

Original resource trail and reading purpose

The original optional pre-read destinations are retained below, not used as primary evidence for numerical claims, vendor incidents, or current API defaults. Video identity, duration, attribution and playback have not been independently verified here; search-result links are discovery links, not a particular verified lecture. Mutable latest documentation may differ from this lesson's pinned release. Use the versioned primary sources above for the governing contract.

• Data Pipeline Overview — original optional video/discovery destination; historical label, not verified attribution.
• Foundations of streaming SQL by Tyler Akidau — original optional video/discovery destination; historical label, not verified attribution.
• Kafka in 100 Seconds — original optional video/discovery destination; historical label, not verified attribution.
• Streaming 101 — The World Beyond Batch — original optional reading destination; historical label, not verified attribution.
• LinkedIn — the ‘Streaming Everything’ decade — original optional reading destination; historical label, not verified attribution.
• What do you mean by ‘Event-Driven’? — original optional reading destination; historical label, not verified attribution.

Read with a task: separate boundedness, event-time progress and output refinement; distinguish event-driven messaging from a stateful stream computation.

Series navigation: previous lesson; learning-plan hub. The original series used a 90-minute intuition → visual → hands-on → production → recall structure; that historical pacing is not a measured completion time for this revised lesson.