Designing for Scale · Real-Time Analytics
Provisional and final aggregates must agree on ownership. A source-bounded lesson with worked reasoning, failure analysis and explicit runtime limitations.
Original publication date unrecordedUpdated
Lesson preparation & details
Level: advanced
By the end, you should be able to
- Provisional and final aggregates must agree on ownership
- A nightly batch replaces yesterday's total while the streaming layer still exposes yesterday's late updates. What happens if the API adds both?
Bring with you
Sources checked: . A source check does not change the publication or update date.
Editorial review: · What review means
In this article · 13 sections
The problem this post solves
A dashboard that updates within seconds needs an explicit freshness definition: seconds since event occurrence, ingestion or processing? It also needs a correctness definition: which events are eligible, which duplicates count once, what corrections/deletes mean, and when a displayed number is provisional versus final.
Analytical scans can compete with transactional work, but neither raw-event ingestion nor querying is mathematically impossible at a universal row rate. An analytical engine may serve raw data directly; a small relational workload may use appropriate indexes. Pre-aggregation, replicas or a separate store are choices justified by measured query shapes and resource budgets. A nightly job does not make every result exactly 24 hours stale: age varies with schedule, data arrival, job duration and failures.
First principles
Aggregation replaces detailed events with statistics at a declared grain, trading query work against lost detail and maintained state. If 10,000 events belong to one minute and one key, a count result may be one row rather than 10,000. If they belong to 10,000 distinct keys, there may still be 10,000 results. A single integer per counter does not imply constant total memory independent of key cardinality, open windows or deduplication history.
For a time-window aggregate, state depends on active keys, overlapping windows, aggregation payload, retained event identity and lateness policy. A five-minute window sliding every minute can place one event into up to five windows. Distinct counts, percentiles and joins have different state/approximation requirements from a simple sum. Define the semantics before claiming an order-of-magnitude reduction.
Architecture: The Lambda Architecture
Lambda separates a recomputed historical/batch view and a lower-latency speed view, with a serving layer reconciling them. It is one architecture, not a universal modern standard. A unified replayable computation can reduce duplicated business logic; a warehouse with incremental ingestion/materialized views can be another choice. None is accurate merely because it is called batch, streaming or exactly-once.
Figure text equivalent — labels and relationships
- Node “Identified source events” (square).
- Node “Replayable acknowledged intake” (square).
- Node “Retained raw data with schema and access policy” (square).
- Node “Versioned batch correction” (square).
- Node “Stateful incremental computation” (square).
- Node “Historical aggregate version” (square).
- Node “Provisional aggregate version” (square).
- Node “Explicit range or version ownership” (square).
- Node “Analytics API with freshness metadata” (square).
- “Identified source events” leads to “Replayable acknowledged intake”. Line: normal; ends: arrow point.
- “Replayable acknowledged intake” leads to “Retained raw data with schema and access policy”. Line: normal; ends: arrow point.
- “Retained raw data with schema and access policy” leads to “Versioned batch correction”. Line: normal; ends: arrow point.
- “Replayable acknowledged intake” leads to “Stateful incremental computation”. Line: normal; ends: arrow point.
- “Versioned batch correction” leads to “Historical aggregate version”. Line: normal; ends: arrow point.
- “Stateful incremental computation” leads to “Provisional aggregate version”. Line: normal; ends: arrow point.
- “Historical aggregate version” leads to “Explicit range or version ownership”. Line: normal; ends: arrow point.
- “Provisional aggregate version” leads to “Explicit range or version ownership”. Line: normal; ends: arrow point.
- “Explicit range or version ownership” leads to “Analytics API with freshness metadata”. Line: normal; ends: arrow point.
The Ingestion Layer (Kafka)
Choose stable event IDs, schema/version and acknowledgment semantics. Fire-and-forget can lose events and is not the default recommendation for financially significant measurements. Kafka's partitioned log supports replay, but producer acknowledgment, replication, retention, compression, payload size and consumer capacity affect the system. It performs networking, replication and other work, not “almost no processing,” and the lesson claims no millions-of- writes-per-second benchmark.
Monitor source-to-broker and broker-to-consumer lag separately, in elapsed time and work/bytes as well as record count. A connector or consumer must explicitly export retained raw data to long-term storage; Kafka does not automatically dump everything into S3. Configure retention so slow consumers/replays can still reach required input, and reconcile source records against the intake contract rather than assuming absence means no event occurred.
The Batch Layer (The Source of Truth)
Retained immutable/raw versions can support reprocessing after a logic bug, schema change or late arrival, subject to privacy and retention policy. A batch aggregate is not automatically perfect: duplicated input, missing records, wrong joins or inconsistent definitions all remain possible. Identify the source snapshot/offset coverage, query/code version, dimensions and correction horizon associated with every published result.
For a rebuild, write a new result version, validate reconciliation checks, then publish it atomically at the chosen table/manifest boundary. Do not expose half a partition as final. Raw observations may be the audit input, but the business source of truth might still be an order/payment ledger; a click log cannot prove completed sales without a defined relationship.
The Speed Layer (The Real-Time View)
A streaming engine maintains incremental state and periodically commits recoverable progress according to its protocol. Spark 3.5.7 documents checkpointing, replayable sources, state stores and sink/output-mode constraints. Such engines can persist/spill/checkpoint state; they do not necessarily keep only an unprotected in-memory tally or process every event the millisecond it arrives. Backpressure, scheduling and data dependencies affect lag.
Specify whether the sink receives full snapshots, changed aggregate rows, append-only final rows or deltas. If a minute count changes from 10 to 12, an update-mode row may mean “replace this key's value with 12,” not “add 12.” An idempotent sink can apply a versioned value again without increasing the count. External side effects require their own cooperation; a framework checkpoint cannot include an arbitrary uncoordinated database increment by implication.
The Serving Layer
For a historical-plus-current dashboard, define nonoverlapping ownership or a versioned base-plus-delta protocol. “First 29 days from batch, today from speed” is a starting partition, not enough when late events revise yesterday. The response should expose coverage/freshness and whether particular windows remain provisional.
Example: batch version B includes 200 yesterday-events, including five late arrivals. Speed still exposes those same five yesterday-events plus 50 events today. Summing every visible number produces 255, although the correct total for that input set is 250. Either let B own the completed historical window and speed own today, or use a known source-position cutoff so a delta contains only events absent from B. Later historical corrections must explicitly replace a version or contribute an identified change; do not silently overlap both paths.
Handling Time
Event time describes occurrence in the event's domain. Ingestion time is when the pipeline receives it. Processing time is the operator's current time when it processes it; those latter two need not be identical. Validate device clocks and retain enough timing metadata for the intended measurement. A five-click event from 12:01 arriving at 12:05 belongs to the 12:01 activity window for event-time behavior analysis, but a 12:05 processing/arrival metric may correctly measure intake load. Calling processing-time analytics a “lie” confuses two different questions.
Watermarks and late data
A watermark is progress information used by the engine under a lateness policy; it is not physical proof that no older event can ever arrive. Spark 3.5.7 says a two-hour delay guarantees that data less than two hours behind the latest processed event time is not dropped by that watermark rule, while older data may or may not be aggregated. It is a one-sided guarantee, not “everything late is always dropped” or “all late data always accepted.” Read the applicable aggregation/output-mode conditions with that statement.
In that version, aggregation state cleanup requires the relevant event-time column/window,
withWatermark before aggregation on the same timestamp column, and a compatible Append or
Update mode. Complete mode preserves the aggregate table and cannot use watermarking to drop
that intermediate state. Multiple-input watermark policies and idle/delayed inputs can change
progress; examine the configured operator rather than treating a single timestamp as a universal
bound on all state.
Choose what happens beyond the online correction horizon: drop with a measured counter, route to a late-data/reconciliation path, or recompute a historical version. More retained state can accept more lateness but costs memory/storage and delays finalization. Some use cases prefer a provisional low-latency number followed by an explicit revision; others need a later reconciled answer. The product determines the trade-off.
Failure modes
- Dual writes lose one side: persist source intent/outbox with the business transaction, or derive change events through a properly configured CDC/log path. A producer that independently updates a database and sends to Kafka needs a recovery protocol for partial success.
- Replay increments twice: stable source identity, deterministic transformations and a sink transaction/version rule must make reprocessing safe. Kafka transactions do not automatically cover every external serving store.
- Deletes/corrections are ignored: an append-only sum of old facts can overstate today's valid total. Model reversals/upserts/retractions according to source semantics and reconcile.
- Unbounded state: high-cardinality keys or incompatible output-mode/watermark configuration can defeat a simplistic “one integer” memory estimate. Observe state rows/bytes and checkpoint duration as well as processing throughput.
- A dashboard scans the raw log on every request: this is usually the wrong indexed-serving interface, but Kafka is not logically incapable of processing queries through derived streaming views. Maintain a queryable view and state its freshness instead of claiming every log scan instantly fails at any size.
- Batch and stream diverge: share definitions where practical and compare both on the same source snapshot/window. A nightly overwrite does not prove the batch result is right.
Worked correction exercise
The batch result for yesterday is 200 at source cutoff 10,000. A stream replay emits an aggregate
row yesterday=200 twice, then a correction produces version 2 with yesterday=198. Today has
50 accepted events. What should the dashboard show, and what evidence is needed?
Versioned replacement, not blind addition
Under a replace-by-window/version contract, yesterday ends at 198 and today contributes 50, so total is 248. The duplicate 200 row changes nothing, and an older version arriving after version 2 must not roll the count backward. Retain the source cutoff, version identity and correction meaning; if the rows were deltas instead, the calculation would be different. The response can state which windows are final/provisional and its source coverage. The number 248 follows this synthetic input/contract, not a claim that an arbitrary stream engine emitted those exact rows or recovered a real failure.
What to carry forward
Freshness, event identity, state retention and serving ownership are independent contracts. Pre-aggregation is useful when its grain matches the queries and its maintained state fits the budget. Batch and streaming both need correctness checks. Versioned documentation supports this conceptual review; no Kafka/Spark/Flink cluster, streaming benchmark or cloud dashboard was executed.
Sources and review boundary
Checked 2026-10-07: Spark 3.5.7 Structured Streaming for watermark asymmetry, state stores, checkpointing and output modes; Kafka 4.1 design for replay and external-effect cooperation. Byte/row/window and serving-version calculations are bounded local reasoning tests, not measurements from either project's performance claims.
Pause / Recall / Apply
Can you explain it without the page?
Close the example. Reconstruct the core idea, then change one assumption. Mark complete when you’re ready; you can always undo it.
Stored in this browser only. No account, no sync. Clearing browser data removes your record.