Replay is a feature; duplicate effects are a design choice

Replay is a feature; duplicate effects are a design choice

A usage event is not a bill. It is evidence from which a consumer may update a ledger. If the consumer writes the ledger and crashes before recording its progress, it cannot safely assume the write failed. Restarting from the previous checkpoint may repeat the effect. This is not solved by putting “exactly once” on the queue box.

Kafka's design documentation separates publishing durability from consumption semantics. It describes output-before-offset as replay-prone and offset-before-output as loss-prone under a crash. Transactions can coordinate Kafka output records and consumed offsets within Kafka. An external database, payment provider or email API does not automatically join that transaction.

The mental model

Kafka is a durable, partitioned log with replication and configurable retention—not an unlimited archive and not a guarantee that every downstream effect happens once. A topic names a family of partition logs; each partition has its own ordered offsets. A producer chooses a partition according to its configured partitioner, explicit partition or key policy. Consumers fetch records and commit progress; committing an offset does not delete records for other groups.

Follow one exposure event. All three conventional groups can fetch it independently. Within a group, each assigned partition has one consumer owner at a time; adding consumers beyond the partition count does not create more active partition owners. This describes ordinary consumer groups, not Kafka 4.1 share groups. A paused training export does not move the feature consumer's offset, but both still compete for broker/network resources and both are limited by retention.

Partition keys: get this right

Keying by user_id with a stable partitioner and partition count puts that user's records together. Partition order is append order, not the order in which the human acted: independent producers, clock skew and delayed upload can put a click after a later event. Use event IDs for deduplication and event-time/watermark rules for temporal features. Record a source sequence where the source provides one; detect missing values rather than sorting timestamps and calling that causality.

Adding partitions changes many hash-to-partition mappings. Old records stay in the old partitions while new records can arrive elsewhere; consuming both concurrently can interleave a key's history. A migration can pause writers, drain the old mapping to an agreed boundary, switch the versioned mapping, and resume. Alternatively create a new topic with an explicit cutover and replay policy. Neither strategy is automatically implemented by an increase-partitions command.

A highly active user is a hot key. Salting it allows parallelism but gives up the one-partition order that the key was chosen to provide; recombining counts is easier than recombining a strict state machine. Choose the invariant before changing the key.

Consumer groups in one line

Consumers in the same ordinary group divide partitions; separate groups have separate checkpoints. Commit the next offset after the corresponding work is safe. With asynchronous processing, finishing offset 12 while offset 11 is unfinished does not justify committing 13: restart would skip 11. Track the largest contiguous completed prefix per partition. On revocation, stop admitting work for the old assignment and fence any stale workers; a rebalance is not a sink transaction.

Things that will bite you

• Monitor lag in records and event age, processing/error rate and partition skew. An increasing backlog over a sustained interval means throughput is below ingress; it does not prove permanent failure or data loss by itself. Compaction/transactions mean offsets are not always contiguous visible records.
• Retention is a deadline for replay. Size/time retention, compaction and topic policy determine what remains; committing offsets is not a reservation on data. Maintain a separately retained raw archive where training/recovery needs a longer horizon.
• enable.auto.commit=false is useful when the application explicitly manages commits after sink success; it is not a magic safety switch. Automatic commits can be safe under a suitable synchronous processing lifecycle, while careless manual commits can lose work.
• Producer acks=all waits for in-sync replicas according to broker settings, not every configured replica forever. min.insync.replicas, replication factor and election policy define failure availability and durability. Idempotent production handles specified producer retries; consumer replays are another boundary.

The minimum useful diagram

The hot/warm/cold diagram above is one topic, three groups and three freshness contracts. A concrete reference procedure, not executed here, is: define schemas and stable exposure/event identities; choose partition keys from ordering requirements; provision replication/retention for the failure budget; configure producer idempotence and acknowledgements; deploy independent groups with explicit sink/checkpoint behavior; observe lag and errors per partition; inject a consumer crash after sink commit in a disposable test environment; check replay and business totals; then test broker loss and a rebalance before production approval. Training exports additionally need point-in-time feature availability and mature labels—Kafka order alone does not prevent leakage.

For a concrete capacity example, a ten-minute outage creates 60,000 records at 100/s. At 150/s processing the backlog drains at only 50/s while new events arrive, so recovery takes 1,200 seconds. At 100/s it never catches up under constant ingress; below 100/s it grows. Allow retention for the oldest unprocessed record, outage, recovery and a safety margin. This is arithmetic, not broker performance measurement.

Partition and recovery exercises — worked answers

1. Six partitions, eight consumers, three groups: each group has at most six active partition owners, with two idle members in a stable assignment; all three groups can read the same retained records. Broker work is not isolated just because offsets are.
2. Offset 11 pending, 12 finished: commit no later than 11 until 11 succeeds (a committed value of 11 resumes at 11). Persist sink identities so retrying 12 is harmless. Do not use the largest finished offset as a checkpoint.
3. Add partitions during an ordered user-state replay: future user records can move. Version and coordinate the mapping/cutover, or explicitly relax ordering; adding consumers alone cannot repair split history.
4. Training consumer lags beyond retention: resetting its offset cannot recover deleted records. Replay from the governed archive, rebuild the derived dataset with a reproducible version, and expose the gap until reconstruction is complete.

Source boundary: Kafka 4.1 design and producer configuration support these broker and producer distinctions. Documentation was checked on 8 October 2026; no Kafka service was run. The exact crash-trace program later in this lesson remains unchanged and locally executable.

Follow one partition

This is a conceptual trace, not a Kafka client implementation. Offsets below are positions in a three-record list. A checkpoint stores the next position to process. Independent consumer groups have their own progress; consumers in one group divide partition ownership. Ordering is within a partition, not a global timestamp order across all partitions.

The second application is harmless only if the sink defines a repeated identity as the same operation. Replacing a value with the same value is idempotent; adding five again is not.

Reproduce the failure

The seen set is intentionally in the simulated sink, not in the worker. It survives the worker restart in this small fixture. It does not survive a process or machine failure, so it is not the persistence solution; the next lesson replaces it with a transaction.

events = [("e1", 5), ("e2", 7), ("e3", 2)]
 
def simulate(deduplicate):
    checkpoint = 0
    total, seen = 0, set()
    attempts = []
    crashed = False
    while checkpoint < len(events):
        event_id, units = events[checkpoint]
        attempts.append(event_id)
        if not deduplicate or event_id not in seen:
            total += units
            seen.add(event_id)
        if not crashed:
            crashed = True  # output persisted, checkpoint not advanced
            continue
        checkpoint += 1
    return total, attempts
 
naive, attempts = simulate(False)
safe, _ = simulate(True)
assert attempts == ["e1", "e1", "e2", "e3"]
assert naive == 19 and safe == 14
print(f"naive={naive}; deduplicated={safe}; attempts={attempts}")

The five-unit difference is the repeated first event, not a floating-point bug or a failure of partition ordering. Committing the checkpoint before the sink update would exchange this duplicate risk for a lost-effect risk. An acknowledgment from the broker also says nothing about whether a later consumer applied the intended business effect.

Choose the identity at the business boundary

A producer retry should preserve the business operation's ID. A worker-generated random ID on every attempt defeats deduplication. Conversely, two genuinely distinct purchases must not reuse one ID merely because their amounts match. Scope the key by tenant and operation type. If the same ID arrives with a different payload, surface the conflict rather than silently accepting it as a duplicate.

Partitioning by a stable user key can keep that user's records together under a stable partitioning arrangement, but this is not a guarantee of real-world event-time order. Multiple producers, late arrivals and changes to the partition mapping still require an explicit application ordering policy. Do not add partitions and assume all historical keys retain the same mapping.

Capacity and retention

If ingress is 100 events/second and a stopped consumer is down for ten minutes, it accumulates about 60,000 events. At 150 processed events/second after restart, net recovery is 50/second, so catch-up takes twenty minutes. Retention must cover the outage and recovery margin; consumer independence does not mean indefinite replay after records have expired.

Exercises

1. In the fixture, what result follows if the crash happens after processing e2 but before its checkpoint?
2. Why doesn't an idempotent Kafka producer alone prevent two SQL increments by a consumer?
3. Where must the durable deduplication fact live relative to the aggregate update?

Limits

This fixture models one partition and one worker. It does not test Kafka brokers, rebalances, retention, producer fencing, replication configuration or disk failure. Use the pinned Kafka 4.1 design page for the semantics being discussed; do not infer that its historical performance examples describe your present hardware.

Sources and verification boundary

Primary sources checked on 7 October 2026:

• Apache Kafka 4.1 design: delivery semantics — Output-before-offset can replay; offset-before-output can lose effects; external destinations need cooperation for exactly-once effects.

The numerical fixtures and Python programs are teaching examples, not measurements of a deployed service. Python blocks were executed locally; no cloud deployment or model inference was performed.

Previous: Token buckets: bursts, clocks and the atomic boundary. Track map.