Kafka — Topics, Partitions, Consumer Groups

Why this session exists

Kafka is a partitioned retained log with independent readers. Treating offset commits as deletion, replication as invulnerability, or a transaction flag as protection for every downstream effect causes design errors. This lesson targets Kafka 3.7 conventional consumer groups, not newer share-group consumption.

You will explain topics/partitions/offsets/groups, choose ordering keys and partition capacity, interpret lag, understand rebalances and trace a failure across producer, broker, processing and sink boundaries. Kafka is one event-streaming option, not the implementation behind every event-driven system.

Prerequisites

NoSQL access patterns explains locality; batch versus streaming explains replay and freshness. Neither implies that every workload should use Kafka.

A shared diary with retention and bookmarks

A topic names a stream such as orders. Its partitions are separate ordered logs, stored as log segments and indexes rather than one eternal file. Producers append records; brokers replicate partitions; consumer groups keep positions. Retention and compaction can remove records independently of readers, so “immutable forever” is not the contract.

An offset is an increasing position within a partition, not a globally unique application event ID. Gaps can remain after compaction and transaction filtering. A consumer commits the next position to read: after processing offset 12345, commit 12346. A new group can read retained history independently; committing in billing does not advance analytics.

The normal keyed Java producer routing uses serialized key bytes and partition metadata. Stable key bytes, partition count and partitioner can keep one entity's records together. Null-key routing can be sticky/adaptive for batching and load, not guaranteed round-robin or perfect balance. Applications can explicitly choose partitions. If different serializers encode the same logical account differently, “same account” does not imply the same key bytes.

Ordering and its derivation

A partition leader establishes one append order. A total order across independent partitions would require another coordination/merge mechanism; Kafka does not provide that topic-wide order. A single-partition topic gives one log order with a throughput/parallelism tradeoff; a sequencer can itself be replicated, so total ordering does not logically require an unreplicated single point of failure.

Per-key routing is useful when account updates require order, but append order is not automatically event-time or business causality. Two independent producers can race. Without idempotence, retry settings can reorder batches; with asynchronous consumer workers, completion order can differ from fetch order. Preserve per-key processing or use sequence/version checks and commit only a contiguous completed prefix. Kafka transactions can cover multiple partitions, so the partition key is not an absolute transaction boundary.

Historical context

Kafka originated at LinkedIn. Idempotent production and transactions arrived with Kafka 0.11. KRaft replaces ZooKeeper-based metadata management with a controller quorum and metadata log; this lesson uses the 3.7 KRaft-era model. KRaft controllers can be dedicated rather than identical to data brokers. Migration, quorum sizing and failure domains remain operational work. There is no substantiated “deployment halves” or linear-metadata-scaling promise, and the old LinkedIn migration/outage attribution is not established here.

Kafka cluster anatomy

Replication factor three means three copies on assigned brokers, not necessarily three in-sync copies at every instant. ISR membership reflects caught-up replicas under Kafka's criteria. Leadership, replica placement and group ownership are distinct: a group consumer need not run on the broker holding its partition.

Producer to partition to consumer

The diagram separates produce acknowledgment from consumer progress and uses a group coordinator for commits rather than treating the data leader as every role. Followers fetch replicated data; the drawing is conceptual, not a wire-protocol transcript.

Acknowledgments and durability

acks=0 does not wait for a broker acknowledgment; acks=1 waits for the leader and can lose an acknowledged record if it fails before replication; acks=all waits for the current ISR under the protocol. With min.insync.replicas=2 and RF=3, writes requiring all acknowledgments fail when the in-sync set is too small. With min ISR=1, an ISR containing only the leader can satisfy all acknowledgments. Replica count, leader-election policy, correlated loss and retention remain relevant; “acks=all means no data loss” is false [Kafka producer/topic].

A Java producer properties example for a robust starting point, not maximum durability under all failures, is:

acks=all
enable.idempotence=true
retries=2147483647
max.in.flight.requests.per.connection=5
delivery.timeout.ms=120000

Idempotence requires compatible acks/retries/in-flight settings and deduplicates supported producer retries; it does not deduplicate two independently submitted business events with the same payload. Delivery timeout can terminate attempts despite a very high retries count, and a timeout can leave the caller uncertain whether an effect occurred. Check errors and use stable application identity when resolving ambiguity. The topic's min ISR and RF must be configured separately; RF=1 cannot demonstrate resilience.

Partition count, key choice and lag

A conventional group assigns a partition to at most one active member at a time. Extra members beyond assigned partitions are idle; one member can own several. This caps independent partition ownership, not every internal thread. More partitions increase parallelism potential, metadata, open segments and recovery/assignment overhead; one broker's six partitions do not create six machines' resources.

Capacity example (assumed measured rates): desired 60 MB/s with 8 MB/s producer-path capacity and 5 MB/s consumer-path capacity per partition needs at least ceil(max(60/8,60/5)) = 12 partitions before headroom and skew. Apply these rates only if measured for the actual record size, replication, compression and consumer work. RF is not a required divisor of partition count; there is no universal “peak consumers times two” or 4,000-partitions-per-broker ceiling valid for every release/hardware.

Increasing partitions does not move existing records, but can change where future records of an existing key hash. Old and new partitions can then be consumed concurrently and lose application order across the transition. Kafka does not support shrinking a topic's partition count in place; migrating to a new topic needs an ordering/cutover plan.

Choose entity keys for required entity order, unkeyed routing for independent events, or a composite/salted key when a heavy entity can safely be divided into smaller ordering domains. A salt does not preserve the old whole-entity order. Unkeyed routing still retains per-partition log order, not “no ordering whatsoever.”

Lag is often log-end-offset - committed-offset per partition. It is an offset distance, not necessarily the number of visible unprocessed business records; it can include gaps or processed-but-not-committed work. Read-committed consumption is constrained by transaction visibility/last stable offset. Monitor lag trend, oldest event age, processing rate and retention headroom, not just total lag. Backlog 6,000 with arrival 100/s and processing 150/s drains in 120s under steady assumptions; processing 90/s never catches up while arrival remains 100/s.

Rebalance and replay

Membership/subscription changes, missed heartbeats and poll-timeout violations can cause reassignment. Eager assignment can revoke all partitions; cooperative protocols reduce disruption through incremental revocation but do not eliminate pauses, duplicate effects or the need for a compatible assignor across members. Group behavior depends on client/protocol/version, not a fixed ten-second delay.

The new owner resumes from committed positions. If the old owner completed an external write but not its offset commit, the record can replay. If it committed first and then failed, the effect can be missing. Revoke handling must stop/finish or fence outstanding work and commit only safe progress. A rebalance is not automatic rollback of in-flight external requests.

Hands-on: bookmarks, assignment, lag and an idempotent sink

This bounded Python 3/SQLite model executes independent groups, keyed repartitioning and both crash boundaries. The routing function is deliberately not Kafka's Murmur2 implementation. Its purpose is to exhibit modulo-partition changes, not predict a real key's Kafka partition.

import sqlite3
from collections import defaultdict
from math import ceil
 
def partition(key, count):
    return sum(key.encode('utf8')) % count  # teaching hash, NOT Kafka Murmur2
logs = defaultdict(list)
for i in range(20):
    key = str(i%5+1)
    p = partition(key,6)
    logs[p].append((f'e{i}',key,i))
bookmarks = {g:{p:0 for p in range(6)} for g in ['billing','analytics']}
for p,records in logs.items(): bookmarks['billing'][p]=len(records)
assert sum(bookmarks['analytics'].values())==0
assert sum(bookmarks['billing'].values())==20
assert sum(map(len,logs.values()))==20  # reading/committing did not delete
assert any(partition(str(k),6)!=partition(str(k),7) for k in range(1,100))
 
def assign(partitions, members):
    if not members: return {}
    return {p:members[p%len(members)] for p in range(partitions)}
a = assign(6,['A','B']); b = assign(6,['A'])
assert list(a.values()).count('A')==3 and set(b.values())=={'A'}
assert len(set(assign(6,list('ABCDEFG')).values()))==6
 
c = sqlite3.connect(':memory:')
c.executescript('CREATE TABLE seen(event_id TEXT PRIMARY KEY, amount INT NOT NULL); CREATE TABLE balance(id INT PRIMARY KEY, n INT); INSERT INTO balance VALUES(1,0);')
def effect(event_id, amount, crash_before_commit=False):
    with c:
        prior = c.execute('SELECT amount FROM seen WHERE event_id=?',(event_id,)).fetchone()
        if prior:
            if prior[0]!=amount: raise ValueError('conflicting event identity')
            return
        c.execute('INSERT INTO seen VALUES(?,?)',(event_id,amount))
        c.execute('UPDATE balance SET n=n+? WHERE id=1',(amount,))
        if crash_before_commit: raise RuntimeError('crash before commit')
try: effect('e0',5,True)
except RuntimeError: pass
assert c.execute('SELECT n FROM balance').fetchone()[0]==0
assert c.execute('SELECT count(*) FROM seen').fetchone()[0]==0
# Effect commits, then crash before Kafka offset commit: retry is harmless.
effect('e0',5); effect('e0',5)
assert c.execute('SELECT n FROM balance').fetchone()[0]==5
try: effect('e0',6)
except ValueError: pass
else: raise AssertionError('identity conflict accepted')
# Negative controls: naive additive replay duplicates; commit-first loses effect.
naive = 0
for _ in range(2): naive += 5
assert naive==10
committed_offset, external_effect = 1, 0
assert committed_offset==1 and external_effect==0
# Asynchronous completion must not commit past unfinished offset 0.
completed={1,2}; next_safe=0
while next_safe in completed: next_safe+=1
assert next_safe==0
assert ceil(max(60/8,60/5))==12 and 6000/(150-100)==120
print({'records_retained':sum(map(len,logs.values())), 'billing_committed':20,'analytics_committed':0,'safe_effect':5,'naive_effect':naive,'partition_lower_bound':12,'catchup_seconds':120})
c.close()

Kafka 3.7 reference operations and the rebalance exercise

The original single-node Docker tour is replaced with source-reviewed operations on an already authorized disposable Kafka 3.7 broker. No broker/container was started here. A provisioner must use unique names, readiness checks and loopback-only listeners for a local exercise; do not publish controller ports, assume sleeps prove readiness, or suppress all errors with || true. RF=1 is a local learning configuration, not the durability example above.

# Run from a Kafka 3.7 distribution against the disposable local broker.
bin/kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --create --topic lesson-orders --partitions 6 --replication-factor 1
bin/kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --describe --topic lesson-orders
printf '1:order-1\n2:order-2\n1:order-3\n' | bin/kafka-console-producer.sh --bootstrap-server 127.0.0.1:9092 --topic lesson-orders --property parse.key=true --property key.separator=:
bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --topic lesson-orders --group lesson-billing --from-beginning --max-messages 3 --property print.key=true --property print.partition=true
bin/kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --describe --group lesson-billing

parse.key separates key/value; output properties print actual partitions; --describe shows group positions. --from-beginning controls where a group lacking a usable committed position starts; it does not override existing commits. Console termination/auto-commit timing must be inspected rather than assuming all three are durably committed. The production consumer needs explicit post-effect commit/error handling; the console tour is not that implementation.

For the two-group exercise, run the same bounded read with a fresh lesson-analytics group and verify retained records can be read independently. For a rebalance exercise, use two live consumers in one fresh group, inspect members/assignment while they are running, then stop one and wait for the actual assignment change. A balanced six-partition/two-member assignment commonly yields three each, but verify the assignor and subscription. The setup must remain alive during this exercise; the original script's exit trap removed its broker before later terminal instructions could work.

Replay answer: inspect inactive group offsets and retained start/end positions first, dry-run a reset on the disposable group, then apply the deliberately selected range. Never reset a live production group as a casual tutorial step. A timestamp reset cannot recover expired or compacted history.

Work through an acknowledgment ambiguity

Take RF=3, ISR={leader, follower A, follower B}, acks=all, and min ISR=2. The leader waits for the current ISR's replication acknowledgments; min ISR is an admission/durability floor, not permission to acknowledge after any two of three current ISR members. If a follower falls out of ISR, the remaining two can suffice under the protocol. With only the leader in ISR, the min-ISR=2 write fails. The caller still needs to handle timeout/error ambiguity rather than infer that a failed response means no append occurred [Kafka producer/topic].

Producer idempotence tracks producer protocol identity and sequence numbers. It does not merge two separate application sends or automatically join a new producer session to the previous session's business events. Preserve an application event ID through restarts; use a transactional ID and the documented fencing/transaction protocol where the Kafka read-process-write flow requires it. The SQLite example's seen marker and balance update commit together: retaining the marker for less time than input can replay reopens the duplicate risk.

Worked oral answer: a partition is an ordered storage/replication log and a unit of conventional consumer-group assignment. It is not one CPU core, one global transaction, or a guarantee that asynchronous worker effects finish in fetch order. If offsets 1 and 2 have finished but 0 has not, commit position 0; when 0 finishes, the contiguous prefix allows committing 3. If 4 finishes but 3 is still running, remain at 3. This is why safely processing with several workers requires a completion frontier rather than committing the largest observed completion.

# Bounded completion-frontier model; no Kafka client or broker.
frontier, finished = 0, set()
observed = []
for offset in [2, 1, 0, 4, 3]:
    finished.add(offset)
    while frontier in finished:
        finished.remove(frontier)
        frontier += 1
    observed.append(frontier)
assert observed == [0, 0, 3, 3, 5]
assert not finished
print({'safe_commit_positions': observed})

This contiguous integer fixture has no compacted/filtered offset gaps. A real consumer tracks the ordered records it was actually delivered and their next-position boundaries; it must not wait for a nonexistent offset or treat aborted/unseen records as unfinished work. Ownership revocation also needs fencing of old workers; this set algorithm does not implement the group protocol.

Exactly-once scope and production reality

Kafka transactions can atomically couple produced Kafka records and consumed group offsets in a read-process-write flow. Consumers using read_committed hide aborted records and wait for transaction visibility. This does not extend the transaction to PostgreSQL or email. The executed sink puts the dedup marker and additive balance change in one SQLite transaction, checks conflicting ID reuse, then permits offset commit. That is a modeled external cooperation boundary [Kafka design].

UPSERT is not inherently idempotent: balance = balance + amount repeats the effect. A stable event ID must survive retries/rebalances, with retention long enough for replay. A replacement value can be idempotent yet regress state if an older version overwrites a newer one; version ordering may be needed. Receiver-supported idempotency keys are required for arbitrary API actions, not merely a locally generated string.

Metadata/failover scenario: many partitions and groups can increase metadata and recovery work. Capacity test planned broker/controller failures and assignment latency rather than quoting unsupported LinkedIn five-minute-to-ten-second results. KRaft removes ZooKeeper as a separate metadata system, not all coordination costs.

Shared-cluster blast radius scenario: workload isolation, quotas, resource budgeting and separate clusters can reduce correlated incidents, at an operational cost. The original Slack outage attribution and historical cooperative-rebalance timeline were not substantiated; the resource-isolation reasoning stands without invented evidence.

Compaction scenario: compacted logs eventually remove superseded values while preserving offsets and ordering of surviving records. They are not instantly one row per key. Tombstones represent deletion and may themselves expire; mixed compact+delete policy can remove more history. Rebuilding latest state is different from replaying every business event forever.

Where this connects next

Stream processors consume retained partitions; CDC projects database changes with snapshot/offset coordination; workflows can react to events; cache invalidation consumers need replay/deletion semantics. Chat fan-out may use Kafka or another mechanism—product ubiquity is not a correctness argument. Orchestrators can coordinate bounded jobs around the log without replacing its consumer protocol.

Recall and worked answers

1. Log, not destructive queue? Reads/offset commits do not delete records; retention/compaction and administration govern availability of history.
2. How many group owners per partition? At most one active member in a conventional group at a time; internal concurrency still needs safe ordering/commits.
3. Ordering when? Within a partition's append log; stable routing plus producer and consumer ordering behavior is needed for an application entity. Not event-time total order across partitions.
4. acks=1 versus all? Leader-only acknowledgment versus ISR-based acknowledgment; min ISR, replication, elections and failure model remain part of durability.
5. Lag? Offset distance from committed position to log end, with transaction/gap/commit-batching qualifications; inspect age and retention headroom too.
6. Why not shrink? Kafka does not support in-place partition reduction; migrate/cut over explicitly. Growing changes future routing but does not redistribute old records.
7. Effectively-once? Retried delivery with a sink protocol yielding one intended effect. It is not universally cheaper or supported by every downstream service.
8. Durable producer config? The properties above plus deliberate topic RF/min ISR and election policy, error handling and failure testing; no setting means invulnerable.
9. KRaft versus ZooKeeper? Internal controller-quorum metadata log replaces the external metadata system. Dedicated controller roles, quorum availability and migration are still operational concerns; no unsupported claim about LinkedIn's motives is needed.

Failure answer: database commit then consumer crash can replay; effect+dedup atomically in the sink makes repetition safe. Offset-first can lose the effect. Explain this sequence before using the phrase exactly-once.

Sources and next step

Primary Kafka 3.7 sources checked 2026-10-07: design, producer configuration, topic configuration, basic operations. Exact passages and source hashes are saved; release-history claims have separate primary evidence.

Next: Stream processing. The exact local model/SQL ran; broker replication, rebalances and cloud throughput were not executed or benchmarked.

Original resource trail and reading purpose

The original optional pre-read destinations are retained below, not used as primary evidence for numerical claims, vendor incidents, or current API defaults. Video identity, duration, attribution and playback have not been independently verified here; search-result links are discovery links, not a particular verified lecture. Mutable latest documentation may differ from this lesson's pinned release. Use the versioned primary sources above for the governing contract.

• Kafka in 100 Seconds — original optional video/discovery destination; historical label, not verified attribution.
• System Design: Apache Kafka — original optional video/discovery destination; historical label, not verified attribution.
• Apache Kafka Fundamentals — Producers + Consumers Deep Dive — original optional video/discovery destination; historical label, not verified attribution.
• Kafka Design — original optional reading destination; historical label, not verified attribution.
• How to Choose the Number of Partitions — original optional reading destination; historical label, not verified attribution.
• Running Kafka at Scale — original optional reading destination; historical label, not verified attribution.

Read with a task: trace the producer acknowledgment, retained log and consumer bookmark separately; evaluate old capacity advice against your version and workload.

Series navigation: previous lesson; learning-plan hub. The original series used a 90-minute intuition → visual → hands-on → production → recall structure; that historical pacing is not a measured completion time for this revised lesson.