S048 · Kafka — Topics, Partitions, Consumer Groups
The distributed commit log that eats the world. Learn what Kafka actually is (a durable log, not a queue), why partitions are the unit of parallelism, and how consumer groups let a thousand consumers share the load without losing a message.
🎯 Model a real problem as Kafka topics + partitions + consumer groups, and reason about ordering, throughput, and rebalance cost like an owner, not a user.
Why this session exists
Every "event-driven" system, every streaming pipeline, every "let's decouple with a queue" architecture built after 2015 is really "we picked Kafka." But Kafka is not a queue, and treating it as one produces a beautiful cascade of production bugs: messages you thought were consumed but weren't, ordering guarantees that don't exist, partition counts that lock you into a limit forever. Ninety minutes here will save you six months of on-call.
- Explain a topic, a partition, an offset, and a consumer group to a junior in one paragraph each.
- Predict message ordering guarantees given a key and partition count.
- Choose a partition count that doesn't screw you 2 years from now.
- Read a consumer-group lag chart and know when to panic.
- Reason about the ‘exactly-once’ contract end-to-end (producer + broker + consumer).
Prerequisites
- S044 — NoSQL landscape (Kafka is the ‘log’ family).
- S046 — Batch vs streaming (Kafka is the streaming substrate).
(a) Intuition · 5 min
Imagine a village of 100 people who all keep a single shared diary. Anyone can write a new page at the end (never edit or delete). Anyone can start reading from any past page. Multiple readers can read at their own pace and remember their own bookmark. That's Kafka: a distributed, append-only, replayable log.
Now imagine the village is huge, so you split the diary into 12 separate diaries kept in different houses (partitions). Messages about crops go to diary 3, messages about weather go to diary 7 — deterministic hash by topic. Any reader can subscribe to some or all diaries.
A topic is a category of messages ("orders", "clicks", "audit_events"). Every topic is split into partitions — the unit of parallelism, ordering, and storage. Each partition is a sequential file on disk, replicated to N brokers. Producers append; consumers track their offset (bookmark).
Multiple consumers form a consumer group. Kafka assigns each partition to exactly one consumer in the group. Add consumers → more parallelism (up to partition count). Kill a consumer → its partitions rebalance to survivors.
Six ideas that unlock everything
- Topic — a category of messages (e.g. ‘orders’). Logical grouping.
- Partition — the unit of parallelism, ordering, storage. A topic has N partitions.
- Offset — a monotonically increasing integer per partition. The consumer's bookmark.
- Producer — appends to (topic, key). Key hashes to a partition; same key → same partition → same order.
- Consumer group — set of consumers sharing work. Kafka assigns each partition to exactly one consumer in the group.
- Replication factor — how many brokers keep a copy. RF=3 is the standard for prod.
A quick history so you know why the world looks like this
- 2010Kafka at LinkedInJay Kreps, Neha Narkhede, Jun Rao build it to unify LinkedIn's activity streams and operational metrics.
- 2011Open-sourcedApache incubation. ‘Kafka’ is Kreps's favourite author.
- 2013Confluent foundedSame team commercialises Kafka. Ships Schema Registry, Kafka Connect.
- 2017Exactly-once semantics shipIdempotent producer + transactions. Multi-year effort.
- 2019KIP-500 announced — remove ZooKeeperSelf-managed metadata via KRaft. Simpler ops.
- 2023Kafka 3.5 · KRaft GAZooKeeper is optional. Deployments halve in size.
(b) Visual walkthrough · 15 min
Kafka cluster anatomy
Producer → partition → consumer flow
Partition count — the one-way door
Bottleneck on parallelism.
- Max consumers in group = partition count
- One consumer per partition; extras idle
- Bump = new partitions, but old data doesn't rebalance to them
- Existing keys keep going to old partitions
Overhead everywhere.
- More open file descriptors per broker
- More metadata in the controller
- Slower controller election, longer failover
- Confluent recommends \<4000 partitions per broker
Room to grow ×2-3, not ×100.
- Target: peak consumer count × 2
- Common: 12, 24, 50, 100 for medium topics
- Multiples of typical replication factor
- Revisit yearly as scale grows
Consumer group rebalance
Heartbeats stop, group coordinator notices.
All consumers pause; coordinator reassigns partitions.
Consumers resume from where the previous owner committed.
If the previous owner processed but didn't commit, the new owner re-processes. Sink must be idempotent.
Only affected partitions pause. Standard in Streams / Connect. Enable for lower disruption.
"Kafka is a message queue. Consumers read messages and the messages are removed, like RabbitMQ or SQS."
Kafka is a distributed, replicated, append-only log. Reading does not remove anything — messages are retained by time or size policy regardless of consumption. A consumer is just a cursor (an offset) into an immutable sequence. That is why many independent consumer groups can read the same topic, and why you can rewind and reprocess history.
Because the API looks like a queue — produce, consume, acknowledge — and the words "topic" and "consumer" come straight from messaging systems. The model breaks the first time you need to add a second consumer of an existing topic, or replay last week's events after fixing a bug. In a real queue, both are impossible: the messages are gone. In Kafka both are trivial, and that single difference is why event-driven architectures are built on it.
The offset is the whole abstraction — reset it and history replays:
# Two independent groups reading the SAME topic, each with its own cursor.
# Neither affects the other, and neither removes data.
kafka-console-consumer --topic orders --group billing --from-beginning
kafka-console-consumer --topic orders --group analytics --from-beginning
# Replay after fixing a bug - impossible in a real queue:
kafka-consumer-groups --group analytics --topic orders \
--reset-offsets --to-datetime 2024-01-01T00:00:00.000 --execute
# Data is removed by RETENTION POLICY, never by consumption:
# retention.ms - drop segments older than N
# retention.bytes - cap partition size
# cleanup.policy=compact - keep the latest value per key, foreverWhy does Kafka guarantee ordering only within a partition, and never across a topic? A single global order sounds strictly better — why refuse to provide it?
- 1A total order over all messages in a topic requires a single point that assigns sequence numbers.forced by · two independent assigners cannot agree on relative order without coordinating on every message
- 2A single sequencer caps throughput at what one machine can do, and makes that machine a single point of failure.forced by · every message must pass through it, so it cannot be scaled horizontally by definition
- 3Kafka's central goal is horizontal scalability, so it splits a topic into partitions, each an independent log with its own leader broker.forced by · independent logs on different machines is the only way throughput scales with broker count
- 4Two messages written to different partitions therefore have no defined relative order — no shared clock and no coordination exists between their leaders.forced by · ordering across independent logs would require exactly the coordination that was removed to gain scale
- 5But ordering is only needed between causally related events — two updates to the same account, two events for the same user — and those can be forced into the same partition by using a partition key.forced by · partition assignment is a deterministic hash of the key, so equal keys always land in the same partition
Therefore per-partition ordering plus key-based partitioning gives you exactly the ordering guarantee you need, at a cost of exactly the coordination you can afford. The key choice is the ordering choice.
And note what this predicts: consumer parallelism within a group is capped at the partition count, because a partition is assigned to at most one consumer in a group — that is the only way per-partition order can be preserved on the read side too. So adding consumers beyond the partition count does nothing, and repartitioning a live topic changes which partition a key hashes to, breaking ordering for in-flight keys. Both of those follow directly from the derivation, and both surprise people in production.
Picture one enormous append-only ledger, sharded into partitions so it can be written by many machines at once. Producers only ever append. The ledger keeps entries according to a retention policy, entirely independent of who has read them.
Consumers are bookmarks. Each consumer group holds its own bookmark per partition and moves it forward as it processes. Nothing is consumed in the destructive sense — reading is just advancing a number. Adding a new consumer means starting a new bookmark, possibly at the very beginning.
- Ordering is per-partition only. If two events must be ordered, they must share a partition key.
- Max useful consumers per group = partition count. Partition count is a capacity decision made up front and painful to change.
- Delivery is at-least-once by default. Exactly-once effects come from idempotent writes or transactional produce-consume, never from delivery guarantees alone.
- Log compaction (
cleanup.policy=compact) retains the latest value per key indefinitely, turning a topic into a durable changelog you can rebuild state from — the basis of CDC and stream-table duality.
Fire this model the moment you see: several systems needing the same events · a need to reprocess history after a bug fix · decoupling producers from consumers · change data capture from a database · a consumer group lagging · a hot partition · "why did these two events arrive out of order?" · anywhere you were about to have service A call service B synchronously for something that is really a notification.
You must choose a partition key for a high-volume topic. Key by entity ID, key by nothing (round-robin), or key by a composite?
Key by the entity whose ordering matters, and check the distribution before committing — a skewed key is the Kafka equivalent of a skewed Spark join, and it has the same signature: one worker saturated while the rest idle. The partition count deserves the same care, because it is far easier to over-provision partitions on day one than to repartition a live topic later, and repartitioning changes key placement and breaks ordering for keys in flight.
The deeper point worth carrying: the partition key encodes your consistency boundary. Everything sharing a key can be processed with local state and guaranteed order; anything crossing keys requires coordination you will have to build yourself. Choosing the key is therefore an architectural decision about where transactions can exist, not a configuration detail — which is exactly the same decision as choosing a shard key in a distributed database.
(c) Hands-on · 25 min
Spin up single-node Kafka in Docker, produce/consume with two consumer groups, watch a rebalance. Save as kafka_demo.sh.
What each block does
Anatomy of the script
Open two terminals, run in each:
docker exec -it kafka-demo /opt/kafka/bin/kafka-console-consumer.sh \
--bootstrap-server localhost:9092 --topic orders --group demo --from-beginningBoth consumers share the load — 3 partitions each. In a third terminal, watch:
watch -n 1 "docker exec kafka-demo /opt/kafka/bin/kafka-consumer-groups.sh \
--bootstrap-server localhost:9092 --describe --group demo"Kill one consumer with Ctrl-C. Within ~10 seconds the survivor's partition set jumps from 3 to 6 — that's the rebalance.
(d) Production reality · 15 min
Early LinkedIn Kafka clusters had huge topics with thousands of partitions each. Controller failover took 5+ minutes because it had to re-elect leaders for every partition serially. During failover, the whole cluster was effectively unavailable.
Sharded topics across clusters and adopted KIP-500 (KRaft). Controller now handles metadata via a Raft consensus quorum rather than ZooKeeper reads/writes. Failover dropped to \\<10 seconds even with 100k+ partitions.
Slack's job-queue Kafka cluster had 6 000 partitions on a small cluster. A single-broker failure triggered a rebalance storm — thousands of consumer groups reassigning simultaneously, each pause blocking downstream systems. The cascade took the messaging pipeline down for hours.
Split into multiple smaller clusters by workload class. Adopted cooperative rebalance protocol (Kafka 2.4+) so only affected partitions pause. Reduced blast radius of any single broker failure by 10×.
A consumer processes a message, writes it to a downstream sink (Postgres row insert), then crashes before committing the offset. On restart, the consumer re-processes the message and inserts a duplicate. Later analytics show inflated numbers no one can explain.
Two options: (a) make the sink idempotent (natural key or dedup step), or (b) use Kafka transactions + read_committed consumer for exactly-once semantics. Most teams choose (a) because (b) has real throughput cost.
Where this shows up in the rest of the plan
(e) Recall + stretch · 10 min
Explain-out-loud test
- Kafka in one sentence — log, not queue.
- Partition = unit of what? — parallelism, ordering, storage.
- When is order guaranteed — per partition, so key by the entity that needs order.
What comes next
Hub: The 6-Month Learning Plan
Part of a 130-session evergreen learning series. Session structure: intuition → visual → hands-on → production war stories → recall. Duration: 90 minutes.