S048 · Kafka — Topics, Partitions, Consumer Groups

Module M05: Data Engineering · Session 48 of 130 · Track: DE 📮

What you'll be able to do after this session

• Explain Kafka to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S046 — Batch vs Streaming — Mental Model & Use Cases

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · Partitions | Apache Kafka® 101 (Confluent) — Tim Berglund on why partitions exist and what they buy you.
• 🎥 Deep dive (30–60 min) · Distributed Systems 1.1: Introduction — Martin Kleppmann — sets up the vocabulary you need to reason about brokers, replication, and ordering.
• 🎥 Hands-on demo (10–20 min) · Kafka in Docker: Step-by-Step Guide for Producers & Consumers — bring up Kafka in Docker and produce/consume real messages.
• 📖 Canonical article · Apache Kafka Documentation — Intro — the official reference. Read sections 1 (intro) and 4 (design).
• 📖 Book chapter / tutorial · Confluent Developer — Apache Kafka 101 — free, structured, 90-min course. The best written Kafka intro on the internet.
• 📖 Engineering blog · Confluent — What makes Apache Kafka the fastest messaging system? — the "why is Kafka fast" answer (zero-copy, sequential IO, page cache).

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: The event backbone of modern systems.

Every non-trivial company eventually has a mess: the app writes to Postgres, then also calls an email API, then also enqueues an ML feature update, then also emits an analytics event. Six months later, the checkout flow makes 14 synchronous calls, half of which fail intermittently, and a slow email service can take down the site. Kafka's core insight is: the app should write one thing — an event to a durable log — and let interested downstream systems read from that log at their own pace. No more tangled point-to-point wiring; instead, one central "spine" of events that anyone can subscribe to.

Concretely: Kafka is a distributed, append-only log. Producers write events (bytes with a key) to topics. Each topic is split into partitions (for parallelism and ordering); each partition is replicated across multiple brokers (for durability). Consumers read from wherever they left off, tracked by an offset. Multiple consumers in the same consumer group split the partitions among themselves — that's how you scale. Because the log is persistent (default retention 7 days, often set to weeks or forever), consumers can rewind, replay, and new services can bootstrap themselves from history — you have decoupled time between producers and consumers, which is the deepest, most useful property of the whole thing.

The gotcha to carry forever: ordering is per-partition, not per-topic. If you need "all events for user 42 to arrive in order," you must give them the same partition key (usually user_id), so they all land on the same partition. Two events with different keys can and will arrive out of order across the topic. Beginners routinely design systems assuming global ordering, then wonder why a payment "arrived" before its "order created" event.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

A tiny mental model:

Notes on the picture:

• Two independent consumer groups (analytics and fraud) each get every event — that's pub/sub for free.
• Within analytics, the 3 partitions are split among 3 consumers — that's horizontal scaling.
• If analytics had 4 consumers and 3 partitions, one consumer sits idle. Max parallelism = partition count. Choose your partition count carefully.
• Each partition is a strictly-ordered, append-only log. Reads are sequential — very fast on modern disks and page cache.

Core concepts in one table:

Worked example: an e-commerce site produces order.created events to a 12-partition topic, keyed by user_id. There are 3 consumer groups: warehouse (send-to-warehouse, 1 consumer), fraud (real-time scoring, 4 consumers), analytics (Spark job, 12 consumers = fully parallel). If you add a notifications service tomorrow, it joins as a 4th group and reads everything from the beginning of the retention window — no changes needed anywhere else. That's the payoff.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

Bring up a single-node Kafka in Docker and produce/consume in 5 minutes.

# 1. Kafka in one Docker container (KRaft mode, no ZooKeeper needed)
docker run --rm -d --name kf \
  -p 9092:9092 \
  -e KAFKA_CFG_NODE_ID=1 -e KAFKA_CFG_PROCESS_ROLES=controller,broker \
  -e KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@localhost:9093 \
  -e KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093 \
  -e KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 \
  -e KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER \
  -e KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT \
  bitnami/kafka:3.7
sleep 8
 
# 2. Create a topic with 3 partitions
docker exec kf kafka-topics.sh --create \
  --bootstrap-server localhost:9092 \
  --topic orders --partitions 3 --replication-factor 1
 
docker exec kf kafka-topics.sh --describe --bootstrap-server localhost:9092 --topic orders
 
# 3. Produce keyed messages — same key always lands on same partition
pip install kafka-python
python3 <<'PY'
from kafka import KafkaProducer
import json, time
p = KafkaProducer(bootstrap_servers='localhost:9092',
                  key_serializer=str.encode,
                  value_serializer=lambda v: json.dumps(v).encode())
for i in range(30):
    user = f"user_{i%5}"                 # 5 distinct users
    p.send('orders', key=user, value={'user':user, 'amt':i*10, 'ts':time.time()})
p.flush()
print("produced 30 messages")
PY
 
# 4. Consume with an explicit group. Ctrl-C after a few seconds.
python3 <<'PY'
from kafka import KafkaConsumer
c = KafkaConsumer('orders',
                  bootstrap_servers='localhost:9092',
                  group_id='demo-group',
                  auto_offset_reset='earliest',
                  enable_auto_commit=True,
                  value_deserializer=lambda b: b.decode())
for msg in c:
    print(f"p={msg.partition} off={msg.offset} key={msg.key.decode()} val={msg.value}")
PY
 
# 5. Prove ordering-per-partition by starting TWO consumers in the SAME group
#    in two terminals — they will split the 3 partitions between them.
#    Then start a THIRD consumer in a DIFFERENT group_id — it re-reads everything.
 
# Cleanup
docker stop kf

Observe (checklist):

1. All messages for user_2 land on the same partition (check the p= in the output).
2. Two consumers in the same group_id divide the partitions between them; a third in the same group with only 3 partitions sits idle.
3. A consumer in a different group_id starts from earliest and reads all messages independently — pub/sub for free.
4. kafka-consumer-groups.sh --describe --group demo-group shows the current offset lag per partition — this is your #1 production metric.
5. Kill the consumer mid-stream, restart it — it resumes from the committed offset, not from zero.

Try this modification: partition count is fixed once created (you can only increase, never decrease). Delete the topic and recreate with 6 partitions, produce 100 keyed messages, then check kafka-topics.sh --describe to see the partition assignment. Observe how a well-distributed key set uses all 6 partitions roughly evenly.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

Kafka's public production war stories are legendary. The most famous: LinkedIn (Kafka's birthplace) ran into "the partition problem" — early topics had 6 partitions; a year later they needed 600 and had to migrate live topics offline for hours to rebalance. Choose partition count generously up-front (10-100 for most topics) because you can never decrease it and increasing it invalidates the key-to-partition mapping for all existing keys.

Second story: exactly-once is subtle. By default, Kafka producers are at-least-once — a network blip may cause the producer to retry and duplicate a message. The fix is enable.idempotence=true (default in modern clients) plus, for cross-topic atomicity, transactions (transactional.id). Consumers must also handle duplicates idempotently (idempotent DB upserts, dedup by event ID). Uber's payment pipelines assume duplicates and dedup downstream — a much simpler mental model than trusting exactly-once end-to-end.

Third: consumer lag is your SLO. Every real Kafka deployment monitors records-lag-max per consumer group and alerts on it. If a consumer group's lag balloons, either the consumers are too slow (add more, up to partition_count) or the producers spiked. Burrow (LinkedIn's tool) and Confluent Control Center exist for this. Netflix's Keystone dashboards this at petabyte-per-day scale.

Fourth: avoid tiny messages and avoid huge messages. 1 KB - 100 KB is the sweet spot. Sub-1KB messages destroy your throughput/message-count ratio (each message has ~50 bytes of overhead). Messages > 1 MB cause replication and consumer memory issues — chunk large payloads or use a claim-check pattern (write blob to S3, put pointer in Kafka).

Finally: schema registry is non-negotiable at scale. With ~50 services all producing to Kafka, one team can ship a bad schema and break every downstream consumer. Confluent Schema Registry + Avro or Protobuf enforces backward/forward compatibility at publish time — bad schemas are rejected before they hit the wire. Every mature Kafka shop runs one.

(e) Quiz + exercise · 10 min

1. In one sentence, what problem does Kafka solve that a synchronous REST call between services doesn't?
2. What does a partition key control, and what happens if two events for the same user have different keys?
3. Why is "max consumer parallelism = number of partitions"?
4. What is the difference between two consumers in the same group vs two consumers in different groups?
5. Kafka defaults are at-least-once. What must you configure (producer and consumer sides) to move toward exactly-once, and what's still your responsibility?

Stretch (connects to S046 — Batch vs Streaming): you have Kafka feeding both a real-time fraud detector (Flink) and a nightly batch warehouse load. Both need the same events. How would you architect the sinks, and what happens if the fraud detector's Flink checkpoint is 6 hours behind — should the batch job wait? Argue trade-offs.

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is Kafka? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S049): Stream Processing — Watermarks, Windows, Exactly-Once
• Previous (S047): Spark — RDD, DataFrame, Jobs/Stages/Shuffles
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.