data engineeringintermediate 12m2026-06-09

Kafka Part 1 — Brokers, Topics, Partitions, Producers

Session 10 of the 48-session learning series.

Date: Wed, 2026-06-17 · Time: 18:00–20:00 IST · Track: 🗂️ Data Engineering (DE) · Parent 28-day topic: Day 07 · Est. read: 2 h

Why this session matters

This is Session 10 of 48 in the Data Engineering track. It builds on the rhythm of one focused topic, paced so you have time to actually absorb it rather than rush.

Agenda

Kafka mental model — distributed append-only log per topic-partition
Brokers, controller, ZooKeeper-vs-KRaft — the metadata story
Topics, partitions, offsets — the unit of parallelism
Producer semantics — acks, idempotence, batching, compression
Where consumers and replication fit (Part 2)

Pre-read (skim before the session)

Kafka is a distributed, durable, append-only log partitioned across brokers. Producers append; consumers read at their own pace and track their own offset. That's it. Everything else is engineering around those primitives.

2. Why it won

Decouples producers from consumers (multiple consumers can read the same topic).
Durable by default — messages survive consumer restarts.
Massive throughput — millions of events/sec on a moderate cluster.
Becomes the event backbone for microservices, ETL, ML, search indexing.

Used at LinkedIn (origin), Uber, Netflix, Microsoft, Goldman Sachs — essentially everywhere with serious data movement.

3. Cluster topology

   producers          brokers (1…N)         consumers
   ---------         ---------------         ---------
      ▼                  ▼                       ▲
     (write)         topic A: partitions 0,1,2,3,4   (read)
                     topic B: partitions 0,1
                          │
                          ▼
                  controller (one elected broker;
                  KRaft uses Raft, replacing ZooKeeper)

Each partition is owned by one leader broker; reads and writes go through the leader. Replicas are followers that pull from the leader.

4. Topics & partitions — the unit of parallelism

A topic is a logical stream (e.g. orders.placed).
A topic is split into N partitions — each is an ordered, append-only log on disk.
Within a partition, messages are strictly ordered. Across partitions, no global order.
Partition key decides which partition a message goes to: partition = hash(key) % N.

Key choices:

Pick a key so that ordering matters within the key (e.g. user_id). All events for one user land on one partition, processed in order.
Don't pick a key with bad cardinality (country → hot partition for the dominant country).
Partition count is fixed-ish — you can add but not remove cleanly. Start at 2× expected concurrent consumers.

5. The controller and metadata (KRaft replaces ZooKeeper)

The controller elects partition leaders, watches broker health, and maintains the cluster metadata. Up to Kafka 2.x this lived in ZooKeeper. From Kafka 3.3+ (production-ready 3.7+), KRaft mode replaces ZooKeeper with a self-hosted Raft quorum. Less operational surface, faster failover, official direction.

6. The on-disk log

Each partition is a series of segment files (.log + .index):

topic-partition-dir/
  00000000000000000000.log     # offsets 0–1,099,999
  00000000000000000000.index
  00000000000001100000.log     # 1.1M onward
  00000000000001100000.index

New writes append to the active segment.
Old segments are deleted (retention by time/size) or compacted (keep latest per key).
The .index is a sparse offset → file-offset map, used to seek without scanning.
Writes are sequential disk I/O — the killer feature. Kafka relies on the page cache more than its own caches.

7. Producer side — the knobs that matter

from confluent_kafka import Producer
p = Producer({
  "bootstrap.servers": "broker:9092",
  "acks": "all",                # see below
  "enable.idempotence": True,   # see below
  "compression.type": "lz4",    # 'snappy' or 'lz4' for prod
  "linger.ms": 5,               # batch window
  "batch.size": 65536,          # max bytes per batch
})
p.produce("orders.placed", key=order.user_id.encode(),
          value=json.dumps(order).encode())
p.flush()

Key settings:

acks:
- 0 — fire and forget. Lose messages on broker crash. Almost never.
- 1 — leader acks. Lose data if leader dies before follower replicates. OK for telemetry where some loss is fine.
- all — leader waits for all in-sync replicas (ISR). Standard for durable data.
enable.idempotence = True — producer dedups retries by sequence number. Required for exactly-once. Always on in modern setups.
Batching: linger.ms = 5–20, batch.size = 64 KB typical. Bigger batches → better throughput, slightly higher latency.
Compression: lz4 for low CPU, snappy for compatibility. ~3–5× throughput improvement, free.
Keys: always set if you need ordering or co-location.

8. Throughput numbers (real-ish)

On a moderate 3-broker cluster (16-core, 64 GB RAM, NVMe each):

~1–2 GB/s sustained ingest.
1–2 M small messages/sec per broker.
p99 produce latency ~5–10 ms with acks=all + idempotence.

These aren't promises — they're what well-tuned production looks like in 2026.

9. Common producer mistakes

No key + ordering matters — random partitioning shuffles events for the same entity.
acks=1 for financial data — silent loss on leader failover.
No idempotence + retries on — duplicates.
Sync produce in a tight loop — single-threaded throughput. Use async + flush() periodically.
Tiny batch.size — misses batching benefits.
Producer per request in a web service — connection storm. Use one long-lived producer.

10. What's next (Session 15 — Kafka Part 2)

Replication, ISR, unclean leader election
Consumer groups, partition assignment, rebalances
Exactly-once semantics (idempotence + transactions)
Schema Registry & evolution
Production monitoring (lag, ISR shrink, throughput)

Link: https://leetcode.com/problems/design-circular-queue/
Difficulty: Medium
Why this problem: Fixed array with head/tail pointers; mod arithmetic.
Time-box: 30 minutes. Look up the editorial only after.

Post-session checklist

By the end of this session you should be able to:

State the one-line mental model and what each word means.
Choose a partition key for a known workload and defend it.
Explain acks 0/1/all trade-offs.
Turn idempotence on and explain what it actually protects.
Tune linger.ms + batch.size + compression for throughput vs latency.
Solve design-circular-queue — same shape as Kafka segments.

Generated from sessions_data.py + content_part*.py. To edit a video / leetcode / title, edit the data file and re-run write_sessions.py.

← previous

RAG Part 1 — Why, Chunking, Embeddings, Vector Stores

SOLID Part 2 — ISP, DIP, and Design Patterns (Strategy, Factory, Observer)