dineshblog

Search Tech Journey

Find topics, journeys and posts

back to blog
data engineeringintermediate 15m2026-07-06

Kafka — Brokers, Partitions, Replication, Consumers

A deep-dive on Brokers, Partitions, Replication, Consumers — part of a 24-topic evergreen learning series.

Why this session matters

Part of a 24-topic learning series on engineering, ML, and LLM systems. Each session is a 90-minute deep-dive on one topic — designed so anyone can pick it up cold. Every two topics are followed by a revision session with recall prompts and hands-on drills.

Part 1: Kafka Part 1 — Brokers, Topics, Partitions, Producers

Why this session matters

It builds on the rhythm of one focused topic, paced so you have time to actually absorb it rather than rush.

Agenda

  • Kafka mental model — distributed append-only log per topic-partition
  • Brokers, controller, ZooKeeper-vs-KRaft — the metadata story
  • Topics, partitions, offsets — the unit of parallelism
  • Producer semantics — acks, idempotence, batching, compression
  • Where consumers and replication fit (Part 2)

Pre-read (skim before the session)

Deep dive

1. The one-line mental model

Kafka is a distributed, durable, append-only log partitioned across brokers. Producers append; consumers read at their own pace and track their own offset. That's it. Everything else is engineering around those primitives.

2. Why it won

  • Decouples producers from consumers (multiple consumers can read the same topic).
  • Durable by default — messages survive consumer restarts.
  • Massive throughput — millions of events/sec on a moderate cluster.
  • Becomes the event backbone for microservices, ETL, ML, search indexing.

Used at LinkedIn (origin), Uber, Netflix, Microsoft, Goldman Sachs — essentially everywhere with serious data movement.

3. Cluster topology

   producers          brokers (1…N)         consumers
   ---------         ---------------         ---------
      ▼                  ▼                       ▲
     (write)         topic A: partitions 0,1,2,3,4   (read)
                     topic B: partitions 0,1
                          │
                          ▼
                  controller (one elected broker;
                  KRaft uses Raft, replacing ZooKeeper)

Each partition is owned by one leader broker; reads and writes go through the leader. Replicas are followers that pull from the leader.

4. Topics & partitions — the unit of parallelism

  • A topic is a logical stream (e.g. orders.placed).
  • A topic is split into N partitions — each is an ordered, append-only log on disk.
  • Within a partition, messages are strictly ordered. Across partitions, no global order.
  • Partition key decides which partition a message goes to: partition = hash(key) % N.

Key choices:

  • Pick a key so that ordering matters within the key (e.g. user_id). All events for one user land on one partition, processed in order.
  • Don't pick a key with bad cardinality (country → hot partition for the dominant country).
  • Partition count is fixed-ish — you can add but not remove cleanly. Start at 2× expected concurrent consumers.

5. The controller and metadata (KRaft replaces ZooKeeper)

The controller elects partition leaders, watches broker health, and maintains the cluster metadata. Up to Kafka 2.x this lived in ZooKeeper. From Kafka 3.3+ (production-ready 3.7+), KRaft mode replaces ZooKeeper with a self-hosted Raft quorum. Less operational surface, faster failover, official direction.

6. The on-disk log

Each partition is a series of segment files (.log + .index):

topic-partition-dir/
  00000000000000000000.log     # offsets 0–1,099,999
  00000000000000000000.index
  00000000000001100000.log     # 1.1M onward
  00000000000001100000.index
  • New writes append to the active segment.
  • Old segments are deleted (retention by time/size) or compacted (keep latest per key).
  • The .index is a sparse offset → file-offset map, used to seek without scanning.
  • Writes are sequential disk I/O — the killer feature. Kafka relies on the page cache more than its own caches.

7. Producer side — the knobs that matter

from confluent_kafka import Producer
p = Producer({
  "bootstrap.servers": "broker:9092",
  "acks": "all",                # see below
  "enable.idempotence": True,   # see below
  "compression.type": "lz4",    # 'snappy' or 'lz4' for prod
  "linger.ms": 5,               # batch window
  "batch.size": 65536,          # max bytes per batch
})
p.produce("orders.placed", key=order.user_id.encode(),
          value=json.dumps(order).encode())
p.flush()

Key settings:

  • acks:
    • 0 — fire and forget. Lose messages on broker crash. Almost never.
    • 1 — leader acks. Lose data if leader dies before follower replicates. OK for telemetry where some loss is fine.
    • all — leader waits for all in-sync replicas (ISR). Standard for durable data.
  • enable.idempotence = True — producer dedups retries by sequence number. Required for exactly-once. Always on in modern setups.
  • Batching: linger.ms = 5–20, batch.size = 64 KB typical. Bigger batches → better throughput, slightly higher latency.
  • Compression: lz4 for low CPU, snappy for compatibility. ~3–5× throughput improvement, free.
  • Keys: always set if you need ordering or co-location.

8. Throughput numbers (real-ish)

On a moderate 3-broker cluster (16-core, 64 GB RAM, NVMe each):

  • ~1–2 GB/s sustained ingest.
  • 1–2 M small messages/sec per broker.
  • p99 produce latency ~5–10 ms with acks=all + idempotence.

These aren't promises — they're what well-tuned production looks like in 2026.

9. Common producer mistakes

  • No key + ordering matters — random partitioning shuffles events for the same entity.
  • acks=1 for financial data — silent loss on leader failover.
  • No idempotence + retries on — duplicates.
  • Sync produce in a tight loop — single-threaded throughput. Use async + flush() periodically.
  • Tiny batch.size — misses batching benefits.
  • Producer per request in a web service — connection storm. Use one long-lived producer.

10. What's next (the next session — Kafka Part 2)

  • Replication, ISR, unclean leader election
  • Consumer groups, partition assignment, rebalances
  • Exactly-once semantics (idempotence + transactions)
  • Schema Registry & evolution
  • Production monitoring (lag, ISR shrink, throughput)

Reading material

Books:

  • Kafka: The Definitive Guide, 2nd ed. — Shapira, Palino, Sivaram, Petty (the Confluent book; chs. 1–4)
  • Designing Event-Driven Systems — Ben Stopford (free O'Reilly PDF from Confluent)
  • Designing Data-Intensive Applications — Kleppmann (ch. 11: Stream Processing)

Papers:

Official docs:

Blog posts:

In-depth research material

Videos

LeetCode — Design Circular Queue

Assignment / Deliverables

Give yourself a clean 2-hour window and complete all of these before moving on:

  1. Read the deep-dive above end-to-end — no skimming. Take notes in your own words.
  2. Solve the LeetCode problem below without help first. Only look at the hint after 15 focused minutes; only look at editorial after 30. Log your time.
  3. Reproduce one code snippet locally. Pick the snippet that felt least obvious and get it running in a scratch file / notebook.
  4. Draw the core diagram from memory. Paper, whiteboard, or tldraw — doesn't matter. If you can't, re-read section 2 and try again.
  5. Write a 3-line takeaway in your prep journal: what surprised you, what you still don't understand, what you'd read next.
  6. Skim one item from the Reading material section. Bookmark the rest for the weekend.
  7. Commit any code + notes to your prep repo with message session-NN: <one-line summary>.

Stretch (optional, +30 min): explain today's topic to a rubber duck / a friend / a voice note. If you can't teach it in 5 minutes, you don't own it yet — flag it and revisit next weekend.

Post-session checklist

By the end of this session you should be able to:

  • State the one-line mental model and what each word means.
  • Choose a partition key for a known workload and defend it.
  • Explain acks 0/1/all trade-offs.
  • Turn idempotence on and explain what it actually protects.
  • Tune linger.ms + batch.size + compression for throughput vs latency.
  • Solve design-circular-queue — same shape as Kafka segments.

Generated from sessions_data.py + content_part*.py. To edit a video / leetcode / title, edit the data file and re-run write_sessions.py.

Part 2: Kafka Part 2 — Replication, ISR, Consumer Groups, Exactly-Once

Why this session matters

It builds on the rhythm of one focused topic, paced so you have time to actually absorb it rather than rush.

Agenda

  • Replication — leader/follower, ISR, high watermark, acks
  • Consumer groups — rebalance, sticky vs cooperative, offset management
  • Delivery semantics — at-most-once, at-least-once, exactly-once
  • Transactions and the idempotent producer — what they actually guarantee
  • Operating Kafka — unclean leader election, min.insync.replicas, retention

Pre-read (skim before the session)

Deep dive

1. Replication primer

Each partition has one leader and zero or more followers (replicas). Writes go to the leader; followers Fetch to copy. The set of followers caught up to the leader is the ISR (In-Sync Replicas).

producer ── write ──▶ leader ── replicate ──▶ follower1
                                  └────────▶ follower2
                                  └────────▶ follower3 (lagging — kicked out of ISR)

A follower stays in ISR as long as it has fetched within replica.lag.time.max.ms (default 30 s). Drop behind and you're out — and writes can succeed without you.

2. The high watermark

The high watermark (HW) is the offset up to which all ISR replicas have replicated. Consumers can only read up to the HW. Why? Because anything past the HW might be lost if the leader dies before followers catch up.

offsets:  0 1 2 3 4 5 6 7 8
leader:   ■ ■ ■ ■ ■ ■ ■ ■ ■
isr1:     ■ ■ ■ ■ ■ ■ ■
isr2:     ■ ■ ■ ■ ■ ■ ■
HW = 7 (consumers can read 0..6)
LEO = 9 (Log End Offset on leader)

3. acks — producer durability knob

acks=0  → fire and forget. Lowest latency, can lose unbounded messages.
acks=1  → leader acks once written to its log. Can lose 1 message on leader failure.
acks=all → leader acks after all ISR replicas have written. No data loss IF
            min.insync.replicas ≥ 2 and ISR ≥ min.insync.replicas.

The gotcha: acks=all alone is not durable. If your ISR shrinks to just the leader (followers all lag), acks=all is effectively acks=1. Always pair with min.insync.replicas=2 (for RF=3) so the leader rejects writes when it can't guarantee.

4. min.insync.replicas — the actual durability knob

RFmin.insync.replicasToleratesNotes
321 broker lossStandard for production
330 broker lossHigher latency, no resilience
312 broker lossBut data can be lost on leader failure

When ISR \< min.insync.replicas, producers get NotEnoughReplicasException and writes pause. That's the desired behaviour — pause rather than silently risk data loss.

5. Unclean leader election

If all ISR replicas die, what happens? Two options:

  • unclean.leader.election.enable=false (default since 2.x): partition unavailable until an ISR replica returns. Data preserved.
  • unclean.leader.election.enable=true: an out-of-sync replica becomes leader. Service restored; data loss (everything past the lagger's offset is gone).

Don't enable unclean election unless you're 100% sure your data can be re-derived.

6. Consumer groups

Multiple consumers in the same group share a topic — each partition assigned to exactly one consumer. Add consumers → throughput scales linearly up to # partitions. Beyond that, extra consumers sit idle.

topic: 6 partitions
group: 3 consumers
→ each consumer gets 2 partitions

group: 6 consumers
→ each consumer gets 1 partition

group: 9 consumers
→ 6 active, 3 idle

7. Rebalance — the operational landmine

When a consumer joins or leaves, Kafka rebalances: all consumers stop, partitions are reassigned. Two modes:

  • Eager (legacy): everyone gives up all partitions, then re-acquires. Full stop-the-world.
  • Cooperative (KIP-429, default since 2.4): only the moving partitions pause. Most consumers keep processing.

Pre-cooperative, a rolling deploy could cause 30+ seconds of cumulative pause. Cooperative cut that to seconds. Always use cooperative now (partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor).

8. Offsets — where you are in the log

Consumers commit their offset to __consumer_offsets (an internal Kafka topic). Two modes:

  • Auto-commit (enable.auto.commit=true, every 5 s): convenient, but commits before you've processed the messages → at-least-once where you might skip on crash. Avoid for non-trivial work.
  • Manual commit (consumer.commitSync() after processing): commit after you've durably written / acked downstream. Standard for production.

9. Delivery semantics

ModeProducerConsumerUse case
At-most-onceacks=0auto-commit before processMetrics, logs (loss OK)
At-least-onceacks=allcommit after processMost pipelines (must be idempotent)
Exactly-onceidempotent producer + transactionsread-process-write in transactionFinancial, audit

10. Idempotent producer (KIP-98)

Setting enable.idempotence=true makes the producer attach a producer ID (PID) and sequence number to each record. The broker dedupes any retried record with the same (PID, seq). This eliminates the classic "retry caused duplicate" bug.

Default since 3.0. Always on. Free correctness improvement.

11. Transactions

producer = KafkaProducer(transactional_id="payments-tx-1", enable_idempotence=True)
producer.init_transactions()

producer.begin_transaction()
producer.send("orders", order_msg)
producer.send("payments", payment_msg)
producer.send_offsets_to_transaction(consumer.position(), consumer_group)
producer.commit_transaction()  # or abort_transaction() on error

Transactions provide atomic multi-partition writes and read-process-write exactly-once when the consumer reads in read_committed mode (skips aborted records). They cost 5–20% throughput. Worth it for billing, ledgers, audit pipelines.

12. Retention

log.retention.hours=168       # 7 days
log.retention.bytes=-1        # unlimited
log.segment.bytes=1073741824  # 1 GB segments
log.cleanup.policy=delete     # or 'compact' for keyed topics

Compacted topics keep only the latest value per key (key=customer_id, value=current profile) — a Kafka topic doubles as a changelog / KV store. State stores in Kafka Streams are built on compacted topics.

13. Operating in production

  • Monitor: under-replicated partitions, ISR shrinks, consumer lag, request latency.
  • Capacity: aim for <70% disk usage; partitions sized so retention fits.
  • Partition count: 100–1000 per broker; too many = high controller overhead, slow leader election.
  • JVM tuning: G1GC, ~25% heap, rest for OS page cache (Kafka leans on page cache hard).
  • Network: 10 Gbps minimum for serious throughput; producers are usually network-bound.

Reading material

Books:

  • Kafka: The Definitive Guide, 2nd ed. — Shapira, Palino et al. (chs. 6–8: reliable delivery, exactly-once, replication)
  • Designing Data-Intensive Applications — Kleppmann (ch. 11.4 on exactly-once semantics)
  • Mastering Kafka Streams and ksqlDB — Mitch Seymour (consumer-group + EOS patterns)

Papers:

Official docs:

Blog posts:

In-depth research material

Videos

LeetCode — Design Hit Counter

Assignment / Deliverables

Give yourself a clean 2-hour window and complete all of these before moving on:

  1. Read the deep-dive above end-to-end — no skimming. Take notes in your own words.
  2. Solve the LeetCode problem below without help first. Only look at the hint after 15 focused minutes; only look at editorial after 30. Log your time.
  3. Reproduce one code snippet locally. Pick the snippet that felt least obvious and get it running in a scratch file / notebook.
  4. Draw the core diagram from memory. Paper, whiteboard, or tldraw — doesn't matter. If you can't, re-read section 2 and try again.
  5. Write a 3-line takeaway in your prep journal: what surprised you, what you still don't understand, what you'd read next.
  6. Skim one item from the Reading material section. Bookmark the rest for the weekend.
  7. Commit any code + notes to your prep repo with message session-NN: <one-line summary>.

Stretch (optional, +30 min): explain today's topic to a rubber duck / a friend / a voice note. If you can't teach it in 5 minutes, you don't own it yet — flag it and revisit next weekend.

Post-session checklist

By the end of this session you should be able to:

  • Explain the ISR + high-watermark mechanism with a diagram.
  • State the difference between acks=all and acks=all + min.insync.replicas=2.
  • Describe cooperative rebalancing and why it beats eager.
  • Implement at-least-once consumer logic with manual offset commits.
  • Explain what transactions guarantee (and what they don't).
  • Solve design-hit-counter — bucketed time window, same primitive as Kafka's log segment trimming.

Generated from sessions_data.py + content_part*.py. To edit a video / leetcode / title, edit the data file and re-run write_sessions.py.

← previous

RAG End-to-End — Chunking, Embeddings, Retrieval, Rerank, Eval

next →

Distributed Systems Core — CAP/PACELC, Sharding, Replication, Multi-Region