S052 · Lakehouse — Delta / Iceberg / Hudi, ACID on Files

Module M05: Data Engineering · Session 52 of 130 · Track: DE 🏞️

What you'll be able to do after this session

• Explain the lakehouse (Delta / Iceberg / Hudi) to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S047 — Spark — RDD, DataFrame, Jobs/Stages/Shuffles

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · What is a Data Lakehouse? — IBM Technology — 8-min whiteboard: warehouse vs lake vs lakehouse in one shot.
• 🎥 Deep dive (30–60 min) · Delta Lake vs Iceberg vs Hudi — comparison talk — Databricks + Onehouse engineers walk through the internals side by side.
• 🎥 Hands-on demo (10–20 min) · Apache Iceberg in 15 minutes with PyIceberg — click-along using Python, no Spark cluster needed.
• 📖 Canonical article · Delta Lake protocol spec — the actual file-level format; skim the "Delta log" section.
• 📖 Book chapter / tutorial · Apache Iceberg — Table format spec (v2) — read "Overview" and "Snapshots"; that's 80% of the mental model.
• 📖 Engineering blog · Netflix — Iceberg at Netflix — how Netflix runs incremental pipelines on Iceberg at exabyte scale.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: The convergence of warehouse and lake.

For 15 years you had two choices. Warehouse (Snowflake, BigQuery, Redshift): fast SQL, ACID transactions, but locked-in proprietary storage and expensive. Data lake (S3 + Parquet files): cheap and open, but no transactions — two jobs writing the same folder could corrupt each other, and there was no way to say "give me the table as it was yesterday."

The lakehouse fixes this by adding a small metadata layer on top of Parquet files. That metadata (a JSON log for Delta, an "Iceberg manifest" tree, or a Hudi timeline) records which files belong to which snapshot of the table. Suddenly you get ACID transactions, time travel, schema evolution, and MERGE — all on cheap S3/ADLS/GCS storage, readable by Spark, Trino, DuckDB, Flink, and Snowflake alike.

Real-life analogy: imagine a shared Google Doc backed by plain .txt files on Dropbox. Without a "who edited what and when" log, two people saving at the same time destroys each other's work. Delta/Iceberg/Hudi are that log — an append-only history of "add file X, remove file Y" operations that makes concurrent writes safe.

Forever gotcha: the data still lives as immutable Parquet files. An "UPDATE" doesn't rewrite a row — it writes new files and marks the old ones deleted in the log. This means small updates create tons of tiny files; you must run compaction regularly or query performance collapses.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

Worked example — a MERGE that becomes an "add file, remove file":

Start: table has part-0001.parquet (100 rows for customer_ids 1–100).

You run:

MERGE INTO customers t USING updates u
  ON t.id = u.id
  WHEN MATCHED THEN UPDATE SET t.email = u.email
  WHEN NOT MATCHED THEN INSERT *

What actually happens on disk:

1. Reader scans part-0001.parquet, finds 30 matched rows.
2. Writer creates part-0002.parquet containing the 100 rows with 30 updated emails.
3. Writer creates part-0003.parquet with 10 brand-new rows.
4. Metadata log gets a new entry: {add: [part-0002, part-0003], remove: [part-0001]}.
5. Old part-0001 is not deleted from S3 — it just becomes invisible to new queries. Time-travel readers pinned to the previous snapshot still see it.

The three big open formats compared:

All three converge on the same idea. In 2024–2025 the ecosystem is standardising: Snowflake, BigQuery, Databricks, and AWS all now read/write Iceberg, and Delta has "UniForm" mode that also speaks Iceberg. If you're starting fresh in 2025, Iceberg is the safest bet for engine flexibility.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

Local, no cluster, no cloud — using PyIceberg + DuckDB.

python -m venv .venv && source .venv/bin/activate
pip install "pyiceberg[pyarrow,duckdb,sql-sqlite]==0.7.1" pyarrow
mkdir -p /tmp/warehouse

# lakehouse_demo.py
from pyiceberg.catalog.sql import SqlCatalog
import pyarrow as pa
 
catalog = SqlCatalog("demo", **{
    "uri": "sqlite:////tmp/warehouse/catalog.db",
    "warehouse": "file:///tmp/warehouse",
})
catalog.create_namespace_if_not_exists("shop")
 
# 1. Create an Iceberg table from Arrow schema
schema = pa.schema([
    ("id", pa.int64()),
    ("name", pa.string()),
    ("amount", pa.float64()),
])
tbl = catalog.create_table_if_not_exists("shop.orders", schema=schema)
 
# 2. Append batch 1
b1 = pa.Table.from_pylist([
    {"id": 1, "name": "Alice", "amount": 25.0},
    {"id": 2, "name": "Bob",   "amount": 42.5},
], schema=schema)
tbl.append(b1)
 
# 3. Append batch 2
b2 = pa.Table.from_pylist([
    {"id": 3, "name": "Carol", "amount": 99.9},
], schema=schema)
tbl.append(b2)
 
# 4. Read the current snapshot
print("=== current snapshot ===")
print(tbl.scan().to_arrow().to_pandas())
 
# 5. Time travel — read the FIRST snapshot
snapshots = list(tbl.snapshots())
print(f"\ntable has {len(snapshots)} snapshots")
first = snapshots[0].snapshot_id
print(f"\n=== snapshot {first} (first append) ===")
print(tbl.scan(snapshot_id=first).to_arrow().to_pandas())
 
# 6. Inspect metadata on disk
import subprocess
print("\n=== files on disk ===")
subprocess.run(["find", "/tmp/warehouse/shop.db", "-type", "f"])

Run: python lakehouse_demo.py

What to observe:

1. Each append() created a NEW parquet file — old files are not touched.
2. list(tbl.snapshots()) shows 2 snapshots; each has a snapshot_id, parent_id, and pointer to a manifest file.
3. Time travel with snapshot_id=first returns just Alice + Bob, no Carol.
4. Under metadata/ you'll see .avro manifest files and versioned .metadata.json files — that's the Iceberg log.
5. Under data/ you'll see .parquet — the actual rows.

Try this modification: call tbl.delete("id = 2") and then re-scan the current snapshot AND the first snapshot. You'll see Bob gone from current but still present in the first snapshot — that's ACID + time travel in one line.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

Netflix runs Iceberg on hundreds of petabytes and processes trillions of rows daily. Apple, LinkedIn, Airbnb, Stripe, and Adobe all publish about their Iceberg deployments. Databricks internal telemetry lives on Delta. Uber built Hudi specifically for their driver-earnings pipeline which needs streaming upserts. This is not experimental tech — it's the substrate of modern data platforms.

Real war stories:

• The small-files apocalypse. A Kafka → Iceberg streaming job wrote a 200KB file every minute. After 30 days: 43,000 files in one partition. Query planning took 90 seconds before any data was read. Fix: schedule OPTIMIZE (Delta) / rewrite_data_files (Iceberg) hourly to compact into 128–512MB files.
• The unstoppable growth. No one ran VACUUM / expire_snapshots. After a year of daily MERGEs, S3 held 50× the "live" table size in dead files. Fix: retention policy — expire snapshots older than 7 days; run vacuum weekly. Just don't set the retention shorter than your longest-running query, or in-flight queries die.
• Concurrent-writer race. Two Airflow tasks both did INSERT OVERWRITE on the same partition. Both committed. One won, one silently lost. Fix: enable optimistic-concurrency control (default in Delta/Iceberg) and let the loser retry — never disable it "to avoid conflicts."
• The schema-evolution footgun. Someone dropped column email and re-added it a week later. Old Parquet files still had the old email — the reader tried to bind by name, got confused, returned nulls. Fix: Iceberg uses field IDs not names precisely to avoid this; make sure you're using ID-based projection.

How top teams handle it: an "OPTIMIZE + VACUUM" nightly job per table, alerting on file-count-per-partition, a REST catalog (Nessie, Polaris, Unity, or AWS Glue) so multiple engines share the same metadata, and treating the table format as a contract — schema-changes go through review just like API changes.

Gotcha to memorise: deleting a row does NOT delete data. It writes a "delete file" (Iceberg v2) or new parquet + metadata pointer (Delta). Storage keeps growing until you VACUUM. Budget for it.

(e) Quiz + exercise · 10 min

1. What problem does a lakehouse solve that plain Parquet-on-S3 cannot?
2. When you UPDATE one row in a Delta table, what physically happens on S3?
3. What are the three main open table formats, and which company originally built each?
4. Why do you need to run OPTIMIZE / compaction regularly on a streaming table?
5. What's the difference between VACUUM / expire_snapshots and dropping a table?

Stretch (connects to S051 — dbt): dbt now has a table_format: iceberg config in Databricks and Snowflake adapters. Sketch how you'd take the mart_revenue model from S051 and materialise it as an Iceberg table that other Trino users can query. What would break if the Trino catalog and dbt catalog disagreed on the schema?

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is a lakehouse? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S053): Data Quality — Freshness, Volume, Schema, Distribution
• Previous (S051): dbt — Models, Tests, Docs, Warehouse-Native ELT
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.