Lakehouse — Delta / Iceberg / Hudi, ACID on Files

Review boundary — 10 October 2026: The exact snapshot/retention model runs on local CPU. The delta-rs example is reviewed against python-v0.17.4 source; Delta protocol claims use v3.3.0, Iceberg documentation 1.9.2, and Hudi 1.0.2. No package installation, storage service, Spark cluster or delta-rs runtime ran. Do not interpret the local model as a distributed commit implementation.

Learning goal: Explain transactional tables over files, follow three local Delta versions, compare Delta/Iceberg/Hudi, and design maintenance without destroying required history.

Why this session exists

A directory of Parquet files does not by itself tell a reader which files form one complete update. Copying lake data into a warehouse is one way to serve governed analytics; an open table format is another. Lakehouse architectures add transactional metadata, compatible engines and operational controls around file storage. They did not universally replace lake-plus-warehouse systems, and retaining both tiers does not necessarily double the entire bill.

You will be able to explain manifests, snapshots and atomic publication; follow delta-rs writes and versioned reads; compare engine/catalog/write-pattern choices; diagnose small files, compaction and metadata growth; and choose a format for BI, streaming updates or retained ML inputs.

Prerequisites

Understand Parquet, object storage, batch/streaming ingestion and the difference between table grain and file partitioning. Session 047 covers Spark, 049 stream processing and 051 warehouse ELT; the original prerequisite numbers were not accurate.

Optional resources for S052

For optional orientation and exploration, retain the lakehouse video pointer, Delta video pointer, and Iceberg tutorial search. Reading options are the Delta introduction, Iceberg specification landing page, and lakehouse background essay. Use these as pointers, not substitutes for the pinned protocol and API references below; the landing-page link alone does not establish a specification version.

No video content, creator attribution, viewing quality or duration has been verified here; the search is not an identified video. The former “30 min total” and 9-, 42- and 35-minute video labels are not a promised preparation schedule. “Clearest”, “best free”, “80% of the mental model”, and claims about who coined or popularised the term are not established by these links. The snapshot, compatibility and maintenance exercises remain the learning work.

(a) Intuition · 5 min

A ledger stapled to a pile of receipts is a useful analogy. The receipts are immutable data files. The ledger says which files belong to a committed table version. Readers consult a committed ledger, not every object they happen to find. A writer may upload replacement receipts before publishing their membership; readers of the previous version must continue seeing its old receipts.

The analogy's “front-desk lock” is not a universal implementation. A table format defines a commit protocol whose storage/catalog integration coordinates writers. Atomic upload of one object does not publish many files atomically as a table, nor detect conflicting logical changes by itself.

The three problems the lakehouse solves

• Consistent membership: publish one metadata transition after all referenced data is ready, so readers select old or new state rather than a partial batch.
• Historical reads: retained metadata identifies older snapshots, but the referenced data must still exist and remain readable by a compatible engine.
• Schema and updates: the format records schema/identity and mutation metadata; the engine implements supported writes, constraints and conflict checks. A table format is not a replacement encoding for Parquet.

“Lakehouse” is an architectural term, not a certification earned by exposing a JSON log. Access control, performance, maintenance and governance still require a platform. Delta commonly uses Parquet data; Iceberg supports more than Parquet, and Hudi's base/log formats depend on its configuration.

How we got here

The meaningful evolution is from file membership inferred through directory/partition listings to explicit transactional metadata. Delta uses a versioned log and checkpoints; Iceberg uses immutable metadata trees and an atomic update to the current table metadata; Hudi uses a timeline and file groups with update-oriented table types. Earlier material mixed a 2020 lakehouse blog with a 2021 CIDR paper and gave unsupported company/adoption/acquisition specifics. The design lesson is retained here without turning those dates or prices into evidence.

Anatomy of a Delta table on S3

The objects present in storage and the live file set differ. After version 2, p1 can remain physically present for old readers while absent from the latest table. Delta readers reconstruct state using suitable checkpoints and log entries; checkpoints are not necessarily one classic file at a universal ten-commit interval. The v3.3.0 protocol includes multiple checkpoint forms and optional features. Raw prefix scans bypass the table contract.

The commit lifecycle

1. Read a base snapshot and record assumptions about the files/schema being changed.
2. Write new data files without making them table-visible yet.
3. Construct file additions/removals and required metadata actions.
4. Validate conflicts and publish through the supported storage/catalog commit mechanism.
5. If publication races, re-read and revalidate. A loser cannot blindly append the same actions at the next version: an overlapping delete/rewrite may invalidate its assumptions.
6. Readers select a committed snapshot. Maintenance can later compact metadata and expire eligible history.

Delta's “never overwrite an existing log entry” requirement is not a blanket claim that every S3 client safely uses the same conditional PUT. Iceberg's atomic metadata replacement is similarly a protocol plus implementation contract. Strong object-store consistency does not create multi-file transactions for raw directory readers.

Three formats, one idea

Delta Lake: an ordered transaction log, file actions and checkpoints, with reader/writer protocol features. A practical candidate where the chosen Spark/Databricks or delta-rs stack supports the required operations. Path-based access is possible, but catalog integration and newer features can impose additional requirements.

Apache Iceberg: table metadata points to snapshots, manifest lists and manifests. The atomic metadata publication is implemented by a catalog or table-operations integration; it need not mean a separately hosted catalog server in every setup. Evaluate engine-specific reads and writes, delete-file support, schema/partition evolution and catalog compatibility. A vendor supporting “Iceberg” does not imply every operation or format version is interoperable.

Apache Hudi: a timeline, file groups and indexes supporting record-level update workflows. Copy-on-Write rewrites base files for affected groups; Merge-on-Read stores changes in logs and merges them during snapshot reads, with periodic compaction. Read-optimised queries can have different freshness from snapshot queries. Hudi's update-oriented design is a reason to evaluate it for CDC, not proof it always wins throughput.

Iceberg schema and partition evolution preserve identity

This explanation uses the Iceberg 1.9.2 documentation and its tagged specification's format-v2 rules; it does not assume every reader implements later v3 features. A schema field has an ID independent of its name and position. Renaming field 2 from customer to buyer keeps ID 2, so historical data still reads as the same field. Dropping it and later adding a different customer assigns a new ID; old ID-2 values must not reappear under the new field. An optional added field absent in an old v2 file reads as null under ordinary projection, not as a computed business backfill. An ID-preserving int → long promotion is supported; int → string is not an arbitrary schema-evolution shortcut.

Partition specifications have their own IDs. Suppose spec 0 partitions by day and spec 1 partitions by hour. Existing files keep spec 0; subsequent writes can use spec 1. A query for 10:15–10:45 must consider the matching day in old manifests and the matching hour in new manifests, then apply the row predicate. Looking only in the new hourly directories loses historical rows. Changing the spec is metadata evolution, not compaction or a rewrite of old data. A future authorised reader/writer acceptance fixture should contain both layouts, a renamed field and an older snapshot; query equality against an unpartitioned reference matters more than a product's support badge.

Deletes are part of the reader contract

In Iceberg format v2, a position delete identifies a data-file path and zero-based row position. An equality delete identifies values by field IDs: a delete on order_id = 2 can match rows in applicable files, without recording their physical row positions. Multiple equality columns are ANDed; a null delete value matches a null field under the specified delete semantics, not SQL NULL = NULL.

Applicability is not merely “apply every delete to every file.” The tagged specification requires data-sequence and partition scope checks. An equality delete applies to strictly older data sequence numbers with the same spec/partition, or globally when its delete spec is unpartitioned. A position delete can apply at an equal data sequence number and must match its file/position with the required partition scope. A delete at sequence 7 must not erase a same-key row inserted at sequence 8 merely because the key matches. Compaction must materialise the correct logical rows and commit valid metadata, not discard delete files before their effects are preserved.

Delta v3.3.0's deletion-vector feature instead records row positions to disregard for a particular data file through its log metadata. It is not the same encoding as Iceberg equality deletes or Hudi logs. A reader must support the table's required protocol/features; opening the underlying Parquet with a generic reader can resurrect logically deleted rows or include superseded files. The pinned delta-rs example below uses its reviewed rewrite-based delete API, not a deletion-vector integration test.

Worked compatibility decision: choose an engine set only after checking the table format version, catalog/commit integration, read and write features, delete support and schema/partition evolution. If a writer enables a feature an older consumer cannot interpret, stop that rollout or upgrade the consumer through its own tested process. “Can read Parquet” and “can query a clean append-only sample” do not establish safe interoperability for deletes.

Layers of a lakehouse write

A SQL/DataFrame API produces a query plan; the writer serialises data and statistics; the table layer constructs a valid metadata change; commit coordination publishes it; readers apply the protocol to obtain a live set. Statistics can prune files, but their completeness and precision vary. Engines still perform decoding, filtering, joins, shuffles and scheduling. File skipping is important, not the entire performance story.

A transaction log resembles a fold over add/remove events, but Iceberg is not simply a sequential JSON log and Hudi has additional timeline/file-group semantics. Treat “Kafka for table state” and “Git for tables” as memory aids, not wire specifications.

Deriving atomic publication and retained history

Suppose a batch writes 200 files independently. A raw listing between uploads can expose 83 files; all 83 are individually valid but the intended table update is incomplete. Moving membership into committed metadata allows readers to choose one complete set. The commit point may be an atomic create, rename or catalog compare-and-swap under the implementation's guarantees; it is not intrinsically one particular S3 request.

Concurrent compactions illustrate why conflict validation matters. Both writers plan to replace p1. After one removes it, the other's assumption that p1 is still live is false. Retrying without validation could publish contradictory membership. An independent append may be safely rebased if its assumptions remain true.

Old metadata alone cannot resurrect old files. A historical query requires the metadata chain/checkpoint, its referenced files, schema/protocol support and permissions. A version number helps identify a training input; reproducing a model also needs all input-table versions, feature/code dependencies, seeds and environment. Time travel is not a backup strategy.

Copy-on-write, merge-on-read, or append plus dedup

Copy-on-write shifts work to writers and simplifies snapshot reads, but a small logical update may rewrite large touched files. Readers still need table-protocol support; plain Parquet scans can include superseded data. Merge-on-read can reduce immediate rewrite work but adds reconciliation and maintenance. Hudi logs, Iceberg delete files and Delta deletion vectors are distinct mechanisms, not interchangeable encodings.

Append-only history is simple when immutable events are the domain. A separate current-state projection must order changes by key, handle ties/tombstones and be shared rather than reinvented by every consumer. Appends can still conflict on schema or metadata, and retaining personal history needs a lifecycle policy.

Choose according to read latency, change frequency, engine compatibility, update distribution and maintenance capacity. Monitor file count/size distribution, planning time, metadata growth, commit conflicts, reader lag and compaction debt. No fixed average file size or vendor-neutral default wins every workload.

(c) Hands-on · 25 min

A version-pinned delta-rs reference

This unexecuted reference targets deltalake==0.17.4 APIs inspected in the official tagged source and uses PyArrow tables directly. A fresh TemporaryDirectory owns all writes and cleanup; it never deletes a shared /tmp/lakehouse_demo folder. Dependency compatibility must be resolved before an authorised future runtime run. The review installed nothing.

# service-reference: delta-rs python-v0.17.4 (not executed)
import json
from pathlib import Path
from tempfile import TemporaryDirectory
import pyarrow as pa
from deltalake import DeltaTable, write_deltalake
 
 
def rows(records):
    return pa.Table.from_pylist(records, schema=pa.schema([
        ('order_id', pa.int64()), ('customer', pa.string()),
        ('amount_cents', pa.int64()), ('day', pa.string()),
    ]))
 
 
def main():
    with TemporaryDirectory(prefix='lakehouse-review-') as root:
        table = Path(root) / 'orders'
        write_deltalake(str(table), rows([
            dict(order_id=1, customer='alice', amount_cents=4200, day='mon'),
            dict(order_id=2, customer='bob', amount_cents=1950, day='mon'),
            dict(order_id=3, customer='carol', amount_cents=8800, day='mon'),
        ]), mode='overwrite', partition_by=['day'])
        write_deltalake(str(table), rows([
            dict(order_id=4, customer='dave', amount_cents=1200, day='tue'),
            dict(order_id=5, customer='eve', amount_cents=6675, day='tue'),
        ]), mode='append', partition_by=['day'])
        # Fixed trusted fixture predicate, not interpolation of user input.
        DeltaTable(str(table)).delete(predicate="customer = 'bob'")
        for version, expected in [(0, [1, 2, 3]), (1, [1, 2, 3, 4, 5]),
                                  (2, [1, 3, 4, 5])]:
            snapshot = DeltaTable(str(table), version=version)
            ids = sorted(snapshot.to_pyarrow_table()['order_id'].to_pylist())
            assert ids == expected
            print(version, ids)
        print(DeltaTable(str(table)).history())
        for path in sorted((table / '_delta_log').glob('*.json')):
            actions = [json.loads(line) for line in path.read_text().splitlines()]
            print(path.name, [list(action)[0] for action in actions])
        # Inspect candidates only; retain all safety defaults.
        print('vacuum candidates:', DeltaTable(str(table)).vacuum(dry_run=True))
 
 
if __name__ == '__main__':
    main()

Expected logical versions are 0: three orders, 1: five, 2: four after deleting Bob. There is no version 3 in this three-commit fixture; the original outcome was off by one. SQL engines may expose VERSION AS OF 2; delta-rs selects version=2 in its constructor. A logical delete is not proof of GDPR erasure while files/history/backups remain.

Anatomy of the script

The first write creates metadata/protocol and data-add actions. The append adds new data without intentionally rewriting existing orders; commits may contain additional bookkeeping, not “only add actions.” The tagged delete API rewrites affected files without matching records. Versioned reads reconstruct table membership; raw log inspection explains it, but hand-editing logs is not a supported repair. Checkpoints are binary Parquet even though JSON commit entries are readable.

For another engine, perform a supported, pinned read while the temporary directory is alive:

# service-reference: Polars Delta reader; not executed or version-certified
import polars as pl
# Insert inside main's TemporaryDirectory block; `table` must still exist.
filtered = pl.scan_delta(str(table)).filter(pl.col('amount_cents') > 2000).collect()
print(filtered)

Expected current IDs satisfying that predicate are 1, 3 and 5. Openness means a published contract can be implemented by multiple engines, not identical runtime features or zero integration work.

Break time travel safely in a model

The original exercise disabled retention safeguards and vacuumed real files. The same learning objective is demonstrated without destructive storage calls. This exact model separates metadata from physical content, rejects a stale conflicting rewrite and demonstrates an unreadable historical snapshot after simulated retention expiry.

# exact-program: snapshot_retention
class Table:
    def __init__(self):
        self.files = {'p1': (1, 2, 3)}
        self.versions = [frozenset({'p1'})]
 
    def commit(self, base, removes, adds):
        current = self.versions[-1]
        if base != len(self.versions) - 1 or not set(removes) <= current:
            raise ValueError('re-read and validate assumptions')
        if set(adds) & self.files.keys():
            raise ValueError('immutable file identity reused')
        self.files.update(adds)
        self.versions.append((current - set(removes)) | set(adds))
 
    def read(self, version):
        return sorted(x for name in self.versions[version] for x in self.files[name])
 
    def expire_unreferenced(self, retained_versions):
        retained_versions = set(retained_versions) | {len(self.versions) - 1}
        keep = set().union(*(self.versions[v] for v in retained_versions))
        self.files = {k: v for k, v in self.files.items() if k in keep}
 
if __name__ == '__main__':
    t = Table()
    t.commit(0, set(), {'p2': (4, 5)})
    t.commit(1, {'p1'}, {'p3': (1, 3)})
    assert [t.read(v) for v in range(3)] == [[1,2,3], [1,2,3,4,5], [1,3,4,5]]
    try:
        t.commit(1, {'p1'}, {'p4': (99,)})
    except ValueError:
        pass
    else:
        raise AssertionError('stale writer accepted')
    t.expire_unreferenced({1, 2})
    assert t.read(0) == [1,2,3]  # a retained snapshot still references p1
    t.expire_unreferenced({2})
    assert t.read(2) == [1,3,4,5]
    try:
        t.read(0)
    except KeyError:
        print('old metadata remains but required data is gone')
    else:
        raise AssertionError('deleted data reconstructed')

This model has no concurrency, clock-based retention, crash recovery, real Parquet or cloud commits. Its exception is a teaching signal, not a promise that delta-rs raises the same Python exception for missing files.

(d) Production reality · 15 min

Small files, compaction and metadata bloat

Assume twelve five-MB files per minute: 17,280 files per day and 518,400 over thirty days before maintenance. Those are arithmetic inputs, not a customer incident. More files can increase planning, opens and metadata overhead; statistics need not require opening every footer because formats retain metadata. Measure rather than assert four-minute planning or an eightfold speedup.

Data compaction combines/reorganises data files; manifest rewriting reorganises metadata; snapshot expiration/VACUUM retires eligible historical references/data. They solve different problems. Delta OPTIMIZE target sizes are configurable and engine-dependent, not universally 128 MB. Iceberg rewriteDataFiles is different from rewriteManifests. Expiration must respect retained branches/tags and snapshots; orphan cleanup has its own safety horizon for in-progress writes.

Concurrency, update latency and retention failures

Replace the unverified Netflix/Uber stories with three concrete diagnoses. A raw reader sees half a batch: require format-aware snapshot access and supported commit coordination. Frequent small corrections cause rewrite amplification: compare CoW with supported MoR/delete-vector approaches and include read cost. A long-running reader fails after cleanup: set data and metadata retention beyond required query/stream lag, rollback and audit windows; do not bypass protection to satisfy a space target.

A nightly maintenance workflow

For an assumed fifty-million-row/day stream, first measure file sizes, metadata growth, conflict rate and oldest active/lagging reader. Select partitions for compaction/clustering based on observed reads. Validate the new snapshot's schema, row counts and business invariants; log statistics alone cannot prove arbitrary DQ rules. Record the committed version and actual lineage dependencies. Expire only history outside the approved horizon and no longer protected by readers/references; preview deletion candidates and verify recoverability. A failed check may require an authorised rollback, not merely a pointer flip that makes all downstream side effects disappear. External manifests are needed only for integrations that require them, not as a universal final step.

Where this shows up in the rest of the plan

Spark can be an engine; DQ validates populations rather than treating any commit as fresh business data; governance controls access and lifecycle; offline features require point-in-time joins in addition to snapshots. A capstone may choose a warehouse or lakehouse according to requirements rather than being forced into one substrate.

(e) Recall + stretch · 10 min

Explain-out-loud test

Explain membership versus physical presence, pick two concrete format/workload matches with compatibility caveats, and distinguish compaction, metadata maintenance and garbage collection.

Series route and pacing for S052

Continue to S053 · Data Quality — freshness, volume, schema, distribution. These are the numbered-series navigation pointers; the hub just below supplies the broader route, not evidence that a linked lesson has been completed or reviewed here.

Use the series' intuition → visual explanation → hands-on → operational caveats → recall sequence as a study route. The original 130-session framing describes the planned series, not a completion claim. Its 90-minute duration and the retained section-time labels are suggested pacing, not measured reading or execution times; optional preparation and setup may take additional sessions. Operational examples are teaching material, not verified production war stories or proof of deployment readiness.

What comes next

Quality asks whether the committed table contains the right population; governance asks who may use it and how long to keep it. Hub: The 6-Month Learning Plan.

Sources and review boundary

Delta protocol v3.3.0, delta-rs writer python-v0.17.4, delta-rs table API, Iceberg 1.9.2 reliability, maintenance, Hudi 1.0.2 table types. Iceberg 1.9.2 evolution and the tagged specification govern the added field-ID, partition-spec and delete examples. Prior captures were re-extracted and decisive passages reread on October 10; exact quotations and hashes are in foundations-batch-33. No actual storage cleanup or service execution is claimed.