systemsadvanced 25m read

Designing for Scale · The Data Lakehouse

Why the data warehouse and data lake converged. Moving from expensive, proprietary compute-storage monoliths to open table formats like Iceberg, Hudi, and Delta Lake.

The problem this post solves

For two decades, data engineering was defined by a strict boundary. The Data Warehouse (structured, fast, expensive, SQL) sat on one side. The Data Lake (unstructured, slow, cheap, files) sat on the other.

This forced a terrible compromise. If you wanted the speed and ACID transactions of a warehouse, you had to pay exorbitant vendor storage costs. If you wanted the cheap storage of a lake, you lost the ability to run simple SQL updates or guarantee consistency across files.

This post covers the architectural convergence of the two: the Data Lakehouse. It breaks the storage-compute monolith apart, allowing you to run warehouse-speed queries directly on cheap, open-format object storage.

First principles

From first principles

Start with the question

Why did Data Lakes fail to replace Data Warehouses?

1
A Data Lake is just a collection of files (like Parquet or CSV) sitting in an object store (like S3).
forced by · It is designed purely for cheap, infinitely scalable storage.
2
If you want to update a single record in a 1GB Parquet file, you must rewrite the entire 1GB file.
forced by · Object stores do not support in-place row updates.
3
While you are rewriting the file, someone else might be reading it, or a concurrent job might also be updating it.
forced by · Data Lakes have no built-in transaction manager.
4
Therefore, Data Lakes cannot provide ACID guarantees or handle streaming updates safely.
forced by · Without a transaction layer, concurrent writes corrupt the data.

⇒ Therefore

To get warehouse features on a data lake, you must add a metadata layer that provides transactions over immutable files.

Architecture: The Lakehouse and Open Table Formats

The defining feature of a Data Lakehouse is the separation of compute and storage, bound together by an Open Table Format (like Apache Iceberg, Apache Hudi, or Delta Lake).

The Metadata Layer

The metadata layer is not a database engine. It is a set of transaction logs and manifest files that sit alongside your data files in the object store.

When a query engine (like Spark) wants to read a table, it does not scan the raw Parquet files. It reads the metadata.

Mental modelThe Table Format as a Ledger

A library where the books (Parquet files) are scattered everywhere. Instead of walking the aisles to find a book, you look at the master ledger (the metadata). The ledger tells you exactly which books belong to 'Version 5' of the database, and exactly which shelves they are on.

If a writer wants to update data, they write a completely new book.
They then add a single line to the ledger saying 'Ignore book A, read book B instead'.
Readers currently reading book A are unaffected. Readers starting a new query read the ledger and see book B.

🔔 Fires when you see

Use to explain how ACID transactions are possible on immutable object storage.

How it solves the Update Problem

If we need to delete a user's data for compliance (GDPR), the traditional data lake approach requires scanning petabytes of data, finding the specific Parquet files containing that user, downloading them, removing the row, rewriting the files, and uploading them. This is wildly expensive.

Modern table formats use Merge-on-Read or Copy-on-Write.

In a Copy-on-Write approach, the engine rewrites the entire file containing the deleted row, and updates the metadata pointer. This is slow for writes, but fast for reads.

In a Merge-on-Read approach, the engine writes a tiny "delete file" indicating that row ID 42 has been deleted. When a query runs, the engine reads the main file and the delete file, and merges them in memory, dropping row 42. This is extremely fast for writes, but requires the read compute engine to do slightly more work.

The tradeoff

Choosing the Storage Architecture

Traditional Data Warehouse (Redshift, BigQuery)

+ you gain Turnkey. No setup required. Highest performance for highly structured SQL workloads.

− you pay Vendor lock-in. Compute and storage are tightly coupled in billing. You cannot query the underlying files with a different engine.

pick when When the team is small and the primary goal is building dashboards quickly without managing infrastructure.

Data Lakehouse (Iceberg/S3 + Compute Engine)

+ you gain Zero vendor lock-in. Storage costs are pennies on the dollar. You can point Spark at the data for ML, and point Trino at the same data for dashboards.

− you pay Requires dedicated data engineering to manage file compaction, metadata cleanup, and compute clusters.

pick when Default for enterprise scale. When data volume makes proprietary storage costs prohibitive.

Time Travel and Schema Evolution

Because the metadata layer tracks every change as an atomic commit, Lakehouses get features previously reserved for advanced databases.

Time Travel: You can query the table exactly as it looked last Tuesday. SELECT * FROM sales FOR SYSTEM_TIME AS OF '2026-08-01'. The metadata layer simply points the query engine to the list of Parquet files that were active on that date. This makes machine learning reproducibility trivial.

Schema Evolution: In a traditional data lake, renaming a column breaks all historical data because the old Parquet files have the old column name. In a Lakehouse, schema changes are tracked in the metadata. The engine knows how to map the old column name in file A to the new column name in file B at read time.

Failure modes

The Small File Problem. Stream processing engines often write data in tiny batches (e.g., a few kilobytes every minute). Object stores and Parquet files are optimized for massive sequential reads (hundreds of megabytes per file). If a Lakehouse accumulates millions of tiny files, the metadata overhead alone will crush the query engine. A background process (Compaction) must constantly merge small files into large ones.

Metadata Bloat. Tracking every transaction creates a massive metadata history. If you do not run regular maintenance jobs (Vacuuming) to delete old snapshots and unreferenced data files, the metadata tree itself becomes too large to parse quickly.

What to carry forward

Key points

You can now

Explain why standard object storage cannot support ACID transactions natively.
Describe how a metadata ledger allows atomic commits over immutable files.
Defend the Lakehouse architecture against traditional Data Warehouses on the basis of vendor lock-in and cost.
Identify the Small File Problem and explain why Compaction is mandatory.