Data Platform — Cost Optimisation, Governance, Lineage

Why this session matters

Part of a 24-topic learning series on engineering, ML, and LLM systems. Each session is a 90-minute deep-dive on one topic — designed so anyone can pick it up cold. Every two topics are followed by a revision session with recall prompts and hands-on drills.

Part 1: Petabyte Cost Optimisation — Compression, Partitioning, Z-Order, File Sizing

Why this session matters

Petabyte-scale data is cheap to store and ruinously expensive to query badly. The difference between a $50k/month and$500k/month cloud bill is rarely the data volume — it's the file sizing, the partitioning, the compression choice. The unsexy decisions that compound.

Agenda

• Storage cost vs query cost — the real money is in scans
• Partitioning — how to pick a partition column without regret
• File sizing — the small-file problem and how to keep it dead
• Compression — Snappy, ZSTD, LZ4, GZIP; columnar layout matters
• Z-Order, liquid clustering, sorted-by — the post-partition optimisations

Pre-read (skim before the session)

• Databricks — Best practices for Delta Lake
• Apache Parquet — File format spec
• Iceberg — Performance tuning
• Brooklyn Data — Lakehouse cost optimization

Deep dive

1. Where the money actually goes

In a typical cloud lakehouse for a 1 PB workload:

The lesson: optimise compute first, storage second. Storage at $20/TB/month for 1 PB is$20k. Compute can easily be $200k. Same workload.

2. Scan amount = cost

Every query reads N TB from storage. Per query cost ≈ scan-bytes × $ per byte. Reducing scan-bytes is your primary lever.

Three knobs:

• Pruning — read only relevant files (partitioning + min/max statistics).
• Projection pushdown — read only relevant columns (columnar format).
• Predicate pushdown — apply filters at the file/row-group level.

All three need the storage layer to cooperate (Parquet/ORC + manifest stats).

3. Partitioning — get this right

A partition is a folder hierarchy on disk. s3://bucket/orders/event_date=2026-07-01/region=eu/. Files inside are filtered by the planner before any read.

Rules:

• Partition by the column most filters use (almost always a date).
• Cardinality 100–10,000 partitions total per table is the sweet spot.
• Cardinality 1M+ partitions → metadata catastrophe (Hive metastore meltdown, planner overhead).
• Don't over-partition — year/month/day/hour for cold data = millions of folders, terrible.

Bad: partition by user_id (cardinality millions). Good: partition by event_date (cardinality 365 per year). Better with Iceberg/Delta: partition by bucket(user_id, 32) — hash bucket; bounded cardinality.

4. The small-file problem

When ingestion writes one tiny file every few seconds, you get millions of 50 KB files. Symptoms:

• Listing the partition takes minutes.
• Scan throughput collapses (each file = one fixed cost + a tiny read).
• Per-file metadata bloat.

Cure:

• Compact periodically — OPTIMIZE (Delta), rewrite-data-files (Iceberg), or a scheduled merge job.
• Aim for 128–512 MB target file size.
• Stream + batch writers — append small files now, compact within minutes.

5. File-size tuning

Too small: too many files, planner choked. Too big: parallelism collapses; one slow reader stalls the job.

Sweet spot for analytical queries: 128–512 MB per Parquet file. Spark default spark.sql.files.maxPartitionBytes=128M is a sane starting point.

For low-latency point lookups: smaller (32–64 MB) sometimes better.

6. Parquet — what's inside

Per file:

• Magic header.
• Row groups (default ~128 MB) — independently readable; have their own statistics.
• Within row group: column chunks; within column chunks: pages.
• Footer with metadata: schema, row group locations, statistics.

Reading is push-down friendly: planner skips entire row groups via stats, then skips pages within them. Wide schemas with hundreds of columns are cheap if you only select 3.

7. Compression choice

Modern default: ZSTD level 3. Excellent ratio, ~80% of Snappy speed on decompress. Saves real money on storage and egress.

8. Column ordering

Sort row groups by frequently-filtered columns. Min/max statistics become tight → pruning becomes effective.

-- Delta
OPTIMIZE orders ZORDER BY (customer_id, event_ts);

-- Iceberg
ALTER TABLE orders WRITE ORDERED BY customer_id, event_ts;

Z-Order: multi-dimensional clustering using a space-filling curve. Filters on either dimension still prune well. Useful when you have 2–4 filter columns of similar importance.

Linear sort: better for 1 dominant filter.

Liquid clustering (Databricks 2024+): automatic, learns from query patterns; less manual tuning.

9. Bloom filters and zone maps

• Zone maps / min-max indices — built into Parquet footer. Free.
• Bloom filters — opt-in per column; great for high-cardinality equality lookups (user_id = ?).
• Bitmap indices — Iceberg / Hive variants; less common.

Adding bloom filters on user_id for a per-user analytics table can take a query from 30s scan to 200ms.

10. Cost monitoring

Instrument:

• Query history with bytes-scanned per query.
• Per-team / per-pipeline rollup.
• Top-10 expensive queries weekly review.
• Alert on bytes-scanned regression after a deploy.

Snowflake, Databricks, BigQuery all expose this. Surface it to query authors via comment in the SQL — -- This query scanned 4.2 TB ($2.10).

11. Tiering

• Hot — last 30 days; standard storage; high query volume.
• Warm — 30–365 days; lower-cost storage class (S3 IA, GCS Nearline); 1.5× retrieval cost.
• Cold — > 1 year; archival (Glacier, Coldline); minutes-hours to thaw.

Auto-tier via lifecycle policies. Query layer should be tier-aware so warm/cold reads warn the user.

12. Vacuum and retention

Delta/Iceberg/Hudi keep old data files for time travel. Without VACUUM:

• Storage grows monotonically forever.
• Old files clog the manifest.

Default retention: 7 days for Delta. Set explicitly per-table based on time-travel needs (regulatory: months; ephemeral: days).

13. Reality check

A petabyte-scale cost-optimisation checklist:

1. Partition by date with hash bucket for high-cardinality joins.
2. Target 256 MB Parquet files; weekly OPTIMIZE.
3. ZSTD level 3 compression.
4. Z-Order or liquid clustering on the top-2 filter columns.
5. Bloom filters on PK-style columns used in equality queries.
6. Lifecycle policy: hot 30d, warm 90d, cold 1yr+.
7. Weekly query-cost review; chargeback by team.
8. VACUUM with 7-day retention (unless regulated).

Doing all of this on a fresh 1 PB pipeline routinely cuts compute spend 50–70%. The work pays back in the first month.

Reading material

Books:

• The Data Engineering Cookbook — Andreas Kretz (free; the partitioning + file-sizing chapters)
• Fundamentals of Data Engineering — Joe Reis & Matt Housley (the cost engineering and storage tier chapters)
• Designing Data-Intensive Applications — Martin Kleppmann (the storage and retrieval chapter; columnar layout from first principles)

Papers:

• Dremel: Interactive Analysis of Web-Scale Datasets — Melnik et al. 2010 (VLDB) — the paper that defined modern columnar query execution (became BigQuery).
• The Snowflake Elastic Data Warehouse — Dageville et al. 2016 (SIGMOD) — Snowflake's micro-partition + pruning architecture.
• Apache Parquet Specification — the format every cost optimisation depends on understanding.
• Lakehouse: A New Generation of Open Platforms — Armbrust et al. 2021 (CIDR) — the Databricks paper that named "lakehouse."

Official docs:

• Apache Parquet documentation — the file-format-spec source of truth.
• Apache Iceberg — Performance — partitioning, compaction, hidden partition transforms.
• Delta Lake — Best practices — Databricks' canonical cost-tuning guide.
• Databricks — Liquid clustering — the post-partitioning era.
• BigQuery — Cost optimization best practices — the canonical "stop scanning that table" reference.
• Snowflake — Designing for cost — clustering keys, micro-partition pruning, materialized views.

Blog posts:

• Maxime Beauchemin — Functional Data Engineering — partitioning + immutability as the foundation for cost control.
• Uber Engineering — Achieving 5x throughput and 2x cost reduction with Iceberg — exabyte-scale cost optimisation in production.
• Netflix Tech Blog — Incremental Processing using Iceberg — how Netflix migrated 100PB+ to Iceberg.
• Databricks — Auto Optimize — the small-file problem solved automatically.
• Tobias Macey — Data Engineering Podcast (cost optimisation episodes) — long-form interviews with practitioners.

In-depth research material

• Apache Parquet — github.com/apache/parquet-format — the spec repo; required reading once.
• Apache Iceberg — github.com/apache/iceberg — ~6k ★, the open table format Netflix, Apple, LinkedIn standardised on.
• Delta Lake — github.com/delta-io/delta — ~7.7k ★, the Databricks table format.
• Apache Hudi — github.com/apache/hudi — ~5.4k ★, the Uber-born table format.
• DuckDB — github.com/duckdb/duckdb — ~24k ★, the in-process analytical query engine that revealed how much modern compute is overkill.
• ClickHouse — github.com/ClickHouse/ClickHouse — ~38k ★, the columnar OLAP engine; canonical reference for "compression matters more than indexes."
• Uber Engineering — Cost Efficiency @ Scale in Big Data File Format — the petabyte-scale write-up.
• Pinterest Engineering — Saving $5M+ per year on EMR + S3 storage costs — the real-numbers cost-optimisation story.
• Spotify Engineering — Cost Insights tool: how Spotify saved millions in cloud cost — the FinOps culture write-up.
• Snowflake — Designing high-performance tables with clustering keys — the Snowflake-side equivalent of Z-Order/liquid clustering.
• Z-Order curve — Wikipedia — the math that makes multi-column data skipping possible.

Videos

• Cost optimization for data engineering at scale — Tobias Macey (Data Engineering Podcast LIVE) — 44 min — the practitioner-focused walkthrough; the right place to start.
• Apache Iceberg: An Architectural Look Under the Covers — Ryan Blue (Tabular) — Ryan Blue (Iceberg co-creator) · 56 min — the format and its cost-saving primitives from the creator.
• Delta Lake Internals: Liquid Clustering — Denny Lee (Databricks) — 28 min — the modern alternative to partitioning.
• Parquet: More Than Just a File Format — Julien Le Dem (Apache Arrow co-creator) — 47 min — why columnar compression actually works.
• Slay the Small File Problem — Databricks — 34 min — the canonical "compaction strategy" talk.

LeetCode — Largest Rectangle in Histogram

• Link: https://leetcode.com/problems/largest-rectangle-in-histogram/
• Difficulty: Hard
• Why this problem: Find the maximum scan-window you can afford given file sizes — same shape as the rectangle area maximisation.
• Time-box: 30 minutes. Look up the editorial only after.

Assignment / Deliverables

Give yourself a clean 2-hour window and complete all of these before moving on:

1. Read the deep-dive above end-to-end — no skimming. Take notes in your own words.
2. Solve the LeetCode problem below without help first. Only look at the hint after 15 focused minutes; only look at editorial after 30. Log your time.
3. Reproduce one code snippet locally. Pick the snippet that felt least obvious and get it running in a scratch file / notebook.
4. Draw the core diagram from memory. Paper, whiteboard, or tldraw — doesn't matter. If you can't, re-read section 2 and try again.
5. Write a 3-line takeaway in your prep journal: what surprised you, what you still don't understand, what you'd read next.
6. Skim one item from the Reading material section. Bookmark the rest for the weekend.
7. Commit any code + notes to your prep repo with message session-NN: <one-line summary>.

Stretch (optional, +30 min): explain today's topic to a rubber duck / a friend / a voice note. If you can't teach it in 5 minutes, you don't own it yet — flag it and revisit next weekend.

Post-session checklist

By the end of this session you should be able to:

• Explain why compute dominates a petabyte bill, not storage.
• Pick a partition column that lands in the 100–10k cardinality sweet spot.
• Diagnose and fix a small-file problem with OPTIMIZE / compaction.
• Choose between Snappy / ZSTD / LZ4 by use-case.
• Apply Z-Order / liquid clustering for multi-dimension filters.
• Solve largest-rectangle-in-histogram — monotonic stack, same area-maximisation flavor.

Generated from sessions_data.py + content_part*.py. To edit a video / leetcode / title, edit the data file and re-run write_sessions.py.

Part 2: Data Governance — Lineage, Quality, Catalogs, Contracts, Observability

Why this session matters

Pipelines that run are easy. Pipelines that you trust six months later, that other teams trust, that regulators trust — that's governance. The data-mesh / lakehouse era turned this from a back-office checkbox into a first-class engineering discipline.

Agenda

• Lineage — how, why, and what column-level lineage gives you
• Data quality — tests, expectations, anomaly detection
• Catalogs — DataHub, OpenMetadata, Unity, Glue
• Data contracts — the producer-side schema agreement
• Data observability — SLAs, SLOs, freshness, the 5 pillars

Pre-read (skim before the session)

• Monte Carlo — 5 Pillars of Data Observability
• Andrew Jones — Data Contracts: A Practical Guide
• DataHub — Why metadata matters
• Great Expectations — Core concepts

Deep dive

1. Why governance, why now

The volume of data in a typical company has tripled in 3 years. The number of producers and consumers has 10x'd. The result: silent breakage everywhere.

Symptoms:

• "Why are revenue numbers different in Dashboard A and B?"
• "Production ran a query against users.email; the column was dropped 6 months ago."
• "This ML model trained on a buggy join — we shipped predictions for a quarter."
• "Auditor asks: which models touched PII X in 2025?"

Governance = answer those questions before they're asked.

2. The five pillars of data observability

Tools (Monte Carlo, Bigeye, Soda, Anomalo) instrument these automatically. You can DIY 80% with dbt tests + alerts.

3. Lineage — the spine of governance

Lineage = the graph of "this column came from those columns came from that source".

Levels:

• Dataset-level — orders_fact came from orders_raw + dim_customer.
• Column-level — total_revenue in orders_fact = SUM(unit_price * qty) from orders_raw.
• Transformation-level — the SQL statement that produced each column.

Use cases:

• Impact analysis — "I want to drop this column; what breaks?"
• Root cause — "Dashboard is wrong; trace back to source."
• Audit — "Show me all data lineage touching PII."

Generate from: SQL parsing (sqlglot, dbt graph), runtime hooks (OpenLineage), or job orchestrator (Airflow, Dagster). Don't try to maintain by hand.

4. Data quality testing

The dbt-style minimum:

models:
  - name: orders_fact
    columns:
      - name: order_id
        tests: [not_null, unique]
      - name: customer_id
        tests:
          - not_null
          - relationships:
              to: ref('dim_customer')
              field: customer_id
      - name: total
        tests:
          - dbt_utils.accepted_range:
              min_value: 0
              max_value: 1000000

Tiers of tests:

• Existence — column present, not null.
• Uniqueness — primary keys.
• Referential integrity — foreign keys exist in dim.
• Range / value — accepted values, min/max.
• Distribution — mean, p95, cardinality within X% of last week.
• Business invariant — revenue = SUM(item.price * item.qty).

Wire to alerting. A failing test that nobody sees is theatre.

5. Great Expectations / Soda / dbt tests

• dbt tests — run during transformations; SQL-based. Free, in-pipeline. Best for shape/value checks.
• Great Expectations — Python-based; richer expectations; can run on any data source; heavier setup.
• Soda — YAML-driven; cloud + OSS; nice UI for tracking violations.
• Anomalo / Monte Carlo — ML-driven anomaly detection on top of metadata.

Start with dbt tests. Add GE/Soda if you need cross-source tests or need to test outside the warehouse.

6. Data contracts

The producer agrees what the schema and semantic looks like. The consumer trusts it.

A contract specifies:

• Schema — column names, types, nullability.
• Semantics — "amount is always in USD cents, integer".
• SLAs — freshness, completeness.
• Owners — who to ping when broken.
• Change policy — additive only? deprecation window?

Tooling:

• Define in YAML or JSON.
• Producer's CI fails if a change violates the contract.
• Versioned in git; reviewed like API contracts.

dataset: orders
owner: orders-team
schema:
  - name: order_id
    type: string
    description: uuid
sla:
  freshness: 30m
  completeness: 99.9%

Push the validation upstream. Test the producer's output, not the consumer's input.

7. Catalogs — discoverability layer

DataHub, OpenMetadata, Unity Catalog, AWS Glue Data Catalog. Capabilities:

• Auto-crawl warehouses, lakes, dashboards.
• Show schemas, sample data, top users.
• Capture lineage (often via OpenLineage).
• Glossary — business definitions tied to physical tables.
• Tags (PII, PCI, restricted).
• Reviews / ratings — humans annotate trust.

For a startup: DataHub or OpenMetadata (OSS, self-hostable). On Databricks: Unity Catalog. On AWS: Glue.

8. Owners and stewards

Every dataset needs an owner (engineer who maintains) and a steward (business person who defines). Without ownership:

• No one fixes broken pipelines.
• No one approves schema changes.
• No one answers "what does customer_tier_v3 mean?"

Bake owner into the catalog. Page the owner when their dataset SLA breaks.

9. PII, classification, masking

Tag columns with sensitivity (Public, Internal, Confidential, PII, PCI). Enforce via:

• Row-level security — different users see different rows.
• Column-level security — analysts see hashed email, support sees plain.
• Dynamic data masking — same query, different output by role.
• Lineage propagation — if PII flows into a downstream table, downstream is automatically tagged.

Modern warehouses (Snowflake, BigQuery, Databricks) support all of this natively. Use it.

10. Data observability vs data quality

• Quality — a set of rules: "X must be > 0", "Y must not be NULL".
• Observability — the running picture: what is happening across freshness, volume, schema, distribution, lineage.

You need both. Quality catches what you can articulate. Observability surfaces the surprises you didn't think to test for.

11. Incidents and postmortems for data

Treat data outages like service outages:

• Severity levels.
• On-call rotation for critical datasets.
• War room / Slack channel during incident.
• Postmortem with action items.
• Public dashboard of dataset health.

Cultural shift. Data engineers learn from ops; ops doesn't tolerate "the dashboard is wrong, we'll fix it Monday".

12. Reality check

A first-90-days governance plan:

1. Top-10 critical datasets — explicit owners, contracts, SLAs.
2. dbt tests on every Gold model — basic shape + value.
3. OpenLineage emission from Airflow/Dagster → DataHub.
4. PII tags on user-facing tables.
5. Slack alerts on failed tests, missed SLAs.
6. Weekly review of broken-test trends.

You don't need a Monte Carlo subscription on day 1. You do need the discipline.

Reading material

Books:

• Driving Data Quality with Data Contracts — Andrew Jones (2023; the canonical book on data contracts)
• Fundamentals of Data Engineering — Joe Reis & Matt Housley (the governance / DataOps chapter)
• Data Management at Scale, 2nd ed. — Piethein Strengholt (the enterprise data-mesh + governance book)
• The DAMA Guide to the Data Management Body of Knowledge (DMBOK), 2nd ed. — DAMA International (the encyclopaedia; reference, not cover-to-cover)

Papers:

• Goods: Organizing Google's Datasets — Halevy et al. 2016 (SIGMOD) — Google's internal catalog paper; the origin of modern lineage thinking.
• Apache Atlas: Data Governance Made Possible — Hortonworks — the Hadoop-era catalog; still influential.

Official docs:

• OpenLineage spec — the open standard for emitting lineage from any tool.
• dbt — Data tests — unique, not_null, relationships, accepted_values; the starter kit.
• Great Expectations docs — the OG Python data-validation library.
• Soda Core docs — the YAML-driven alternative.
• DataHub documentation — LinkedIn's open-source catalog.
• Databricks Unity Catalog — the lakehouse-native catalog + access control.

Blog posts:

• Chad Sanderson — Data Contracts: The Mesh in Practice — the canonical practitioner essay series.
• Monte Carlo — What is Data Observability? The 5 Pillars — freshness, distribution, volume, schema, lineage.
• Airbnb Engineering — Data Quality at Airbnb (Midas) — the certification programme that defined a generation of DQ.
• Lyft Engineering — Amundsen — the discovery / catalog system that inspired DataHub.

In-depth research material

• OpenLineage — github.com/OpenLineage/OpenLineage — ~1.9k ★, the cross-tool lineage standard supported by Airflow, dbt, Spark.
• DataHub — github.com/datahub-project/datahub — ~10k ★, LinkedIn's metadata + lineage + quality platform.
• Amundsen — github.com/amundsen-io/amundsen — ~4.5k ★, Lyft's discovery engine.
• OpenMetadata — github.com/open-metadata/OpenMetadata — ~6k ★, modern unified metadata.
• Marquez — github.com/MarquezProject/marquez — ~1.9k ★, the OpenLineage reference store.
• Great Expectations — github.com/great-expectations/great_expectations — ~10k ★, the canonical Python data validation framework.
• Soda Core — github.com/sodadata/soda-core — ~1.9k ★, declarative YAML data checks.
• Uber Engineering — DataMesh: How Uber transformed data governance — the petabyte-scale data-mesh implementation.
• Netflix Tech Blog — Metacat: Making big data discoverable and meaningful — Netflix's federated metadata service.
• Convoy / Chad Sanderson — Shifting Left on Data Quality (Data Council talk) — the contract-driven approach at Convoy.

Videos

• Data Contracts: From Theory to Implementation — Chad Sanderson — Chad Sanderson · 41 min — the canonical talk on data contracts.
• Data Mesh: Principles and Logical Architecture — Zhamak Dehghani — Zhamak Dehghani · 44 min — the creator of the data mesh paradigm explaining it.
• Data Observability: The Next Frontier of Data Engineering — Barr Moses — Monte Carlo CEO · 27 min — the 5 pillars in a single talk.
• Apache Atlas: Data Governance for Hadoop — Hortonworks — 32 min — the Hadoop-era origin of modern catalog tools.
• Building a Self-Service Data Platform at Netflix — Joey Lynch — 47 min — how Netflix scaled discovery + quality across thousands of users.

LeetCode — Graph Valid Tree

• Link: https://leetcode.com/problems/graph-valid-tree/
• Difficulty: Medium
• Why this problem: Verify acyclic + connected — the structural invariant of lineage graphs.
• Time-box: 30 minutes. Look up the editorial only after.

Assignment / Deliverables

Give yourself a clean 2-hour window and complete all of these before moving on:

1. Read the deep-dive above end-to-end — no skimming. Take notes in your own words.
2. Solve the LeetCode problem below without help first. Only look at the hint after 15 focused minutes; only look at editorial after 30. Log your time.
3. Reproduce one code snippet locally. Pick the snippet that felt least obvious and get it running in a scratch file / notebook.
4. Draw the core diagram from memory. Paper, whiteboard, or tldraw — doesn't matter. If you can't, re-read section 2 and try again.
5. Write a 3-line takeaway in your prep journal: what surprised you, what you still don't understand, what you'd read next.
6. Skim one item from the Reading material section. Bookmark the rest for the weekend.
7. Commit any code + notes to your prep repo with message session-NN: <one-line summary>.

Stretch (optional, +30 min): explain today's topic to a rubber duck / a friend / a voice note. If you can't teach it in 5 minutes, you don't own it yet — flag it and revisit next weekend.

Post-session checklist

By the end of this session you should be able to:

• Name the 5 pillars of data observability.
• Explain dataset- vs column-level lineage and one use-case for each.
• Write 4 dbt tests covering existence, uniqueness, referential integrity, range.
• Sketch a data contract YAML with schema, SLA, and owner.
• Choose between dbt tests, Great Expectations, and a SaaS observability tool.
• Solve graph-valid-tree — union-find or BFS for tree validation (lineage graph sanity check).

Generated from sessions_data.py + content_part*.py. To edit a video / leetcode / title, edit the data file and re-run write_sessions.py.