Subject / Published lessons
data engineering
Explore the ideas, work through an example, and save a lesson to return to.
Filters · 1 active
27 articles
Recently updated firstThe Relational Model — Tables, Keys, Normalisation
Design keys, dependencies and historical facts; normalise worked schemas and test real SQLite constraints.
18 min readSQL Basics — SELECT, WHERE, ORDER BY, LIMIT
Query rows safely with explicit NULL, alias, ordering and pagination semantics, verified against exact SQLite results.
16 min readJoins — INNER, LEFT, RIGHT, FULL, Anti-Join
Choose join semantics, preserve measure grain and compare physical algorithms without fan-out or NULL mistakes.
17 min readAggregations — GROUP BY, HAVING, Subqueries
Compute honest grouped metrics, exact medians and weighted rollups; distinguish missing populations and approximation contracts.
16 min readWindow Functions — the Game-Changer
Rank deterministically, reason about peers and frames, and compute calendar-aware deltas and session boundaries.
17 min readCTEs & Recursive Queries
Compose named SQL stages and safely traverse graphs with explicit cycle, reachability and truncation semantics.
17 min readIndexes — B-Tree Intuition, When to Add
Design and validate query-driven indexes; distinguish access paths, output work, visibility and concurrent-build safety.
16 min readTransactions & ACID — Isolation Levels, MVCC
Protect transaction invariants with precise isolation, retry and durability contracts; test local rollback and bounded anomaly models.
18 min readQuery Planning — EXPLAIN, Execution Plans, Tuning
Read PostgreSQL plan nodes and per-loop accounting; distinguish row estimates, buffers and output work from measured SQLite access paths.
20 min readNoSQL Landscape — KV, Document, Column, Graph
Compare key-value, document, wide-column and graph access models with query-specific consistency, indexing and traversal contracts.
20 min readData Modelling — Dimensional, Data Vault, OBT
Define fact grain, design star schemas and model historical dimensions with explicit keys, interval validity and Type 6 current-value semantics.
20 min readBatch vs Streaming — Mental Model & Use Cases
Choose bounded or unbounded processing using freshness, event-time progress, late-data handling and consistent recovery contracts.
20 min readSpark — RDD, DataFrame, Jobs/Stages/Shuffles
Trace Spark plans through jobs, stages, tasks and exchanges; reason about dependencies, skew, broadcast and driver-memory limits.
20 min readKafka — Topics, Partitions, Consumer Groups
Reason about partition ordering, replication acknowledgements, consumer offsets and transactional effects without assuming end-to-end exactly-once behavior.
20 min readStream Processing — Watermarks, Windows, Exactly-Once
Define event time, windows, progress and state cleanup; preserve timestamp meaning and specify atomic sink effects and replay boundaries.
20 min readdbt — Models, Tests, Docs, Warehouse-Native ELT
Structure staging, intermediate and mart models with explicit grain; Use ref, source, macros and generic/singular test contracts; Choose materialisations and reconcile late changes, updates and deletes.
20 min readLakehouse — Delta / Iceberg / Hudi, ACID on Files
Explain atomic snapshot membership and validated commit retries; Trace Delta versions and distinguish compaction from retention; Compare Delta, Iceberg and Hudi compatibility and maintenance.
20 min readData Quality — Freshness, Volume, Schema, Distribution
Check eligible freshness, volume, schema, validity and distribution; Distinguish unknown checks from passing data and route by consequence; Design consumer-specific publication gates and seasonal baselines.
20 min readGovernance & Cost — Lineage, PII, Attribution
Trace column dependencies and enforce classification separately; Reconcile direct, shared and unallocated cost against a declared total; Design justified retention and verifiable authorised erasure workflows.
20 min readApache Airflow - open source orchestration engine
Design retry-safe Airflow tasks without mistaking DAG ordering for a database transaction. A source-bounded lesson with worked reasoning, failure analysis and explicit runtime limitations.
19 min readData Infrastructure for AI and Experimentation at Scale
Trace events, time-valid features, recommendation decisions and experiment outcomes through a source-reviewed data platform with executable SQL and statistical fixtures.
55 min readOverall Engineering Clarity — Data, Distributed Systems and AI (Deep Dive)
Trace compute, commit, indexing and authorization boundaries across a data/AI platform. Mechanisms, worked examples, failure analysis and complete practice answers.
119 min readData Platform & Orchestration
A data platform is a chain of independently testable contracts. A source-bounded lesson with worked reasoning, failure analysis and explicit runtime limitations.
8 min readDesigning for Scale · Real-Time Analytics
Provisional and final aggregates must agree on ownership. A source-bounded lesson with worked reasoning, failure analysis and explicit runtime limitations.
9 min readDesigning for Scale · Change Data Capture (CDC)
Build CDC and transactional-outbox reasoning around coordinated snapshots, durable intent, replay identity, schema evolution and WAL retention.
12 min readDesigning for Scale · The Data Lakehouse
Explain atomic table snapshots over immutable files, writer conflicts, row deletes, schema identity, maintenance and measured architecture trade-offs.
12 min readOrchestration — Airflow, DAGs, Retries, Backfills
Orchestration coordinates dependencies, scheduling, retries and historical runs. It does not make arbitrary writes atomic or correct. This lesson targets **Airflow 2.10.5** and preserves the full daily-order pipeline, sensor, retr
20 min read