Data Platform & Orchestration

Models get the attention; the platform underneath decides whether any of it works. This series walks the data backbone in build order — the event spine first, then scheduling, then the query layer for telemetry, then how they compose into something that feeds ML and experimentation.

Who this is for

You've written pipelines that work on your machine and now need them to survive schedules, retries, backfills and other people. You can write SQL and Python. You do not need distributed-systems theory going in — the parts that matter get built up as they're needed.

---

The path

Part 1 · The event spine (~5 min, intermediate)

Kafka 101 for ML Engineers

Topics, partitions and consumer groups — deliberately only the parts that matter when ML is downstream. Partition count is the decision you can't easily undo, and consumer-group semantics are what determine whether your feature pipeline double-counts under a restart.

Start here. Almost everything later assumes events arrive from somewhere.

Part 2 · Orchestration (~7 min, intermediate)

Apache Airflow — Open Source Orchestration Engine

Architecture, how DAGs express dependencies, and the operational reality: retries, backfills, and what actually happens when a task fails at 3am. The mental shift is that a DAG is a contract about ordering, not a script.

Prerequisite: part 1 helps but isn't strictly required.

Part 3 · The query layer (~11 min, intermediate)

Exploring Azure Data Explorer and Best Practices

A self-sufficient deep-dive on ADX/Kusto: architecture, the KQL language, and the ingestion and partitioning choices that determine whether queries over large telemetry return in a second or a minute.

Part 4 · Composition (~33 min, intermediate)

Data Infrastructure for AI & Experimentation at Scale

Where the previous three become one system. The data backbone powering ML, personalisation and experimentation — feature pipelines, the batch/streaming split, and how experimentation gets its measurements without corrupting them.

Prerequisite: parts 1–3. This is the payoff post.

Part 5 · The wider view (~76 min, intermediate)

Engineering Clarity — Data, Distributed Systems and AI

A long-form study companion: internals, decision trees, code and reasoned Q&A. Treat it as a reference to return to rather than a single sitting.

---

How to read this series

Start at part 1 and move forward. Every post has prev/next navigation at the bottom, so you can walk the whole track without coming back here.

If you only have an hour: parts 1 and 2 cover the two things you'll touch every single day.

What this series does not cover

• Spark and distributed compute internals — covered in the 6-month plan, module M05.
• Warehouse modelling (dimensional, Data Vault, OBT) — also M05.
• Vendor comparisons — this series teaches one concrete stack rather than surveying all of them.

Open the full series →