· Topic
Data Engineering
Spark, Kafka, lakehouses and petabyte pipelines. Build data that engineers can trust.
Roadmap
stage 1
Foundations
stage 2
Core patterns
stage 3
Advanced
stage 4
In production
Articles
- intermediate 55m
Governance & Cost — Lineage, PII, Attribution
The unglamorous but career-defining half of data engineering: knowing where every row came from, who owns it, which team's budget pays for it, and how to prove nothing sensitive leaked. A working playbook for lineage, PII discovery, and cost attribution.
- intermediate 55m
Data Quality — Freshness, Volume, Schema, Distribution
Trust is a bag of four numbers: is the data late, missing, wrong-shaped, or weird? Learn the four checks every serious pipeline runs, wire them into a real DAG, and see the outages they catch before executives do.
- intermediate 55m
Lakehouse — Delta / Iceberg / Hudi, ACID on Files
How Parquet-on-object-storage grew a transaction log and killed the data warehouse monopoly. Delta, Iceberg, and Hudi compared with real code, real failure modes, and the three-line commit that changed data engineering.
- intermediate 55m
dbt — Models, Tests, Docs, Warehouse-Native ELT
The tool that turned the data team into a software team. Learn dbt's core primitives — models, sources, tests, macros — and why ‘transform in the warehouse’ won the last decade.
- intermediate 55m
Orchestration — Airflow, DAGs, Retries, Backfills
Every pipeline eventually becomes an ‘orchestration problem’. Learn Airflow's mental model — DAG, task, operator, XCom — and the four ideas that separate a stable warehouse from a 3am pager: retries, idempotence, backfills, SLAs.
- intermediate 55m
Stream Processing — Watermarks, Windows, Exactly-Once
The three ideas that separate ‘I read from Kafka in a loop’ from ‘I run a real stream processor’: watermarks for time, windows for scope, exactly-once for correctness. Flink and Spark Structured Streaming demystified.
- intermediate 55m
Kafka — Topics, Partitions, Consumer Groups
The distributed commit log that eats the world. Learn what Kafka actually is (a durable log, not a queue), why partitions are the unit of parallelism, and how consumer groups let a thousand consumers share the load without losing a message.
- intermediate 55m
Spark — RDD, DataFrame, Jobs/Stages/Shuffles
The most misunderstood distributed engine in the world. Learn what actually happens when you write .groupBy().agg() — the DAG, the shuffle, the stages, the spills — and why 90 % of Spark performance is avoiding shuffles.
- intermediate 55m
Batch vs Streaming — Mental Model & Use Cases
Not ‘which is better’ — a mental model for when latency matters more than throughput, when boundedness matters more than freshness, and why every modern platform is really both at once.
- intermediate 55m
Data Modelling — Dimensional, Data Vault, OBT
Kimball vs Inmon vs One Big Table. The three modelling schools that dominate 30 years of data warehouses — when each wins, when each fails, and what modern cloud warehouses changed.
- intermediate 55m
NoSQL Landscape — KV, Document, Column, Graph
Four families, one truth: NoSQL means giving up something you had in Postgres for something you needed more. Learn which one to reach for and — more importantly — when not to.
- intermediate 55m
Query Planning — EXPLAIN, Execution Plans, Tuning
Every slow query is a lie the optimizer believed. Learn to read EXPLAIN plans, spot the six pathological patterns, and rewrite queries into shapes the planner can love.
- intermediate 55m
Transactions & ACID — Isolation Levels, MVCC
The single feature that lets banks, e-commerce, and every serious business run on a database. Learn ACID, the four isolation levels, MVCC, and why ‘SERIALIZABLE’ is not the answer you want.
- beginner 55m
Indexes — B-Tree Intuition, When to Add
The single biggest performance lever in any database. Learn the B-tree ‘book index’ intuition (~4 disk reads instead of 4 billion), the write tax you pay for every index, when the planner ignores the index (low selectivity, functions on the column, leading wildcards), the composite-index column-order rule, and the CREATE INDEX CONCURRENTLY story every senior engineer has learned from a locked-table outage.
- beginner 55m
CTEs & Recursive Queries
Compose complex SQL like Python functions. Learn WITH … AS as the SQL equivalent of extracting a helper, the analytics-engineering CTE-chain pattern that powers every dbt project, WITH RECURSIVE for traversing trees and graphs (org charts, folder structures, dependency graphs), the four-part recursive anatomy, and the depth-safety guard that stops runaway cycles from filling your disk.
- beginner 55m
Window Functions — the Game-Changer
Keep every row AND attach a group-wide computation. Learn PARTITION BY as ‘per-bucket without collapsing’, the RANK vs DENSE_RANK vs ROW_NUMBER tie-breaker table, the frame clause (ROWS vs RANGE) that silently changes running totals, LAG/LEAD for period-over-period, and why every senior SQL user reaches for a window function before a correlated subquery.
- beginner 55m
Aggregations — GROUP BY, HAVING, Subqueries
Turn a million rows into one insight. Learn the aggregate functions (COUNT/SUM/AVG/PERCENTILE), the GROUP BY mental model of ‘rows into buckets’, the WHERE-vs-HAVING trap that breaks half of all ad-hoc SQL, the three shapes of subqueries, and why COUNT(DISTINCT) on a billion rows is a code smell (reach for HLL).
- beginner 55m
Joins — INNER, LEFT, RIGHT, FULL, Anti-Join
Recombine what normalisation split apart. Five join flavours differ only in what to do with unmatched rows — but that single choice decides whether Carol appears in your customers report. Learn the visual model, the row-multiplication trap that makes marketing dashboards over-report by 40×, and the three join algorithms (nested-loop, hash, sort-merge) your query planner picks between.
- beginner 55m
SQL Basics — SELECT, WHERE, ORDER BY, LIMIT
The universal data language. Learn the five verbs that answer 90 % of single-table questions, the execution order that isn't the written order, three-valued logic with NULLs, the operators every dashboard uses, and the two mistakes (SELECT *, string-concat SQL) that cause the most production incidents.
- beginner 55m
The Relational Model — Tables, Keys, Normalisation
Codd's 55-year-old idea that still runs the world: split entities into tables, link by keys, store every fact exactly once. Learn primary vs foreign keys, the first three normal forms with worked examples, when to deliberately denormalise for OLAP, and the specific data bugs (update anomalies, natural-key collisions, missing FK integrity) that this discipline prevents.
- intermediate 8m
Data Platform & Orchestration
The data backbone that feeds ML and experimentation: Kafka as the event spine, Airflow for orchestration, Azure Data Explorer for telemetry at scale, and how the pieces compose into a platform.
- intermediate 32m
MS Stack Ch 12 — Kusto / KQL
Kusto Query Language: pipeline syntax, summarize, joins, time-binning, materialized views, KQL injection defence. The language powering Azure Data Explorer, App Insights, Log Analytics, and most Microsoft telemetry.
- beginner 5m
Kafka 101 for ML engineers
Topics, partitions, consumer groups — the parts of Kafka that actually matter when you put ML features behind it.
- advanced 16m
Taking the Azure Fabric Ignite Edition Challenges to Complete
Microsoft Learn Challenge conducting a challenge to get good in few of the challenges which are super useful to complete to gain knowledge on Microsoft Fabric.
- intermediate 15m
Exploring different services in GCP
Exploration and documentation of different services offered in GCP
- intermediate 11m
Exploring Azure Data Explorer and Best Practices
A self-sufficient deep-dive on Azure Data Explorer (ADX/Kusto) — architecture, the KQL language from zero to advanced, ingestion patterns, performance/cost levers, and operational best practices.