S050 · Orchestration — Airflow, DAGs, Retries, Backfills

Module M05: Data Engineering · Session 50 of 130 · Track: DE 🎼

What you'll be able to do after this session

• Explain Orchestration to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S047 — Spark — RDD, DataFrame, Jobs/Stages/Shuffles

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · Getting Started with Apache Airflow: Introduction & DAG Creation — the fastest "what is Airflow, why do I need it" primer.
• 🎥 Deep dive (30–60 min) · Apache Spark Architecture Explained — Driver, Executors, DAGs, Stages & Tasks — a DAG-execution mental model that transfers directly to Airflow's scheduler.
• 🎥 Hands-on demo (10–20 min) · Batch Processing vs Stream Processing Explained — grounds why batch orchestration is the default answer.
• 📖 Canonical article · Apache Airflow — Tutorials — official, up-to-date tutorial: your first DAG in ~15 minutes.
• 📖 Book chapter / tutorial · Hands-on Apache Airflow 3.0 Tutorial (Start Data Engineering) — modern hands-on walkthrough with Docker Compose.
• 📖 Engineering blog · Airbnb — Airflow, a workflow management platform — the original announcement from where Airflow was born. Read for the why.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: Scheduling pipelines like an adult.

You have 30 data jobs. Job B needs Job A to succeed first. Job C needs both A and B. Some jobs must run at 3am, some hourly, some only on Mondays. Some fail on flaky APIs and need to be retried. When one fails, downstream jobs should not run. On weekends you want to re-run last Tuesday's jobs because you found a bug. You could do all this with cron and a lot of bash. You would regret it in about 2 weeks. Orchestration is the discipline of saying "here is the graph of my jobs, their dependencies, their schedules, and what to do when things break" in a single declarative place — and Apache Airflow, born at Airbnb in 2015, is the de-facto standard.

An Airflow DAG (Directed Acyclic Graph) is a Python file that says: "these tasks, in this order, run on this schedule." The Airflow scheduler decides which tasks are ready to run right now, an executor hands them to workers, and a lovely web UI shows you the whole story — every task, every run, every retry, every log. When a task fails, Airflow retries per your policy, waits, then marks the run failed and blocks downstream tasks. When you need to reprocess history (a "backfill"), Airflow can re-run the DAG for any past date range, and each task run is parameterized by that date so it produces the right historical output.

The gotcha to carry forever: Airflow is an orchestrator, not a data processor. Do not compute anything inside a DAG file — a DAG file is code that runs every 30 seconds on the scheduler to decide what to schedule. Import heavy libraries, hit databases, or call APIs at parse time and your scheduler falls over. The right pattern: DAGs are thin definitions of "run this container / this SQL / this Spark job on this schedule with these deps." The actual work happens on separate compute (Spark, dbt-core, containers, cloud services). Airflow just conducts.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

A tiny DAG:

extract_orders and extract_customers run in parallel (no deps). join_orders_customers waits for both. publish_dashboard and trigger_alerts again run in parallel after build_metrics. Airflow enforces this ordering automatically — you just declare upstream >> downstream.

Core primitives:

Worked example — timeline of a daily pipeline:

day 1 03:00  scheduler wakes → DAG "sales_daily" ready → task extract runs → succeeds → clean runs → fails → retry after 5m → succeeds → aggregate → publish → alert
day 2 03:00  same thing, with "logical date = day 1" (yesterday's data)
day 3 09:00  you found a bug in clean(). fix code. run:
             airflow dags backfill sales_daily -s 2026-06-01 -e 2026-06-15
             Airflow re-runs 15 daily invocations in order (or parallel, if set)

Airflow's brilliant trick: each task gets an execution_date (Airflow 2.x calls it logical_date) — a parameter representing "which slice of time this run is for." So extract becomes extract WHERE created_at BETWEEN {{ ds }} AND {{ next_ds }} — the same DAG code works for today and for a 6-month-old backfill.

Anti-pattern (do not do this):

# BAD — this runs on the scheduler every ~30 seconds
customers = pd.read_sql("SELECT * FROM customers", conn)  # <- 500 MB pulled at parse time!
for c in customers:
    PythonOperator(task_id=f"process_{c.id}", ...)

The fix: dynamic task mapping (task.expand(...)) or a single task that iterates internally.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

Fire up Airflow with Docker and write your first real DAG.

mkdir airflow-lab && cd airflow-lab
curl -sSLO 'https://airflow.apache.org/docs/apache-airflow/stable/docker-compose.yaml'
mkdir -p ./dags ./logs ./plugins ./config
echo -e "AIRFLOW_UID=$(id -u)" > .env
docker compose up airflow-init
docker compose up -d
# Web UI: http://localhost:8080  (login: airflow / airflow)
sleep 30

Now drop this DAG into ./dags/sales_daily.py:

from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.operators.bash import BashOperator
 
default_args = {
    "owner": "chinni",
    "retries": 2,
    "retry_delay": timedelta(minutes=1),
    "email_on_failure": False,
}
 
def extract(**ctx):
    ds = ctx["ds"]                                    # e.g. '2026-07-23'
    print(f"[extract] pulling orders for {ds}")
    # Real code: SELECT * FROM orders WHERE created_at::date = %s
    ctx["ti"].xcom_push(key="row_count", value=1234)
 
def clean(**ctx):
    n = ctx["ti"].xcom_pull(task_ids="extract", key="row_count")
    print(f"[clean] cleaning {n} rows")
    if datetime.now().second % 5 == 0:               # flaky on purpose
        raise RuntimeError("simulated flaky failure")
 
def build_metrics(**ctx):
    print(f"[metrics] aggregating for {ctx['ds']}")
 
with DAG(
    dag_id="sales_daily",
    default_args=default_args,
    start_date=datetime(2026, 7, 1),
    schedule="@daily",
    catchup=False,                                    # don't backfill on turn-on
    tags=["s050", "learning-series"],
) as dag:
 
    extract_t  = PythonOperator(task_id="extract",       python_callable=extract)
    clean_t    = PythonOperator(task_id="clean",         python_callable=clean)
    metrics_t  = PythonOperator(task_id="build_metrics", python_callable=build_metrics)
    publish_t  = BashOperator(  task_id="publish",       bash_command="echo published for {{ ds }}")
 
    extract_t >> clean_t >> metrics_t >> publish_t

Then trigger it manually from the UI (sales_daily → play button) and watch the Graph, Grid, and Logs tabs.

Observe (checklist):

1. Green squares in the Grid view = successful runs; red = failed. Click any cell → Log → see stdout of that task instance.
2. When clean fails, Airflow waits 1 minute, then retries. After 2 retries fail, the task is marked failed and downstream build_metrics is skipped.
3. {{ ds }} in the BashOperator is a Jinja template evaluated at task run time with the logical date — this is how backfills work.
4. XCom values pushed by extract are visible in the Admin → XComs view (do not put large blobs here).
5. The Gantt chart tab shows critical-path timing — where your DAG spends its wall-clock.

Try this modification: run a backfill for 5 past days:

docker compose exec airflow-scheduler airflow dags backfill sales_daily -s 2026-07-15 -e 2026-07-19

Watch the Grid view fill with 5 new dated columns, all executed in dependency order. Note that each {{ ds }} is the historical date, not today.

Cleanup: docker compose down -v

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

Airflow's biggest anti-pattern in the wild: doing real work in DAG parse time. A pandas read, a REST call, an API listing — all executed every ~30 seconds when the scheduler re-parses the DAG file. This crashes the scheduler at 10-100 DAGs. Every serious Airflow shop bans I/O at parse time in code review. The rule: DAG files are declarations, tasks are actions.

XCom is not a data pipe. It's serialized JSON in the metadata DB, capped at ~48 KB by default. Passing DataFrames or file contents through XCom is a design smell — the answer is to write to blob storage (S3, ADLS, GCS) and pass the path through XCom. This is called the "pass-by-reference" pattern and every mature Airflow team enforces it.

Timezone handling burns everyone once. Airflow 2 defaults execution_date to UTC. If you're in Hyderabad computing "yesterday's data," {{ ds }} at 03:00 IST is today in UTC by 3 hours. Always be explicit: set dag.timezone and think in the timezone your business owners think in. Airbnb's own post-mortem said this cost them a full quarter of reporting errors before they standardized.

Retries need idempotency. If a task partially wrote 100k rows and failed on row 100k+1, the retry will double-write those first 100k rows. Every DE-facing task should be idempotent: INSERT ... ON CONFLICT DO UPDATE, MERGE, or "drop-and-rewrite the partition for this date." This is why partitioned tables and dbt's incremental models exist — they make idempotency easy.

Airflow at scale is operationally heavy. Netflix, Uber, and Airbnb all built teams around it. The 2020s alternatives — Prefect, Dagster, Kestra, Argo Workflows, Temporal — each solve some Airflow pain (Dagster: asset-centric + strong typing; Prefect: dynamic tasks; Temporal: durable workflows for services). For a starting DE team in 2026 the "safe default" is still Airflow (biggest community, most operators) or dbt-core + a lightweight scheduler, but keep an eye on Dagster if you're greenfield.

Finally: the scheduler is a single point of failure in Airflow < 2.2. Modern Airflow supports multi-scheduler HA. Run at least two. Managed offerings (MWAA on AWS, Cloud Composer on GCP, Astro) handle this for you at a cost. At Microsoft internally we ran managed Airflow for exactly this reason — the operational overhead of running Airflow yourself at scale is real.

(e) Quiz + exercise · 10 min

1. In one sentence, what problem does Airflow solve that a set of cron jobs cannot?
2. What is a DAG's execution_date / logical_date and why is it crucial for backfills?
3. Give one anti-pattern of doing real work inside a DAG file (not inside a task).
4. Why must every retryable task be idempotent, and give one SQL pattern that makes it so.
5. What is XCom for, and what is it explicitly not for?

Stretch (connects to S047 — Spark): you have a nightly Spark job orchestrated by Airflow that reads yesterday's partition, does a big aggregation, and writes back. One morning the job silently produced empty output — Airflow shows it as success. What layered checks would you add (in the DAG, in Spark, in the warehouse) so this never happens again undetected?

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is Orchestration? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S051): dbt — Models, Tests, Docs, Warehouse-Native ELT
• Previous (S049): Stream Processing — Watermarks, Windows, Exactly-Once
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.