Apache Airflow - open source orchestration engine

Review boundary — 7 October 2026: This lesson combines corrected conceptual analysis with retained worked material. External-service and vendor-runtime examples are reference exercises, not executed deployments; synthetic numbers are assumptions, not benchmark results. The source notes identify the substantively checked claims.

Apache Airflow - Going deep

Background & Prerequisites — What You Need to Know Before Writing This Blog

A nightly training job is not just a call to a model library: new data must arrive, validation must pass, and evaluation must approve the result before registration. Airflow records and schedules those dependencies. Read the prerequisites below as a map of that workflow: DAGs describe the order, operators do the work, and XCom carries small results between tasks. The reference DAG later in the article ties them together using Airflow 2.x-style APIs.

1. Python Fundamentals (Intermediate Level)

Why: Airflow DAGs are written in Python. You need solid Python skills to write operators, hooks, and custom logic.

• Decorators — Airflow 2.x uses the @dag and @task decorators extensively. Understand how Python decorators wrap functions, accept arguments, and return callables.
• Context managers — Used for resource management (DB connections, file handles) inside tasks.
• Generators & iterators — Useful for processing large datasets in tasks without loading everything into memory.
• Type hints — In Airflow 2.10.5 a mapping return annotation can infer multiple_outputs=True. It does not make arbitrary objects serializable; XCom serialization is a separate backend contract.

2. Airflow Core Architecture

Why: You cannot write about Airflow without understanding its internal components and how they interact.

• Scheduler — Creates eligible DAG runs and submits ready tasks through its configured executor. The DAG processor parses and serializes DAG files; in 2.10.5 it may run as part of the scheduler or separately. Parsing a file does not execute its decorated task bodies, but top-level Python still executes, so keep network calls and expensive computation out of module scope.
• Webserver — Airflow 2.10.5 exposes a UI for inspecting runs, graph relationships, task states and logs. It is an operational view, not the task execution engine.
• Executor — The scheduler's configured task-execution mechanism, not necessarily a separate service. Choices have different isolation and operating costs:
  • SequentialExecutor — runs one task at a time; useful for a minimal learning environment, not a parallel-workload claim
  • LocalExecutor — runs tasks in local processes; plan machine resources and isolation explicitly
  • CeleryExecutor — distributes tasks across worker machines using Celery + a message broker (Redis/RabbitMQ)
  • KubernetesExecutor — launches task instances in pods; pod startup and cluster operations are costs, not a universal best choice
• Metadata Database — Stores workflow/task state and serialized DAG metadata. The default XCom backend also stores values there; connections and variables may instead resolve through environment variables or a secrets backend. Database-engine support depends on the selected executor and release.
• Workers — Processes that actually execute the tasks (relevant for Celery/K8s executors).

3. DAGs (Directed Acyclic Graphs)

Why: The central concept in Airflow — every workflow is a DAG.

• What is a DAG — A collection of tasks with directed dependencies and no cycles. If Task A → Task B → Task C, then C cannot depend back on A.
• DAG definition — Writing a Python file that instantiates a DAG object with dag_id, schedule, start_date, catchup, default_args.
• Schedule intervals — Cron expressions (0 2 * * *), presets (@daily, @hourly), timedelta objects, or dataset-triggered schedules (Airflow 2.4+).
• Catchup & Backfill — For a time-based schedule, catchup creates eligible unrun data intervals, subject to start/end bounds and scheduler concurrency. Backfill is a deliberate historical run operation. catchup=False does not make a task retry-safe and does not forbid all manual historical runs.
• DAG dependencies — Using TriggerDagRunOperator or Datasets to create cross-DAG dependencies.

4. Operators, Sensors & Hooks

Why: These are the building blocks of tasks in Airflow.

• Operators — Pre-built task templates:
  • BashOperator — runs shell commands
  • PythonOperator — runs Python callables
  • EmailOperator — sends emails
  • Provider-specific operators submit warehouse, storage or cloud-pipeline jobs. Pin the provider package and use its documented operator/import names; core Airflow 2.10.5 does not establish compatibility of every cloud provider release.
• Sensors — Special operators that wait for a condition:
  • FileSensor — waits for a file to appear
  • HttpSensor — polls an HTTP endpoint
  • ExternalTaskSensor — waits for a task in another DAG
  • Poke mode vs Reschedule mode — Poke holds the worker slot; Reschedule frees it between checks.
• Hooks — Interfaces to external systems (databases, APIs, cloud services). They handle authentication and connection management. Resolve a configured connection ID rather than embedding a credential in the DAG; the selected secrets backend may supply it instead of the metadata database.

5. TaskFlow API (Airflow 2.x)

Why: Modern way of writing Airflow DAGs that is cleaner and more Pythonic.

• @task decorator — Turns a Python function into an Airflow task with automatic XCom push/pull.
• @dag decorator — Defines the DAG as a decorated function.
• Automatic dependency inference — When you pass the output of one @task to another, Airflow infers the dependency automatically.
• XCom serialization — TaskFlow uses XCom to pass data between tasks. Understand size limits and serialization formats.

6. XComs (Cross-Communication)

Why: The mechanism for tasks to share data.

• Push/Pull — Tasks push values to XCom (return value or xcom_push()), others pull with xcom_pull().
• Limitations — The default backend stores XComs in the metadata database and the docs design them for small values, not dataframes. There is no universal safe 64 KB contract. Pass a stable URI, checksum, row count or small metric dictionary; keep bulk data in an appropriate external store.
• Custom XCom backends — The common-IO provider offers an object-storage backend with a configured path and byte threshold. Check its version and filesystem dependencies; switching storage does not remove serialization, security, retention or deserialization costs.

7. Connections, Variables & Secrets

Why: How Airflow manages credentials and configuration.

• Connections — Stored in the metadata DB or secrets backend. Each connection has a conn_id, type (e.g., aws, azure, postgres), host, port, login, password, and extras (JSON).
• Variables — Key-value pairs for configuration. Accessed via Variable.get('key').
• Secrets Backends — Airflow can pull connections/variables from AWS Secrets Manager, Azure Key Vault, HashiCorp Vault, etc.

8. ML Pipeline Integration with Airflow

Why: The blog's core goal — orchestrating ML training and monitoring.

• Data ingestion tasks — Pulling training data from external sources, validating schema.
• Feature engineering — Transformation tasks that prepare features.
• Model training — Triggering training jobs (local Python, Spark, or cloud ML services like Azure ML, SageMaker, Vertex AI).
• Model evaluation — Computing metrics, comparing against baseline, deciding whether to deploy.
• Model registry — Logging the trained model to MLflow, Azure ML, or a custom registry.
• Monitoring tasks — Checking model drift, data drift, performance degradation.
• Retraining triggers — Dataset-driven or scheduled retraining pipelines.

9. Deployment & Infrastructure

Why: Practical knowledge needed for running Airflow in production.

• Docker Compose — The 2.10.5 quick-start is a learning environment; its documentation explicitly does not provide production security guarantees. Use the matching release file in a disposable environment, with private bindings and no real secrets.
• Helm Chart — A supported deployment path with its own chart version, secrets, resource, database, upgrade and recovery decisions; a chart install is not operational acceptance.
• Managed Airflow — Evaluate a provider's supported Airflow/provider versions, network isolation, identity, quotas, upgrade policy and recovery responsibility. A managed control plane does not guarantee task idempotence.
• DAG deployment — Git-sync, CI/CD pipelines, or shared NFS.

Study contract and runtime boundary

The code below is an Airflow 2.10.5 TaskFlow reference with a tiny, fully specified local model calculation. Its pure functions are tested separately without installing or starting Airflow. The DAG, scheduler UI, sensor integration, hooks and external registry publication have not been executed. Use the later validation checklist to study those deployment boundaries; they are not reasons to discard the conceptual material.

Reference DAG — End-to-End ML Pipeline with TaskFlow API

This is a deterministic regression-training workflow, not a recommendation-quality result. Ingest creates six synthetic observations; validation checks the complete tiny input; training fits a line to the first four; evaluation compares holdout MSE on the final two against a training-mean baseline. Registration prepares a small approved manifest, not a real model-registry write. The numerical data and thresholds are teaching fixtures.

The pure functions intentionally sit outside the decorated DAG. That makes their contract independently testable. Defining those functions is cheap at import time; doing the training or fetching a dataset at module scope would not be.

# Airflow 2.10.5 reference: dags/train_tiny_model.py
from math import isfinite
import pendulum
from airflow.decorators import dag, task
from airflow.exceptions import AirflowSkipException
from airflow.operators.python import get_current_context
 
def fit_line(rows):
    if len(rows) < 2:
        raise ValueError("need at least two training rows")
    xs, ys = zip(*rows)
    mx, my = sum(xs) / len(xs), sum(ys) / len(ys)
    denominator = sum((x - mx) ** 2 for x in xs)
    if denominator == 0:
        raise ValueError("constant training input")
    slope = sum((x - mx) * (y - my) for x, y in rows) / denominator
    return {"slope": slope, "intercept": my - slope * mx, "baseline": my}
 
def score(model, rows):
    if not rows:
        raise ValueError("empty holdout")
    candidate = sum((model["slope"] * x + model["intercept"] - y) ** 2
                    for x, y in rows) / len(rows)
    baseline = sum((model["baseline"] - y) ** 2 for _, y in rows) / len(rows)
    return {"mse": candidate, "baseline_mse": baseline, "n": len(rows)}
 
@dag(
    dag_id="train_tiny_model",
    schedule="0 2 * * *",
    start_date=pendulum.datetime(2025, 1, 1, tz="UTC"),
    catchup=False,
    max_active_runs=1,
    default_args={"retries": 2, "retry_delay": pendulum.duration(minutes=5)},
    tags=["teaching", "synthetic"],
)
def train_tiny_model():
    @task(multiple_outputs=False)
    def ingest():
        context = get_current_context()
        return {
            "partition": context["data_interval_start"].isoformat(),
            "rows": [[x, 2 * x + 1] for x in range(6)],
        }
 
    @task(multiple_outputs=False)
    def validate(batch):
        rows = batch["rows"]
        if len(rows) != 6 or any(len(row) != 2 for row in rows):
            raise ValueError("unexpected fixture shape")
        if any(type(v) not in (int, float) or not isfinite(v)
               for row in rows for v in row):
            raise ValueError("nonfinite or nonnumeric value")
        return batch
 
    @task(multiple_outputs=False)
    def train(batch):
        return {"partition": batch["partition"],
                "model": fit_line(batch["rows"][:4]),
                "holdout": batch["rows"][4:]}
 
    @task(multiple_outputs=False)
    def evaluate(result):
        metrics = score(result["model"], result["holdout"])
        if not isfinite(metrics["mse"]) or metrics["mse"] >= metrics["baseline_mse"]:
            raise AirflowSkipException("no strict holdout improvement")
        return {"partition": result["partition"],
                "model": result["model"], "metrics": metrics}
 
    @task(multiple_outputs=False)
    def prepare_registration(result):
        # Small candidate manifest only; no registry, promotion or network write.
        return {"candidate_id": "tiny-line-v1:" + result["partition"],
                "model": result["model"], "metrics": result["metrics"]}
 
    prepare_registration(evaluate(train(validate(ingest()))))
 
training_dag = train_tiny_model()

Trace it: training sees (0,1), (1,3), (2,5), (3,7), so mean x is 1.5 and mean y is 4. The centered cross-product is 10 and squared-x sum is 5; slope is 2 and intercept 1. Holdout predictions are 9 and 11, so model MSE is zero. Predicting training mean 4 gives (25 + 49) / 2 = 37. The candidate passes the strict comparison. This tiny exact line neither estimates generalization uncertainty nor justifies automatic production promotion. A real gate needs representative holdouts, leakage controls, uncertainty and product guardrails.

TaskFlow mechanics: calls made while building the DAG return dependency/XCom references, not the computed dictionaries. Runtime returns are small JSON-compatible values. Setting multiple_outputs=False deliberately treats each dictionary as a single result rather than unpacking its keys. Task bodies use context only during execution. The scheduled interval, not the wall clock at retry, supplies the partition identity. A 02:00 daily cron interval is not automatically a midnight calendar-day partition; define the business interval explicitly.

External registration design: persist an immutable artifact and checksum, validate them, then conditionally create the registry entry keyed by candidate identity. A retry must return the existing identical result or raise a conflict on changed content. The function above only constructs the manifest and does not prove this external effect. Separate registering a candidate from routing production traffic to it; evaluation is not a rollout or rollback plan.

Architecture — How the Scheduler Drives This

The DAG describes what must precede what; this diagram explains who makes that happen. Trace a ready task from scheduler to executor to worker, then follow the recorded state into the UI. XCom represents the small dictionaries returned above. Only this six-row synthetic fixture passes data directly; a real training dataset stays in external storage behind a versioned manifest.

Feature transformation and a drift feedback loop

The five-task reference above deliberately keeps its original six-row calculation. Here is an explicit extension of its mechanism, not a claim that those five tasks already monitor production. Split first; fit feature parameters only on the training partition; transform both training and holdout with the same saved parameters. The scikit-learn 1.5 common-pitfalls chapter explains why fitting preprocessing on the test set leaks information.

For the raw feature x, use z = (x - training_mean) / training_scale, where scale is the population standard deviation of training x. The first four rows give mean 1.5, variance 1.25 and scale about 1.1180. The transformed training values are approximately -1.3416, -0.4472, 0.4472 and 1.3416. Holdout x=4 and x=5 become about 2.2361 and 3.1305 using those same parameters, not a newly fitted holdout mean. Fit the existing fit_line on these transformed rows; its slope becomes about 2.2361 and intercept 4, but holdout predictions remain 9 and 11. This linear rescaling is a teaching transformation, not a quality improvement. Save feature version, mean and scale with the model so serving applies the same transform.

# Pure CPU reference; use fit_line and score from the preceding reference block.
from math import isfinite, sqrt
 
def fit_feature_params(training_rows):
    if len(training_rows) < 2:
        raise ValueError("need at least two training rows")
    xs = [row[0] for row in training_rows]
    if any(type(x) not in (int, float) or not isfinite(x) for x in xs):
        raise ValueError("invalid feature")
    mean = sum(xs) / len(xs)
    scale = sqrt(sum((x - mean) ** 2 for x in xs) / len(xs))
    if not isfinite(scale) or scale == 0:
        raise ValueError("constant or invalid training feature")
    return {"version": "center-scale-v1", "mean": mean, "scale": scale}
 
def transform_rows(rows, params):
    if not isfinite(params["mean"]) or not isfinite(params["scale"]) or params["scale"] <= 0:
        raise ValueError("invalid transform parameters")
    if any(len(row) != 2 or any(type(v) not in (int, float) or not isfinite(v)
                              for v in row) for row in rows):
        raise ValueError("invalid row")
    return [[(x - params["mean"]) / params["scale"], y] for x, y in rows]
 
def drift_report(reference_counts, current_counts, threshold=0.20, minimum=1000):
    # Fixed, exhaustive categorical bins in the same order; never drop unseen values.
    if not reference_counts or len(reference_counts) != len(current_counts):
        raise ValueError("bin schema mismatch")
    if any(type(n) is not int or n < 0 for n in reference_counts + current_counts):
        raise ValueError("counts must be nonnegative integers")
    if not isfinite(threshold) or not 0 <= threshold <= 1 or minimum < 1:
        raise ValueError("invalid monitoring policy")
    nr, nc = sum(reference_counts), sum(current_counts)
    if min(nr, nc) < minimum:
        return {"state": "insufficient-data", "distance": None, "exceeded": False}
    distance = max(abs(a / nr - b / nc)
                   for a, b in zip(reference_counts, current_counts))
    return {"state": "measured", "distance": distance, "exceeded": distance > threshold}
 
def feedback_action(report, preceding_window_exceeded, request_pending=False):
    if report["state"] != "measured":
        return "investigate-data-arrival"
    if not report["exceeded"]:
        return "observe"
    if request_pending:
        return "attach-evidence-to-open-request"
    return "request-human-review" if preceding_window_exceeded else "watch-next-window"

Window/statistic/threshold: freeze a reference histogram from an approved 28-day baseline and compare each complete UTC day [00:00, next 00:00) after a two-hour lateness allowance. Use the same feature definition, eligible population and exhaustive bins in both. For this fixture the raw x bins are x < 2, x >= 2, plus a missing/invalid category tracked separately by validation; a newly introduced category requires an explicit bin-schema revision, not silent removal. The code accepts already validated counts, not raw event timestamps. A manifest must prove window bounds, deduplication and completeness before these counts enter it.

The statistic is the L-infinity distance between empirical bin frequencies: D = max_i |reference_i / reference_total - current_i / current_total|. Reference counts [500, 500, 0] and current counts [200, 800, 0] give D=0.30. At the deliberately chosen threshold strictly greater than 0.20, this window is flagged. Equal-threshold observations are not flagged. [490, 510, 0] gives 0.01; [20, 80, 0] is insufficient data under the 1,000-observation minimum, not evidence of no drift. These are policy fixtures, not calibrated statistical significance levels. More bins, seasonality and repeated tests change false-alert behavior; calibrate on stable historical windows and evaluate detection delay and false alarms. TFDV's guide illustrates categorical L-infinity drift/skew comparators and thresholds; this small Python implementation is not TFDV execution or its complete anomaly contract.

Where the tasks go: in a feature-aware variant, insert a prepare_features task between validate and train. Inside that task, call fit_feature_params(rows[:4]), transform both splits, and return their versioned manifest plus parameters. Train consumes transformed training data; evaluate consumes transformed holdout; registration includes the transform alongside the model. Bulk data remains external, with only manifest identities in XCom. A separate daily monitoring DAG reads the completed production-window manifest and the versioned baseline, computes drift_report, reads the immediately preceding complete window's flag for the same model/feature/baseline policy, then calls feedback_action. Missing windows break the consecutive-window streak. Persist the small report keyed by model, feature version, baseline version and interval; retries must upsert that same identity.

Feedback rather than automatic promotion: the first flag watches the next window; a second consecutive flag requests human review. An existing request absorbs new evidence rather than creating duplicate retraining jobs. Inspect ingestion/transform bugs and cohort changes first. If genuine drift warrants retraining, create an idempotent training request with a frozen new training snapshot; run validation, holdout comparison and candidate registration again. Only a separately approved rollout can change traffic. Data drift need not harm quality, and unchanged marginals do not rule out concept drift. When labels arrive, join predictions to outcomes by stable event identity and compute MSE for the same mature cohort (using the earlier score logic), recording label coverage and delay. A synthetic policy might investigate MSE above 1.5× a nonzero approved baseline for two mature windows; use an absolute budget when baseline MSE is zero. This is delayed performance monitoring, not proof of causation or a production threshold recommendation.

Exercise and solution: D=0.30 today and yesterday, no request pending: request review, not promotion. With only 100 observations today, investigate data arrival and do not count it as a second valid window. If serving refits its scaler every day, unchanged raw inputs can acquire different feature meaning; reuse the registered transform and repair that skew before proposing a new model.

A task needs a retry-safe external effect

This lesson targets Airflow 2.10.5; imports, SLA features and scheduling interfaces must be checked separately before porting to Airflow 3. A DAG file is parsed repeatedly, but not at one universal heartbeat interval. Avoid network access, expensive computation and large data loads at module scope. XCom carries small coordination values, not a substitute for an object store.

A scheduled run's data interval is the input contract. Read and write that partition rather than “latest” or datetime.now(), so retries and backfills do not silently switch inputs. Idempotence means repeating the same logical run has the intended single effect, not just obtaining the same return value. Stage outputs under a run identity, validate them, and atomically publish a manifest or replace the intended partition. A downstream task should consume only committed outputs. Pools limit pressure on dependencies; retries need bounded backoff and a policy for poison input. Airflow, Dagster, Prefect and Argo should be compared on deployment constraints, data/asset semantics, operational skill and integrations rather than a universal ranking.

Failure exercise and worked answer

A task uploads three of four files and fails. The retry writes all four with new random names. What did the DAG dependency graph fail to guarantee?

Operators, sensors and hooks in one dependency chain

TaskFlow is not mandatory. A PythonOperator invokes a configured Python callable and can return through XCom; a BashOperator invokes a shell command. The explicit operator style makes constructor options visible, while TaskFlow makes ordinary function data dependencies concise. Both still produce task instances scheduled by Airflow. Neither makes local files shared across different Celery workers or Kubernetes pods.

For a conceptual file-arrival pipeline, use FileSensor(task_id="wait", filepath="ready.json", fs_conn_id="landing", mode="reschedule", deferrable=False, poke_interval=60, timeout=600) before a processing task. This is the Airflow 2.10.5 core FileSensor contract, not a worker setup performed here. The connection identifies the base path accessible to the sensor worker. The producer should publish a validated ready manifest only after all data files are durable; file existence alone cannot prove a multipart upload complete. A FileSensor uses its filesystem hook to resolve the connection. With distributed workers, shared storage or an appropriate remote provider sensor/hook is needed; a file on the scheduler's disk may be invisible to workers.

poke holds a worker slot while waiting. reschedule gives it back between checks but still has scheduling/metadata overhead and poll latency. Deferrable operators use a different triggerer-based mechanism; mode="reschedule" does not make a sensor deferrable. Set a finite timeout and choose failure/skip behavior deliberately. A slow partner feed should not consume all worker slots or retry forever. Pools bound pressure on a dependency, while DAG/task concurrency limits bound a different dimension of work.

A traditional two-operator DAG, line by line

Save this separate Airflow 2.10.5 reference as dags/two_operator_demo.py. It deliberately uses a constant shell command, no credentials and no user-controlled Jinja interpolation. Constructing operators defines work; the callable and Bash command run later in a task process. No Airflow service execution is claimed here.

import pendulum                                      # 1
from airflow import DAG                              # 2
from airflow.operators.python import PythonOperator  # 3
from airflow.operators.bash import BashOperator      # 4
 
def add_counts(a, b):                                # 5
    return a + b                                    # 6
 
with DAG(                                           # 7
    dag_id="two_operator_demo",                     # 8
    start_date=pendulum.datetime(2025, 1, 1, tz="UTC"),  # 9
    schedule=None,                                  # 10
    catchup=False,                                  # 11
) as demo:                                          # 12
    count = PythonOperator(                         # 13
        task_id="count",                            # 14
        python_callable=add_counts,                  # 15
        op_kwargs={"a": 3, "b": 4},                  # 16
    )                                               # 17
    announce = BashOperator(                        # 18
        task_id="announce",                         # 19
        bash_command="printf '%s\\n' 'count task succeeded'",  # 20
        do_xcom_push=False,                          # 21
    )                                               # 22
    count >> announce                               # 23

1. Imports a timezone-aware datetime factory; no clock is read to choose the start date.
2. Imports the DAG context manager that associates enclosed tasks with a workflow.
3. Imports the operator that calls Python inside task execution.
4. Imports the operator that launches Bash inside task execution.
5. Defines a callable receiving two explicit arguments; defining it does not add them yet.
6. Returns 7 for the supplied 3 and 4. PythonOperator normally pushes the return value to XCom.
7. Opens a DAG constructor; this is graph-definition time, not task execution time.
8. Gives the DAG its unique discovery/UI identity.
9. Supplies a fixed UTC start boundary instead of a moving now() value.
10. Selects manual runs, so this demonstration creates no periodic schedule.
11. Disables schedule catchup; it is not an idempotence guarantee.
12. Enters the context; demo holds the DAG object and the following operators attach to it.
13. Constructs the first task, not a call to add_counts at import time.
14. Names that task count, unique within this DAG.
15. Passes the function object without parentheses; add_counts() would call it too early.
16. Supplies keyword arguments that Airflow passes to the callable at execution.
17. Closes the first operator constructor.
18. Constructs the second task, a BashOperator.
19. Names it announce, distinct from count.
20. Runs Bash's printf to emit one constant line and exit normally. The escaped newline belongs to the printf format. No arithmetic result is interpolated into the shell.
21. Disables publishing shell output to XCom; task logs are a separate output.
22. Closes that constructor.
23. Adds one directed edge: count upstream of announce. The default all_success trigger rule permits the shell task after the Python task succeeds, not after it fails.

Comparison: the explicit edge carries ordering, not the value 7. To consume that value, you must deliberately pull its XCom or use an appropriate templated field. In the earlier TaskFlow DAG, train(validate(ingest())) expresses value references and inferred edges together. Both forms need serialization contracts and retry-safe effects. A downstream shell command cannot assume an upstream worker's local file exists on its own worker. Practice: make add_counts raise an exception: announce must not run as a success notification. Changing the edge does not make two external writes transactional.

Secrets and operational notifications

A DAG references a connection ID, not a password embedded in source or printed into logs. In Airflow 2.10.5, lookup checks an enabled alternative secrets backend first, then environment variables, then the metastore; without an alternative backend the latter two apply. The UI shows metastore entries, not a complete list of values supplied by other backends. Diagnose a duplicate ID by checking that precedence without disclosing its credential. Pin the selected provider and configure its workload identity through the supported deployment mechanism. Keep ordinary configuration separate from secrets and avoid repeated Variable.get calls at DAG-import time, which can cause repeated backend/database access during parsing.

The earlier Slack-only leaf sketch was not a reliable whole-pipeline success/failure contract. one_failed evaluates upstream failures and does not wait for every upstream task to finish. A successful notification leaf can mask an upstream failure in the DAG run status, because Airflow 2.10.5 determines terminal run state from leaves. Do not replace the real terminal work with a success-only notification leaf. Task/DAG failure callbacks or an external monitor can send a deduplicated alert keyed by DAG/run/task/attempt while preserving the workload's failure status; pin and test that integration separately. An alert should include the run identity, failed stage and log link, not a credential or private training data.

A skipped evaluation is different from broken training: insufficient improvement is an expected no-promotion result here. Test upstream failure, evaluation skip and successful registration separately; do not infer them from a single green UI screenshot.

Worked connection, variable and backend resolution

Use only dummy, non-secret values in a disposable Airflow 2.10.5 lab. The core LocalFilesystemBackend makes precedence observable without cloud accounts or provider credentials. It is not encrypted secret storage and is not a recommendation to put real secrets in Git, a DAG, a Docker environment dump or this article.

1. In the UI, create connection ID demo_http, type HTTP, host metastore.invalid, and variable demo_window_hours with value 72. Leave login/password empty. These two metastore entries are what Admin → Connections / Variables shows.
2. In the lab's already mounted config/ directory, create demo-connections.json and demo-variables.json with these separate contents:

{"demo_http": {"conn_type": "http", "host": "backend.invalid"}}

{"demo_window_hours": "24"}

3. Inspect the release YAML and merge these four keys into its shared x-airflow-common.environment mapping, preserving the existing environment anchor and service settings. The shared config mount already exposes the files at /opt/airflow/config/. Configure every process that resolves these IDs; setting an environment variable only in the host terminal does not automatically put it in a worker.

AIRFLOW__SECRETS__BACKEND: airflow.secrets.local_filesystem.LocalFilesystemBackend
AIRFLOW__SECRETS__BACKEND_KWARGS: '{"connections_file_path":"/opt/airflow/config/demo-connections.json","variables_file_path":"/opt/airflow/config/demo-variables.json"}'
AIRFLOW_CONN_DEMO_HTTP: '{"conn_type":"http","host":"environment.invalid"}'
AIRFLOW_VAR_DEMO_WINDOW_HOURS: '48'

After validating that lab configuration, recreate the affected containers with docker compose up -d so they receive the new environment. A simple process restart does not necessarily recreate a container's environment. Ensure those mounted files are readable by the configured Airflow UID and absent from any public artifact. The actual task-side lookup is:

# Airflow 2.10.5 reference: invoke inside a task, never at DAG module import.
from airflow.hooks.base import BaseHook
from airflow.models import Variable
 
def check_demo_resolution():
    connection = BaseHook.get_connection("demo_http")
    hours = int(Variable.get("demo_window_hours"))
    # Dummy configuration assertions only; no HTTP request or credential printing.
    assert connection.host == "backend.invalid"
    assert hours == 24
    return {"dummy_resolution_checked": True}

4. Predict before testing: with the backend configured and both files populated, the effective host/hours are backend.invalid and 24, even though the UI shows metastore.invalid and 72. A metastore edit does not override a higher-priority match. Next, remove these two dummy keys from the files (keep valid JSON {}), restart the isolated lookup task in a fresh process, and expect environment.invalid and 48. Then remove the two dummy environment keys, recreate the containers, and expect the metastore host and 72. Adjust the two expected assertions for each phase. Missing keys exercise fallback; a broken file, inaccessible backend or malformed value is a separate error experiment, not proof of successful fallback. If all three stores omit the IDs, lookup without a default fails; do not quietly use a production credential instead.

Why this works: an enabled alternative backend is searched before environment variables, then the metastore. Connection environment names use AIRFLOW_CONN_ and variables use AIRFLOW_VAR_, while configuration uses the double-underscore AIRFLOW__SECTION__KEY convention. File keys use the requested connection/variable identifiers. No HTTP hook or connection test is needed to inspect this dummy resolution; the .invalid hosts deliberately cannot designate a real external endpoint. For a cloud backend, replace the backend class, kwargs, secret naming and workload-identity setup with that pinned provider's documented contract; the local fixture does not establish cloud authentication or provider compatibility.

Acquire, initialize and inspect an Airflow 2.10.5 learning environment

This is a procedure for a separate disposable workstation, not a report of a running service. Docker Engine must already be available, Compose must be at least v2.14.0, and Docker needs at least 4 GB RAM (the guide recommends 8 GB). The release's initialization script recommends at least two CPUs and 10 GB free disk. Do not attempt this on a host with only 1.3 GB free. No images, packages or containers were installed or started for this review. The release pin is for reproducing this lesson, not a claim that an old release is currently supported or secure for production.

1. Create a new directory; if it exists, stop and inspect rather than overwriting an existing environment. Obtain the release-specific file, not /stable/. On Linux:

mkdir airflow-2.10.5-lab && cd airflow-2.10.5-lab
curl -fL --output docker-compose.yaml 'https://airflow.apache.org/docs/apache-airflow/2.10.5/docker-compose.yaml'
mkdir dags logs plugins config
printf 'AIRFLOW_UID=%s\nAIRFLOW_IMAGE_NAME=apache/airflow:2.10.5\n' "$(id -u)" > .env

On non-Linux hosts, the guide uses AIRFLOW_UID=50000. These are lab file-creation instructions, not permission to overwrite an existing .env. Check that no inherited AIRFLOW_IMAGE_NAME, AIRFLOW_PROJ_DIR or _PIP_ADDITIONAL_REQUIREMENTS changes the intended configuration. Leave runtime package installation empty. A tag pins the release name, not an immutable image digest; record the pulled image digest for an actual run.

2. Inspect the downloaded YAML before running it. In this release, replace the webserver's exact port mapping "8080:8080" with "127.0.0.1:8080:8080", and the optional Flower mapping "5555:5555" with "127.0.0.1:5555:5555". Make these replacements, not extra mappings; an additional loopback entry would leave the public listener. Redis uses expose: 6379 inside the Compose network and PostgreSQL has no host port mapping in this file. Verify every host-published port is loopback-bound. Leave Flower disabled. The bundled default lab accounts are public, so private bindings and a disposable environment are essential, not a substitute for a production security design.

3. Save the two-operator example above in dags/two_operator_demo.py. Validate configuration, initialize the metadata database and first lab account, and start the stack only after initialization exits successfully:

docker compose version
docker compose config --quiet
docker compose up airflow-init
# Continue only if airflow-init exits 0 with migrations/account creation successful.
docker compose up -d
docker compose ps
docker compose exec airflow-worker airflow version

Expect version 2.10.5. config --quiet is a syntax/configuration check, not a port audit or proof of security. The release's stack uses CeleryExecutor, Redis and PostgreSQL; successful initialization alone does not prove scheduler/worker/webserver health. If unhealthy, inspect docker compose logs airflow-init airflow-scheduler airflow-worker airflow-webserver, checking memory, permissions and import errors before retrying. Keep logs private; do not publish credentials or whole environment dumps.

4. On that workstation, open http://127.0.0.1:8080 and use the disposable lab account created by initialization (the official quickstart documents its defaults). Find two_operator_demo; if absent, inspect the DAG import errors rather than repeatedly triggering unrelated examples. Unpause this manual-only DAG if necessary, trigger one run, and inspect Graph/Grid: count precedes announce. The Python return is 7; the shell log contains count task succeeded. Inspect task start/end/duration and attempt logs; a green task state is not evidence that an external transaction was atomic. Load train_tiny_model.py separately when studying scheduled data intervals; the manual two-operator demonstration is not evidence of its scheduling behavior.

5. Record expected versus observed states and any failure; a screenshot is optional. Finish with docker compose stop in this same lab directory. That stops containers while retaining data for inspection; deleting volumes is a separate destructive decision. No completion screenshot, login, task run or successful Airflow service is claimed here.

Validation and deployment exercise

For an Airflow 2.10.5 environment with matching constraints/provider packages, review these checks before enabling a schedule. They are instructions for an isolated runtime, not tests claimed to have run during this review:

1. Load the DAG with DagBag from airflow.models.dagbag; assert no import_errors, the expected DAG ID, five task IDs and the ingest-to-manifest edges. This checks discovery and graph construction, not service availability.
2. Unit-test pure training and score functions, including constant x, empty holdout, invalid shape and nonfinite data. For real data, test schema/partition contracts separately.
3. Trigger one controlled run. Inspect graph dependencies, task states, retries, data interval, duration and logs. Verify that the run uses its recorded interval on retry and that no credentials appear. A screenshot is optional evidence, not a learning prerequisite.
4. Inject an upstream exception, a non-improving candidate and a registration timeout after commit. Check failure versus skip, preserved run status, idempotent output and deduplicated notifications. Check overlapping/backfill runs against the same output partition.
5. For a production design, choose executor and workers, secure the web/API surface, isolate task identities, provision metadata persistence/backups and remote logs, set pools and resource bounds, and define DAG/provider deployment and rollback. Verify recovery from worker loss and metadata outage; the Compose quick-start does not establish these properties.

Exercise: a sensor finds ready.json, then training writes three of four artifact files and fails. The retry uses a new random directory. What can the scheduler guarantee, and what must the application change? What happens if the sensor uses a path mounted only on one worker?

Sources and review boundary

The earlier source checks target Airflow 2.10.5, accessed 2026-10-07. Airflow 3 has different architecture/APIs; provider and chart versions are separate compatibility decisions.

• Architecture: scheduler, optional separate DAG processor, executor and metadata roles.
• TaskFlow tutorial: data dependencies, dictionary-output inference, runtime context and mixed operator/sensor patterns.
• XComs: small values and object-storage backend configuration, not a universal payload limit.
• DAG runs: data intervals, catchup and leaf-state failure caveat.
• DAG trigger rules: one_failed, skips and upstream dependencies.
• Secrets backends: lookup precedence and UI visibility.
• Sensors: poke and reschedule worker-slot behavior.
• Docker Compose guide: learning setup and production-security warning.
• Best practices: stable partitions, idempotent writes and avoiding expensive top-level work.

The pure model calculation and failure inputs are locally tested; the full DAG is syntax- and documentation-reviewed only. No scheduler, cloud training service, Slack send or registry promotion was executed. Conceptual deployment coverage is not a deployment success claim.

Additional repair evidence and execution boundary

The feature, monitoring, operator, setup and dummy-resolution additions were checked against saved primary passages on 9 October 2026; this evidence date is not a publication date. Airflow behavior remains bound to 2.10.5. The scikit-learn source is 1.5; the TFDV guide is a current access-date snapshot, not a claim about the Airflow release or a pinned TFDV installation.

• PythonOperator and BashOperator: callable arguments, shell commands and unsafe user interpolation.
• Release Compose file: exact image default, service mounts, ports and resource checks, read alongside the Compose guide above.
• Local filesystem backend, connections and variables: dummy configuration formats and resolution lookup.
• scikit-learn common pitfalls: training-only preprocessing fit and consistent later transformations.
• TFDV getting started: categorical L-infinity statistics, comparator thresholds and drift/skew investigation.

Bounded CPU fixtures check the actual pure feature/drift functions and earlier model math. Python AST, JSON/YAML and inert MDX checks are not Airflow imports or integration tests. The callable/shell example, Compose lifecycle, UI and backend precedence procedure are source/static-reviewed references, not observed service outcomes. No credentials, cloud resources, containers or live notification/registry effects were used.