Orchestrating Pipelines with Airflow
Retraining the churn model each week means five steps: extract a data snapshot, validate it, train, evaluate against the champion, register the winner. A cron job running one big script can do that — until step two fails at 3 a.m. and step three happily trains on bad data. An orchestrator runs the steps as separate tasks that know about each other.
The rule
Tasks, arrows, a schedule: nothing runs before its inputs succeed.
An orchestrator runs a DAG — a directed acyclic graph. Each task is one step. Each arrow says "this task needs that one first". Acyclic means the arrows never loop back, so there is always a valid order.
Apache Airflow is the most widely used orchestrator. A DAG is a Python file:
from datetime import datetime
from airflow.sdk import dag, task
@dag(schedule="@weekly", start_date=datetime(2026, 1, 1), catchup=False)
def churn_retrain():
@task(retries=2)
def extract() -> str: ... # returns the snapshot path
@task
def validate(path: str) -> str: ...
@task
def train(path: str) -> str: ... # returns the MLflow run id
@task
def evaluate_and_register(run_id: str) -> None: ...
evaluate_and_register(train(validate(extract())))
churn_retrain()Passing one task's result into the next draws the arrows. Airflow then gives each task:
- Its own status and logs, visible in a web UI.
- Retries —
extractretries twice, useful when a database is briefly unavailable. - Dependency safety — if
validatefails,trainand later tasks are marked upstream failed and never run. - Rerun from the failure — after a fix, clear the failed task and the rest continue, without repeating the successful ones.
Kubeflow Pipelines solves the same problem inside Kubernetes: each step runs as its own container, which suits teams whose training already runs on a cluster. Airflow is more general and often schedules non-ML data work too.
The shortcut
To design a DAG, draw an arrow only where data flows. If task B reads something A produced, draw A → B. If not, leave them unconnected so they can run in parallel.
New tasks
listening_featuresandbilling_featuresboth read the validated snapshot.
Drills
Answer fast — trust the rule.
In the churn DAG, validate fails. What happens to train?
What does "acyclic" in DAG guarantee?
Two feature tasks read the same snapshot but not each other's output. How should they be connected?
After fixing a bug in train, what is the main advantage of rerunning in Airflow over rerunning a cron script?
Remember
- Tasks, arrows, a schedule: nothing runs before its inputs succeed.
- Draw an arrow only where data flows; unconnected tasks run in parallel.
- Trap: one giant script, where one failure means rerunning everything or training on bad data.