Orchestration and Scheduling
PedalCity's first pipelines ran on cron: extract rides at 02:00, build fact_rides at 02:30, refresh the dashboard at 03:00. One night the extract took 45 minutes. The build started on half-copied data, the dashboard published the wrong numbers, and nobody was alerted. The jobs ran on time. They just did not run in the right order relative to each other. An orchestrator fixes that.
The rule
Schedule the graph, not the scripts: each task runs when its inputs are ready.
An orchestrator runs a pipeline as a DAG, a directed acyclic graph. Tasks are the boxes and dependencies are the arrows. There are no loops, so there is always a finishing point.
extract_rides ────┐
extract_payments ─┼──► build_fact_rides ──┬──► build_finance_mart
extract_weather ──┘ └──► refresh_ops_dashboardThe schedule says when the graph starts, for example daily at 02:00. The arrows decide everything else:
- Dependencies:
build_fact_rideswaits until all three extracts succeed, however long they take. - Parallelism: the three extracts do not depend on each other, so they run at the same time.
- Retries: a failed task retries a set number of times, with a pause between attempts, before it is marked failed.
- Stop on failure: if a task finally fails, everything downstream waits, and someone is alerted. No wrong numbers get published.
- Backfills: each run carries a logical date, so you can rerun the graph for 10 September, or for all of last month. This is safe only because the loads are idempotent.
Common orchestrators include Apache Airflow, Prefect, Dagster and Luigi. Their syntax differs, but all are built on these ideas.
The shortcut
Draw the arrows before writing any schedule. For every task ask "What must be finished before this can start?" The answers are the arrows. Tasks with no arrow between them can run in parallel.
List the tasks
Three extracts, one build of
fact_rides, the finance mart and the dashboard refresh.
Drills
Answer fast — trust the rule.
In PedalCity's DAG, which tasks can run at the same time?
extract_weather fails after all its retries. What should happen to build_finance_mart?
What was the real flaw in running the jobs on cron at fixed times?
You backfill the whole DAG for every day of last month. What makes this safe?
What does the "acyclic" in DAG guarantee?
Remember
- Schedule the graph, not the scripts: tasks run when their inputs are ready.
- Draw the arrows first by asking what must finish before each task.
- Retries, alerts and backfills all rely on dependencies, logical dates and idempotent tasks.