DVC Pipelines and Data Lineage
Training the churn model takes two steps: turn the raw CSV into features, then fit the model. Run them by hand and you will one day forget to rerun the first after changing it — and train on stale features. A DVC pipeline writes the steps down with their inputs and outputs, so the tool decides what needs to run.
The rule
Declare the inputs; rerun only what they touched.
A pipeline lives in dvc.yaml. Each stage has a command, its deps (inputs), its params (settings read from params.yaml) and its outs (outputs):
stages:
prepare:
cmd: python prepare.py
deps:
- data/subscriptions.csv
- prepare.py
outs:
- data/features.parquet
train:
cmd: python train.py
deps:
- data/features.parquet
- train.py
params:
- max_depth
outs:
- models/model.pkl
metrics:
- metrics.json:
cache: falseBecause train depends on prepare's output, DVC knows the order. Running dvc repro does three things:
- Hashes every stage's deps and reads its params.
- Compares them with what was saved in
dvc.lockfrom the last run. - Reruns only stages whose inputs changed. A rerun that produces a new output changes the inputs of the stages downstream, so they rerun too. Then it updates
dvc.lock.
dvc.lock is committed to Git. That makes it a lineage record: for any commit, it lists the exact hashes of the data, code and model involved, plus the parameter values. Asked "what made this model?", you read the lock file of the commit that produced it.
The shortcut
To predict what dvc repro will do, find the changed input, then walk downstream. Everything upstream of the change is skipped.
Find the change
Only
params.yamlchanged, and onlymax_depth.
A trap: a script that reads a file you did not list in deps. DVC cannot see that file change, so it will wrongly skip the stage. Declare every input.
Drills
Answer fast — trust the rule.
You change how prepare.py computes a feature, so features.parquet comes out different. Which stages does dvc repro run?
What does dvc.lock record?
train.py also reads config/thresholds.json, which is not listed in deps. You change that file. What does dvc repro do?
Six months later, how do you find which data produced the model in commit a1b2c3?
Remember
- Declare the inputs; rerun only what they touched.
- Predict
dvc reproby finding the changed input and walking downstream. - Trap: undeclared inputs are invisible to DVC, so stages get skipped wrongly.