Experiment Tracking
"Last Tuesday I got recall 0.78, but I can't remember which settings I used." Every data scientist has said a version of this. Training is a search: dozens of runs with different features, depths and learning rates. Without a record, the best result of the week can be lost by Friday.
The rule
Log what you set, what you got, what you made, and where it came from.
An experiment tracker stores one record per training run. MLflow is the most common open-source one:
import mlflow
mlflow.set_experiment("churn")
with mlflow.start_run(run_name="depth-6-with-playlists"):
mlflow.log_params({"max_depth": 6, "learning_rate": 0.1}) # what you set
mlflow.set_tag("data_version", "v2026-06") # where it came from
model = train(features, max_depth=6, learning_rate=0.1)
mlflow.log_metric("recall", 0.74) # what you got
mlflow.log_metric("precision", 0.66)
mlflow.sklearn.log_model(model, "model") # what you made
mlflow.log_artifact("confusion.png")The four parts of the rule map to four kinds of record:
- Parameters — settings you chose:
max_depth,learning_rate, the cutoff date. - Metrics — numbers you measured: recall, precision, training time. Metrics can also be logged per step, to draw learning curves.
- Artifacts — files the run produced: the model itself, plots, a sample of predictions.
- Source — the code and data versions. When the script runs from a Git repository, MLflow records the commit automatically; the DVC data version you add as a tag.
mlflow ui opens a web page where runs sit in a table you can sort, filter and compare side by side. For common libraries, mlflow.sklearn.autolog() logs parameters, metrics and the model with one line.
The shortcut
Before running any experiment, ask: "If this turns out to be the best run, could someone else reproduce it from the record alone?" If any of set, got, made or source is missing, log it first.
Find
Open the churn experiment, sort by recall, and the 0.78 run is at the top.
The common gap is the source: parameters and metrics are logged, but not the data version, so the run cannot be repeated on the same data.
Drills
Answer fast — trust the rule.
In MLflow terms, what is learning_rate = 0.05?
Which is an artifact?
A run logged parameters and metrics but not the data version. What can you not do?
What does mlflow.sklearn.autolog() do?
Remember
- Log what you set, what you got, what you made, and where it came from.
- Test each run with: could someone reproduce it from the record alone?
- Trap: forgetting the data version, which makes a great run unrepeatable.