Pebellslearn by building
0%
RoadmapsMatrixCoursesSandboxSign in
Evaluating Models  ›  Capstone Project

Data Leakage

Machine Learning 15 minLeakage happens when training data contains information that will not exist when the model is used, such as features created by the outcome or computed from the future, and it makes offline scores look far better than live results.

Data Leakage

A colleague's new churn model scores a precision of 0.97 and a recall of 0.98 on August, far beyond anything before it. Its most important feature is exit_survey_sent. Pantry sends that survey the moment a member presses "cancel". The model has not learned to predict cancelling; it has learned to notice that a cancellation already started. Live on the 1st of the month, when nobody has pressed anything yet, it will be close to useless.

The rule

Every feature must be knowable at the moment you predict.

Leakage is information in the training data that will not be available when the model is actually used. It makes offline scores look wonderful and live results disappoint. It comes in three common forms:

  • Target leakage: a feature created by the outcome itself. exit_survey_sent, cancellation_reason and membership_fee_refunded all exist only because the member cancelled.
  • Time leakage: a feature computed with data from after the prediction moment. An avg_orders_per_month worked out over the whole year includes the very months you are trying to predict.
  • Train–test contamination: information from validation or test rows seeping into training. Filling blanks with the mean of all rows, scaling with statistics from the full dataset, or letting the same member appear on both sides of a split all do it.

The warning signs are consistent: a score that seems too good to be true, one feature that dominates everything, and a model that shines offline but falls flat in production.

The shortcut

Run the timestamp test on every feature: "At midnight on the 1st, when the model runs, would this value already exist, exactly as stored?" If the answer is no, drop the feature or rebuild it from earlier data.

Auditing four churn featuresStep 1 of 5

  1. orders_last_30_days

    Counted from orders before the 1st. It exists at prediction time: keep it.

The time-based split from the first lesson in this section already keeps each member's future rows out of training. The timestamp test catches what a split cannot: a leak baked into a single column.

Drills

Answer fast — trust the rule.

quick drill

Which feature leaks the outcome in Pantry's churn model?

quick drill

A model is excellent on validation data but poor once it runs live. What should you suspect first?

quick drill

Before splitting, you fill missing values with the mean of the whole dataset. What is the problem?

quick drill

avg_orders_per_month is calculated over the full year, then used to predict June cancellations. What kind of leak is it?

Remember

  • Every feature must be knowable at the moment you predict.
  • Timestamp test: at prediction time, would this value already exist, exactly as stored?
  • Trap: scores that look too good to be true. Hunt for the leak before you celebrate.

Practice — level up

0/1 solved

Write your answer, then compare it with a rubric and a model answer.

ExplainAudit the too-good model

Your colleague's churn model has precision 0.97 and recall 0.98 on August. Its features include orders_last_30_days, exit_survey_sent, avg_orders_per_month (computed over the full year), support_tickets_this_month and months_since_joined. Explain which features you would keep, drop or rebuild and why, and what you would check before trusting the model's score again.