Train, Validation and Test Sets

Pantry wants a model that predicts which Pantry Plus members will cancel next month, so the voucher from the expected value lesson goes to the right people. A teammate's first model is thrilling: on the data it learned from, it spotted 97 of every 100 members who cancelled. Run on the following month, it spotted 52. The model had not learned who cancels. It had memorised who had cancelled.

The rule

Judge a model only on data it never learned from.

A flexible model can fit the rows it is shown almost perfectly, noise included. That is overfitting, and a score on seen data measures memory, not skill. So the data is divided three ways:

• Training set: the model learns its patterns here.
• Validation set: you compare models, tune their settings and choose the decision threshold here. Each choice you make by looking at it leaks a little of it into the model, so its score slowly becomes optimistic.
• Test set: locked away until the end and used once, to estimate how the finished model will do in the real world.

How you split matters as much as whether you split. The rule is to split the data the way the model will be used:

• If rows are independent and the future looks like the past, a random split is fine.
• If the model predicts the future, train on the past and test on later periods, or the model learns from information it will never have.
• If one person or store has many rows, keep all of their rows on the same side of the split.

The shortcut

Ask "when will this model run, and on whom?" Then build the split to recreate that moment.

If you check the test score after every tweak, it quietly becomes a second validation set, and you lose your only honest estimate.

Drills

Answer fast — trust the rule.

Remember

• Judge a model only on data it never learned from.
• Split the way the model will be used: by time for forecasts, by person when people repeat.
• Trap: peeking at the test set until it becomes a second validation set.