Beat a Baseline First
The retrained churn model reports 93% accuracy on August, the validation month. The room is impressed until someone asks what share of members did not cancel in August. The answer is 90%. A "model" that says "nobody cancels" to everyone scores 90% without learning anything, so the real model's 93% is only about three points above doing nothing.
The rule
A score means nothing until a simple baseline sets the bar.
A baseline is the score of the simplest reasonable approach. It turns "93%" into "93% where doing nothing gets 90%", which is a very different claim. Every problem has a ladder of baselines:
- Most common answer: predict "stays" for everyone, or predict the average for a number.
- A simple rule a manager could write on a whiteboard: "members with no order in the last 30 days will cancel".
- Last value: for forecasts, "next week's orders equal this week's".
- A simple model, such as logistic regression, before anything elaborate.
Your model earns its complexity only by beating every rung, and by enough to pay for building and maintaining it.
Baselines also expose a weak metric. In August, 1,000 of Pantry's 10,000 members cancelled. Here is how the options did:
| Approach | Members flagged | Cancellers among them | Accuracy |
|---|---|---|---|
| "Nobody cancels" | 0 | 0 | 90% |
| No order in 30 days | 1,500 | 600 | 87% |
| Churn model | 800 | 560 | 93.2% |
The rule catches 600 cancellers, far more useful than catching none, yet its accuracy is lower than doing nothing. Accuracy rewards the easy "stays" majority, so on an imbalanced problem it cannot tell you which approach helps. The next lesson replaces it with better measures.
The shortcut
Climb most common → rule → simple model → yours, and score every rung in the units of the decision. From the expected value lesson, each canceller caught is worth 0.25 × ₹1,000 = ₹250 and each voucher costs ₹100.
Nobody cancels
No vouchers sent, no one saved: ₹0.
Drills
Answer fast — trust the rule.
A churn model reaches 91% accuracy when 90% of members stay. What is the honest reading?
What is a sensible first baseline for forecasting each store's orders next week?
The model beats the rule by ₹60,000 a month but needs an engineer's time to keep running. What is the right question?
Remember
- A score means nothing until a simple baseline sets the bar.
- Climb the ladder: most common, simple rule, simple model, then yours.
- Trap: accuracy on imbalanced data, where doing nothing already scores high.