Precision and Recall
Accuracy could not tell a useful rule from doing nothing. The marketing team needs answers to two different questions instead. "Of the members we send vouchers to, how many were really about to leave?" tells them how much money is wasted. "Of the members who really left, how many did we reach?" tells them how many chances were missed. Those two questions have names.
The rule
Precision: how many flags were right. Recall: how many real cases you caught.
Every prediction lands in one of four boxes. Here is the churn model on August, flagging members whose score is 0.5 or more:
| Flagged | Not flagged | |
|---|---|---|
| Cancelled | 560 caught (true positives) | 440 missed (false negatives) |
| Stayed | 240 false alarms (false positives) | 8,760 correctly left alone (true negatives) |
This table is the confusion matrix. From it:
So 70% of vouchers go to members who really were leaving, and the model reaches 56% of all leavers. For the 30-day rule, precision is 600 ÷ 1,500 = 0.40 and recall is 600 ÷ 1,000 = 0.60: it reaches slightly more leavers but wastes far more vouchers.
The two pull against each other. Flag more members and you catch more leavers, so recall rises, but you add more false alarms, so precision usually falls. Flag every member and recall is a perfect 1.0 while precision drops to the base rate, 0.10.
Which matters more depends on what each mistake costs. A false alarm here wastes a ₹100 voucher; a miss loses a ₹250 chance. For a fraud model that blocks orders, a false alarm stops an honest customer from buying, so precision matters most. When you need one summary number and have no costs to hand, the F1 score, the harmonic mean of precision and recall, is a common compromise, but real costs beat it every time.
The shortcut
Precision divides by what you flagged; recall divides by what really happened. Find the right row or column total and the rest is one division.
Find the true positives
Members who were flagged and did cancel: 560.
Drills
Answer fast — trust the rule.
A model flags all 10,000 members, of whom 1,000 cancel. What are its recall and precision?
Pantry's fraud model blocks an order automatically when it fires. Each false alarm blocks an honest customer. Which metric should you guard most closely?
You lower the churn threshold from 0.5 to 0.4. What usually happens?
Remember
- Precision: how many flags were right. Recall: how many real cases you caught.
- Precision divides by the flagged total; recall divides by the real total.
- Trap: quoting one without the other. Flagging everyone gives perfect recall.