Choosing a Threshold
The churn model does not say "cancel" or "stay". It gives each member a score between 0 and 1, and someone has to decide where to draw the line. So far the line has been 0.5, because that is what the software did by default. But back in the expected value lesson you found that a voucher pays for itself whenever a member's chance of cancelling is above 0.4. The threshold is a business decision, and you already know how to make it.
The rule
Set the cut-off where acting stops paying, not at 0.5 by habit.
Sending a voucher to a member with cancel risk gains rupees on average and costs 100 rupees, so it pays when , that is, when . If the model's scores behave like real probabilities, the threshold is simply 0.4.
That "if" is called calibration: among members scored around 0.4, about 40% should really cancel. Check it on the validation data by grouping members into score bands and comparing each band's average score with its actual cancel rate. Many models rank members well but are over- or under-confident, and then you choose the threshold directly from value instead.
Here is the value at several thresholds on August, using ₹250 per canceller caught and ₹100 per voucher:
| Threshold | Flagged | Cancellers caught | Net value |
|---|---|---|---|
| 0.6 | 400 | 320 | ₹40,000 |
| 0.5 | 800 | 560 | ₹60,000 |
| 0.4 | 1,200 | 740 | ₹65,000 |
| 0.3 | 1,600 | 880 | ₹60,000 |
Value peaks at 0.4, just where the break-even rule predicted, which is what well-calibrated scores should give.
The shortcut
Judge each band of scores on its own. Moving the threshold down adds one band of members at a time; include the band only if its cancel rate beats the break-even rate.
The 0.4 to 0.5 band
400 members, of whom 180 cancel: a cancel rate of 45%, above the 40% break-even.
The best threshold moves whenever the costs do. If the voucher cost fell to ₹50, break-even would drop to 0.2 and more members would qualify. If the budget covered only 1,000 vouchers, you would send them to the 1,000 highest scores instead.
Drills
Answer fast — trust the rule.
Why not keep the threshold at 0.5?
What does it mean for the churn model to be well calibrated?
The voucher cost falls from ₹100 to ₹50. What happens to the best threshold?
Remember
- Set the cut-off where acting stops paying, not at 0.5 by habit.
- Judge each band of scores on its own: include it only if its cancel rate beats break-even.
- Trap: forgetting to re-check the threshold when costs, budgets or the model change.