Calibration and Debiasing
After a busy refund season, someone added four refund emails to the triage prompt's examples. Soon a third of all emails were being labelled refund, including plain repair bookings. And when asked how sure it was, the model said "95 percent" about answers that were right only about three times in four. The prompt had developed a tilt, and its confidence could not be taken at face value.
The rule
Balance the examples, then check that confidence matches accuracy.
Prompt bias here means a systematic lean in answers caused by the prompt itself, not the input. Few-shot prompts are especially prone to it:
- Majority-label bias — if most examples share a label, the model over-predicts that label.
- Recency bias — labels near the end of the prompt pull harder than those at the start.
- Common-token bias — answers that are frequent words in general can be favoured over rarer ones.
Debiasing means removing these tilts: balance the labels across examples, shuffle their order, and use neutral wording that does not hint at an answer.
Calibration is about confidence. A well-calibrated system is right about 80 percent of the time when it says it is 80 percent sure. Models' stated confidence ("I'm 95 percent sure") is often poorly calibrated, so never use it as a probability until you have checked it against real results.
Stated confidence Emails Actually correct
90-100% 120 89 (74%) <- overconfident
70-89% 60 45 (75%)The shortcut
Run the tilt test: send the prompt a few content-free inputs — an empty email, or "N/A" — and see what it answers. A fair prompt spreads these across labels or declines. A tilted prompt keeps picking the same label.
Tilt test
Five content-free inputs all come back
refund. The prompt, not the email, is choosing.
Bias can also come from wording that carries assumptions about customers — for example, examples that always make one group of names the angry ones. Reviewing examples for that kind of pattern is part of the same job.
Drills
Answer fast — trust the rule.
Four of six few-shot examples are labelled refund. Which bias is most likely?
What does it mean for a model to be well calibrated?
The model says "95% sure" on answers that are correct 74% of the time. What is it?
Content-free inputs all come back labelled refund. What does that show?
Remember
- Balance the examples, then check that confidence matches accuracy.
- Shortcut: the tilt test — content-free inputs expose a prompt's lean.
- Trap: trusting a model's stated confidence as if it were a measured probability.