LLM Self-Evaluation
Classification is easy to score: the label is right or wrong. But how do you score two hundred drafted replies a day? Reading them all is not practical. A common answer is to have a model grade them — as long as you give it a precise rubric and keep checking the grader itself.
The rule
A model can grade well — against a rubric, with the facts, and with checks.
LLM self-evaluation means using a model to review output: either its own draft or another model's. It takes several forms:
- Self-critique — the same model reviews its draft and revises it ("Check your answer against the policy, then fix any problems").
- Model as judge — a separate call grades the output against criteria and returns a verdict.
- Pairwise comparison — the judge picks the better of two outputs, useful when comparing prompt versions.
What makes it work is the rubric. "Is this reply good?" gets vague, generous answers. Specific, checkable criteria get useful ones.
Grade the draft reply using the policy and the customer email below.
Criteria:
1. Every factual claim appears in the policy.
2. It answers the question the customer actually asked.
3. It is under 120 words.
4. It does not promise a refund.
For each criterion write PASS or FAIL with a one-line reason.
Then write "Verdict: send" if all pass, otherwise "Verdict: revise".
<policy> ... </policy> <email> ... </email> <draft> ... </draft>Giving the judge the source facts is essential. Without the policy, it can only judge whether the reply sounds right — the same trap that causes hallucination.
The shortcut
Before trusting a judge, check its CARD: Criteria that are specific, Access to the source facts, a Reason before each verdict, and Double-checks by humans on a sample.
Criteria
Four pass/fail checks, each one a yes-or-no question someone could verify.
Known weaknesses: judges tend to be lenient, can prefer longer answers, may favour text written in their own style, and share blind spots with the model being graded — a fact both got wrong looks fine to both. Human spot-checks are what keep the grader honest.
Drills
Answer fast — trust the rule.
Which judging prompt will give the most useful grades?
Why must the judge see the shop's policy?
Both the drafting model and the judge share the same mistaken belief. What is likely?
What is pairwise comparison?
Remember
- A model can grade well — against a rubric, with the facts, and with checks.
- Shortcut: CARD — criteria, access to facts, reasons, human double-checks.
- Trap: vague rubrics and unchecked judges that are lenient and share blind spots.