Evaluating Prompts and Context
A teammate rewrites the Pedal Point system prompt, tries five questions, and says "it's much better". Last month a similar change fixed returns answers and quietly broke helmet answers. Nobody noticed for two weeks. With a model that varies from run to run, impressions are not evidence. An evaluation is.
The rule
Change one thing, run the same test set, compare the scores.
An evaluation, or eval, has three parts.
1. A fixed set of test cases. Realistic inputs, ideally from real (anonymised) questions: common ones, hard ones, and ones the app should refuse or answer "I don't know". Each case records what a good answer must include and must avoid.
2. Scoring. Use the cheapest reliable method for each quality:
- Deterministic checks: code tests what it can. Is the JSON valid? Does the answer contain the correct order status? Does it avoid promising a refund?
- Model-based grading: another model scores qualities code cannot check, like helpfulness or tone, against a written rubric. Check its grades against your own on a sample first; graders have biases too.
- Human review: people read a sample, especially for new features and high-stakes answers.
3. Comparison. Run the old and new versions on the same set and compare. Look beyond the average: which cases got worse?
The eval also measures context, not just wording. Test retrieval on its own: for each question, did the right chunk come back? A great prompt cannot rescue missing context, and the eval should tell you which one failed.
Keep the set, and grow it with every bug report. Re-run it after every change to the prompt, the context, the model or the settings. That habit is regression testing: making sure fixes do not break what already worked.
The shortcut
One change, same set, compare, inspect the losers.
Fix the set
Fifty anonymised questions, each with required facts and forbidden content.
The trap is changing the prompt, the model and the retrieval all at once. If scores move, you will not know why.
Drills
Answer fast — trust the rule.
Which check is best done deterministically, in code?
Before trusting a model-based grader, what should you do?
The new prompt raises the pass count from 41 to 44 of 50. What should you still check?
Answers are wrong, and the eval shows the right chunk was retrieved for only 30 of 50 questions. What should you work on first?
A customer reports a new bug and you fix it. What should happen to the eval set?
Remember
- Change one thing, run the same test set, compare the scores.
- Code checks what it can; graders and humans check the rest.
- Keep the set, grow it with bugs, re-run it after every change.