Evaluating the AI Feature
The code that Asha's builder generates is only half of what can go wrong. The other half is what Pantry Pal's own model generates for users. She tweaked the prompt to make dinners "more exciting", tried three pantries, and liked what she saw. A week later, users with small pantries were getting dishes that needed ten extra ingredients. Three examples checked by eye had told her nothing about the cases she never tried.
The rule
Keep a fixed set of test cases, and rerun all of them after every change.
An evaluation set, or eval set, is a saved list of realistic inputs, each paired with what a good answer must do. The model's wording changes from run to run, so you do not check for exact words. You check properties: things every good answer has, however it is phrased.
| Property | How it is checked |
|---|---|
| No allergens | Code compares each ingredient with the allergy list |
| Mostly uses what is listed | Code counts ingredients not in the pantry (at most two, say) |
| Steps a beginner can follow | A person, or a second model given a written rubric, grades it |
| Fast enough | The time for each case is measured |
Include the easy cases and the awkward ones: a nearly empty pantry, unusual items, misspellings, several allergies at once, a vegetarian whose pantry holds little more than rice and spices. A few dozen well-chosen cases teach you more than hundreds of easy ones.
Record the cost and time of each run as well. A change that improves quality but doubles the bill is a trade-off you want to see before your users do.
The shortcut
Change one thing, rerun everything, compare the scores.
Baseline
Run the current prompt on all 30 cases: 29 pass "mostly uses what is listed", and all 30 pass "no allergens".
Every failure a real user reports becomes a new case. Over time the eval set turns into a record of every way the feature has gone wrong, checked automatically on every change.
Drills
Answer fast — trust the rule.
Why not test an AI feature by checking for the exact expected words?
Which is a property check for Pantry Pal's suggestions?
Asha changes the prompt and the model on the same day, and the scores drop. What went wrong with her method?
A user reports a bad suggestion that no test case covers. What should happen?
Remember
- Keep a fixed set of test cases, and rerun all of them after every change.
- Check properties, not exact words, and track cost and time per run too.
- Trap: judging a change on a few examples by eye, or changing two things at once.