Automatic Prompt Engineering
The owner had rewritten the triage instruction eleven times, each version "feeling" better than the last. The test set said otherwise: accuracy wobbled between 88 and 92 percent with no clear winner. There is another way to search for a good prompt: let a model write the candidates, and let the test set choose.
The rule
Let a model propose prompts; let labelled data pick the winner.
Automatic prompt engineering (APE) turns prompt writing into a search:
- Generate — show a model some input/output pairs from your task and ask it to write many candidate instructions that would produce those outputs.
- Score — run every candidate on a labelled evaluation set and measure accuracy, or whatever metric matters.
- Select and refine — keep the best few, ask the model for variations of them, and score again.
- Confirm — check the final winner on a separate held-out set it was never tuned on.
Here are pairs of customer emails and the correct category:
<examples> ...20 labelled emails... </examples>
Write 10 different instructions that would lead an assistant to produce
exactly these categories. Vary the wording, the definitions and the
level of detail.The value is not that the machine writes more cleverly than you. It is that the choice is made by measurement instead of taste, and far more candidates can be tried than a person would bother to write. Sometimes the winner uses a phrasing no human would have guessed.
The shortcut
Remember the three sets: examples to inspire, eval to choose, held-out to confirm. Never let one set do two jobs.
Generate
Give the model 20 labelled emails and ask for 10 candidate instructions.
The trap is overfitting: a prompt tuned hard on one small set can learn that set's quirks rather than the real task. A small drop on the held-out set is normal; a big drop means the "winner" was lucky. APE also needs a genuine labelled dataset, which is often the most valuable thing you build in the whole project.
Drills
Answer fast — trust the rule.
In automatic prompt engineering, what decides which prompt wins?
What is the model asked to do in the generate step?
Why confirm the winner on a held-out set?
The winner scores 96 percent on the eval set but 78 percent on held-out emails. What does this suggest?
Remember
- Let a model propose prompts; let labelled data pick the winner.
- Shortcut: examples to inspire, eval to choose, held-out to confirm.
- Trap: overfitting — a big drop on held-out data means the win was luck.