Max Tokens and Stop Sequences
Two odd bugs appeared in the inbox system. Some drafted replies ended mid-sentence: "Thanks for your patience — your wheel will be ready on". And the triage call sometimes answered repair, then carried on writing a brand-new, invented customer email underneath. Both are problems of where output ends, and two settings control that.
The rule
Max tokens is a hard cut; a stop sequence is a clean ending.
Max tokens (the name varies between tools, such as max output tokens) caps how many tokens the model may generate in one reply. When the cap is reached, generation simply stops, wherever it is — mid-word, mid-sentence, mid-JSON. Crucially, the model does not know about the cap in the way a writer knows a word limit. It does not plan a shorter answer; it just gets cut off.
A stop sequence is a string you choose. As soon as the model produces it, generation ends, and the stop string itself is usually left out of the returned text. It lets you end output at a natural boundary.
Prompt (few-shot triage):
Email: My gears slip on hills.
Label: repair
Email: Where is order 4471?
Label: order
Email: Can I return these gloves?
Label:
Stop sequence: "\n" -> output: " refund"Without the stop sequence, the model sees a repeating Email/Label pattern and may happily continue it, inventing the next email itself.
The shortcut
Split the job in two:
- Want a shorter answer? Ask for it in the prompt ("at most three sentences"), then set max tokens comfortably above that as a safety net.
- Want output to end at a boundary? Set a stop sequence at that boundary: a newline, a closing tag like
</answer>, or the start of the next pattern likeEmail:.
Diagnose the cut-off reply
The reply stopped at exactly 60 tokens, the configured cap. The model was cut, not finished.
A useful check: most APIs report why generation ended — for example because the length limit was hit or because a stop sequence or natural end was reached. If you see the length reason on a reply you expected to be complete, your cap cut it short.
Drills
Answer fast — trust the rule.
What happens when a reply reaches the max tokens limit?
You want replies of about three sentences. What is the best approach?
The triage model writes its label and then invents another email. Which setting fixes this most directly?
The model is asked to put its answer inside <answer> tags. Which stop sequence ends output right after the answer?
Remember
- Max tokens is a hard cut; a stop sequence is a clean ending.
- Shortcut: ask for length in the prompt, cap as a safety net, stop at a boundary.
- Trap: using max tokens to make answers shorter and getting sentences cut in half.