Sampling Settings — Temperature, Top-p, Penalties and Length

Tripmate writes the arguments for book_ticket a little differently each time, and once it picks the wrong train. Meanwhile, its trip descriptions for Asha read like the same sentence over and over. Both problems come from the same place: the settings that control how the model picks each next token.

The rule

Tools want low randomness; writing can afford a little more.

At every step the model scores every possible next token, and those scores become probabilities. Sampling settings shape how the next token is chosen from them:

• Temperature — sharpens or flattens the probabilities. Low (near 0) almost always takes the most likely token: steady and repeatable. Higher values give unlikely tokens more chance: more varied, and more error-prone.
• Top-p — keeps only the smallest group of top tokens whose probabilities add up to p, then samples among them. With top_p = 0.8, rare long-shot tokens are never picked.
• Frequency penalty — lowers a token's chance more each time it has already appeared. Cuts repeated words and phrases.
• Presence penalty — lowers a token's chance by a flat amount once it has appeared at all. Nudges the model toward new words and topics.
• Max length (max output tokens) — a hard cap on how much the model may write. When it is hit, the answer is simply cut off.

Providers name and support these a little differently, and a common piece of advice is to tune temperature or top-p, not both at once.

The shortcut

Ask: "Would I want the same output if I asked twice?" Yes → low temperature, no penalties (tool calls, data extraction, classification). No → moderate temperature, maybe a small penalty (brainstorming, descriptions).

The trap with max length is treating it as "be brief". It does not make the model plan a shorter answer; it chops the answer mid-sentence. To get short answers, ask for them in the prompt and keep the cap as a safety limit.

Drills

Answer fast — trust the rule.

Remember

• Tools and extraction want low randomness; open-ended writing can take a little more.
• Shortcut: "Would I want the same output if I asked twice?"
• Trap: using max length to mean "be brief" — it truncates, it does not summarise.