Temperature, Top-k and Top-p

Ask the Pedal Point assistant the same question twice and you may get two differently worded answers. Ask it to pull an order number from an email and you want the same answer every time. The difference comes from a few settings that control how the model picks each token.

The rule

Top-k and top-p pick the shortlist; temperature picks how boldly.

At every step the model produces probabilities for the next token. Say the text so far is "Your order has", and the top candidates are:

Sampling means choosing from these. Three dials shape the choice:

• Top-k keeps only the k most likely tokens. With k = 2 the shortlist is shipped and arrived.
• Top-p (also called nucleus sampling) keeps the smallest group of top tokens whose probabilities add up to at least p. With p = 0.9: shipped 0.60, plus arrived makes 0.85, not enough, plus been makes 0.95. Three tokens.
• Temperature reshapes the probabilities before picking. Low temperature sharpens them, so the favourite wins almost every time; at or near zero the model behaves close to always taking the top token. High temperature flattens them, so unlikely tokens get picked more often.

Providers expose different subsets of these dials, and names and ranges vary, so check the documentation of the one you use. Also, a very low temperature makes outputs much more consistent but is not always a guarantee of identical results on every system.

The shortcut

Facts cold, ideas warm. Extraction, classification and support answers want low temperature. Brainstorming and creative writing can run warmer. For top-p, walk down the list adding probabilities until you reach p.

The trap is changing several dials at once. Adjust one, test, then move on; many teams leave top-p alone and tune only temperature.

Drills

Answer fast — trust the rule.

Remember

• Top-k and top-p pick the shortlist; temperature picks how boldly.
• Facts cold, ideas warm.
• Top-p: add probabilities from the top until you reach p.