Top-K and Top-P — Trimming the Choices
Even at a moderate temperature, the reply-drafting prompt now and then produced a bizarre word — "Your brake cables are now fully marzipan." The word was very unlikely, but not impossible, so once in thousands of tokens it got picked. Temperature reshapes the odds; top-k and top-p go further and throw the long tail of unlikely tokens away.
The rule
Top-k keeps a fixed number; top-p keeps enough to reach a share.
Both settings shrink the pool of candidate tokens before one is sampled. The kept tokens' probabilities are then rescaled so they add up to 1 again.
Take this next-token distribution after "Please check the":
brake 0.50 chain 0.25 bell 0.15 tyre 0.06 spoke 0.04- Top-k keeps the k most likely tokens. With
k = 2, onlybrakeandchainsurvive, rescaled to about 0.67 and 0.33. - Top-p (also called nucleus sampling) keeps the smallest set of top tokens whose probabilities add up to at least p. With
p = 0.9: brake 0.50, plus chain = 0.75, plus bell = 0.90. Three tokens survive;tyreandspokeare dropped.
The difference shows up when the model's confidence changes. Suppose after "Please ring the" the distribution is bell 0.92, with everything else tiny. Top-p 0.9 keeps only bell, because it reaches 0.9 alone. Top-k 2 still keeps two tokens, dragging in an unlikely second choice. When the model is unsure, top-p keeps more tokens; when it is sure, fewer. It adapts; top-k does not.
The shortcut
Picture the tokens as a ranked list and draw a cut line:
- Top-k — cut after position k, always.
- Top-p — walk down, adding probabilities, and cut as soon as the running total reaches p.
Rank
brake 0.50, chain 0.25, bell 0.15, tyre 0.06, spoke 0.04.
A practical habit: adjust either temperature or top-p, not both at once, so you can tell which change had which effect. Many tools default top-p to 1, which keeps every token and leaves the job to temperature.
Drills
Answer fast — trust the rule.
With top-k = 3, how many candidate tokens are kept?
Using the list brake 0.50, chain 0.25, bell 0.15, tyre 0.06, spoke 0.04, which tokens does top-p = 0.7 keep?
The model gives one token a probability of 0.95. What does top-p = 0.9 keep?
Why is top-p often preferred over top-k?
Remember
- Top-k keeps a fixed number; top-p keeps enough to reach a share.
- Shortcut: rank, add up, cut at the line, then sample from what is left.
- Trap: tuning temperature and top-p together and not knowing which one mattered.