Temperature — The Randomness Dial
The triage prompt labelled the same email refund on Monday and order on Tuesday, with no change to the prompt. Meanwhile, the owner's request for spring-sale names kept returning near-identical lists. Both problems had the same cause: the temperature setting was wrong for each job.
The rule
Low temperature for right answers, higher temperature for fresh ideas.
Remember that at each step the model has a probability for every possible next token. Temperature is a number, usually between 0 and 2, that reshapes those probabilities before one token is picked.
- Low (near 0) — sharpens the distribution. The favourite token wins almost every time. Output is focused and close to repeatable.
- Around 1 — leaves the learned probabilities roughly as they are.
- High (above 1) — flattens the distribution. Unlikely tokens get a real chance. Output becomes varied, then odd, then incoherent.
Here is one next-token choice after "The noise is coming from the" at three temperatures, treating these three tokens as the only options:
Token T = 0.5 T = 1.0 T = 2.0
brake 0.78 0.60 0.47
chain 0.20 0.30 0.33
bell 0.02 0.10 0.19The ranking never changes. What changes is how strongly the favourite dominates.
At temperature 0, many systems simply take the top token each time, which is called greedy decoding. Results are then very consistent, though not always perfectly identical between runs.
The shortcut
Ask: "Is there one right answer, or do I want many good ones?"
- One right answer (classify, extract, convert, calculate) → 0 to 0.3.
- Many good answers (brainstorm, write variations, invent names) → 0.7 to 1.
Triage
Each email has one correct label. We want the same answer every run. Set temperature to 0.
The trap: believing low temperature makes the model correct. It only makes it consistent. If the prompt lacks the facts, temperature 0 gives the same wrong answer every time.
Drills
Answer fast — trust the rule.
What does lowering the temperature do to the next-token probabilities?
Which task suits a high temperature best?
The triage prompt gives the same wrong label every time at temperature 0. What does that tell you?
What usually happens if you set temperature very high, such as 2?
Remember
- Low temperature for right answers, higher temperature for fresh ideas.
- Shortcut: one right answer → 0 to 0.3; many good ones → 0.7 to 1.
- Trap: low temperature means consistent, not correct.