Tokens and the Context Window

The owner had an idea: paste the shop's entire 300-page parts catalogue into every triage prompt, "so the model knows everything". The API rejected the request as too long. Even a shorter catalogue would have made every email slow and costly to process. Both problems come from the same two ideas: tokens and the context window.

The rule

Models count in tokens, and input plus output must fit one window.

Tokens are the units a model reads and writes. Common words are often one token; rare words, names and numbers split into several. For English, a rough rule of thumb is about four characters, or three quarters of a word, per token. Other languages and code can use noticeably more tokens for the same meaning.

Exact splits vary by model, so treat these as illustrations, not facts to memorise.

The context window is the maximum number of tokens the model can handle in one call. It is shared: the system instructions, the chat history, pasted documents, your question and the model's reply all count against the same limit.

Tokens matter for three practical reasons:

• Cost — paid APIs usually charge per token, in and out.
• Speed — more tokens take longer to read and to generate.
• Attention — a key fact buried in a very long input is easier for the model to overlook than one in a short, focused prompt.

The shortcut

Treat the window as a budget: fixed instructions + variable input + room for the answer must stay under the limit, with a margin.

The trap is thinking "bigger window, so paste everything". Even when it fits, stuffing the prompt costs money on every call and dilutes the facts that matter. Include what the task needs, and no more.

Drills

Answer fast — trust the rule.

Remember

• Models count in tokens, and input plus output must fit one window.
• Shortcut: budget fixed instructions + variable input + answer room.
• Trap: pasting everything just because it fits.