Tokenization — How a Model Reads

All through this course you will build one agent: Tripmate, a travel assistant that helps Asha plan train journeys. Before Tripmate can plan anything, the language model inside it has to read Asha's message. It does not read letters or words the way you do. It reads tokens, and almost every cost, limit and quirk of a model traces back to them.

The rule

Models never see text — they see numbered chunks called tokens.

A tokenizer cuts text into pieces from a fixed vocabulary and swaps each piece for a number. Common words are usually one token. Rare words, names and long numbers get split into several. Spaces and punctuation are often folded into the pieces too. An illustrative split (the numbers are made up):

The exact split and the numbers depend on the model's tokenizer; two models can cut the same sentence differently. What stays true everywhere:

• English prose averages roughly three-quarters of a word per token, so 1,000 tokens is about 750 words. Treat this as a rough guide, not a rule.
• Other scripts and code often need more tokens for the same meaning. A Hindi sentence, a JSON blob or a long ID can cost far more tokens than its length suggests.
• The model predicts one token at a time. When it "writes", it is picking the next token again and again.

The shortcut

When a question involves a model's limits, speed or price, translate the text into tokens first. Ask: "How many tokens is this, roughly?" Words × 4 ÷ 3 is a good first guess for plain English.

The common mistake is to think in characters or words. A model that "can't count the letters in a word" is not being careless — it never saw the letters, only the chunks.

Drills

Answer fast — trust the rule.

Remember

• Models read tokens — numbered chunks — never raw letters.
• Shortcut: words × 4 ÷ 3 ≈ tokens for English prose; code and other scripts cost more.
• Trap: forgetting that instructions and tool descriptions are tokens too, sent every turn.