Streaming and Prompt Caching
The Pedal Point assistant now answers correctly, but customers complain: they hit send, stare at an empty box for several seconds, then the whole answer appears at once. Meanwhile the bill shows the same long policy text being paid for thousands of times a day. Two techniques fix these, and neither changes what the model says.
The rule
Stream to feel fast; put stable content first so it can be cached.
Streaming. The model writes one token at a time anyway. Without streaming, your app waits for the last token before showing anything. With streaming, the API sends small pieces as they are produced and your interface displays them straight away. The total time barely changes, but the time to first token, the pause before anything appears, drops sharply. People judge speed by that pause.
Streaming needs care in two places:
- Structured output. Half a JSON object is not valid JSON. Stream to a buffer, and validate once it is complete.
- Output checks. If a safety filter must approve text before a customer sees it, you need to check chunks as they go or hold them back.
Prompt caching. Most requests to the assistant start the same way: the system prompt, then the same policy text. Only the end differs. Many providers can cache the work done on a prompt's beginning and reuse it when the next request starts with exactly the same tokens. Reused parts are usually cheaper and faster. Some providers do this automatically, others need you to mark what to cache; the details differ, so read your provider's documentation.
Caching only helps if the beginning is identical. One changed character early on, like a timestamp at the top, and nothing after it can be reused.
The shortcut
Stable first, changing last. Order every prompt from least likely to change to most likely to change.
List the parts
System prompt, policy text, customer's name and order history, the new question.
The trap is thinking streaming makes the model faster. It makes waiting feel shorter; long answers still take as long to finish.
Drills
Answer fast — trust the rule.
What does streaming mainly improve?
Which prompt order gives caching the best chance?
A prompt starts with the current time, then the system prompt and policies. What happens to caching?
The assistant streams JSON for the ticket sorter. When should the app validate it?
Remember
- Stream to feel fast; put stable content first so it can be cached.
- Stable first, changing last — no timestamps at the top.
- Streaming shortens the wait for the first words, not the whole job.