Short-Term Memory — Keeping the Working Context Lean
Planning a three-city trip, Tripmate makes 30 tool calls. Each train search returns a big block of text, and by call 25 the context is nearly full, every call is slow and expensive, and the model starts overlooking details. Short-term memory is the working context of one task, and it needs managing like a desk that keeps filling up.
The rule
Keep the goal and the decisions; shrink the raw material.
Short-term memory is everything carried from turn to turn inside the context window: the instructions, the conversation, the plan, and tool results. Three techniques keep it useful:
- Trimming — cut tool results down to what matters before storing them. Fifteen trains with every stop become the three candidates, with times and fares.
- Sliding window — keep the most recent turns in full and drop older ones. Simple, but anything important from early on must be saved separately first.
- Summarization (compression) — replace a long stretch of old turns with a short summary written by the model: decisions made, facts learned, open questions.
What must survive every cut:
- the user's goal and constraints (dates, budget, "vegetarian"),
- decisions already made ("Pune → Jaipur leg: booked the Friday sleeper"),
- what is still open ("Jaipur → Udaipur leg: not yet searched").
A healthy working context often has three layers:
[pinned] Goal: Pune → Jaipur → Udaipur → Pune, 15–22 Oct, ₹8,000 total, vegetarian
[summary] Leg 1 booked (Friday sleeper, ₹1,180). Leg 2 options: 2 trains shortlisted.
[recent] last 6 turns in fullThe shortcut
Before anything enters the context, ask: "Will the next decision need this exact text, or just its conclusion?" Just the conclusion → store the conclusion. Most raw tool output falls in that bin.
Before
Turns 1–12: the request, four searches of about 2,000 tokens each, a comparison, a booking. Around 12,000 tokens.
The trap is summarising away something that turns out to matter, such as a booking reference. Pin exact identifiers, numbers and constraints verbatim; summarise only the reasoning and chatter around them.
Drills
Answer fast — trust the rule.
Which technique replaces old turns with a short written recap?
What is the main risk of a plain sliding window?
Which detail should be kept word-for-word, not summarised?
A search returned 15 trains; Tripmate needs only the 3 that fit. What should enter the history?
Remember
- Short-term memory is the working context: keep goals, constraints and decisions; shrink raw material.
- Shortcut: does the next decision need this exact text, or just its conclusion?
- Trap: summarising away exact values — pin IDs, numbers and constraints verbatim.