Showcase / Benchmark
One line in a style sheet cut a long conversation's requests from 4.6 seconds to 0.5
An agent in a long conversation sends the whole conversation to the model on every step. Model servers have a trick that makes this cheap, and most agents break it without knowing. We fixed that with one line, and each request went from 4.6 seconds to 0.5.
The trick: the server remembers the start of the last prompt
A model reads its prompt from the first token to the last, and that reading is most of the work. When a new prompt starts with the same tokens as the previous one, the server reuses the work it already did for them and only reads what is new. This is called the prefix cache.
It suits an agent well, because each prompt is the last one plus one more turn.
How a full conversation breaks it
A model can only read so many tokens at once. That limit is its context window. When the conversation no longer fits, the agent has to drop something, and most agents drop the oldest turn, just enough to fit the new one.
Now the prompt starts at a different place than last time. The beginnings no longer match, and the server reads the whole prompt again. From then on this happens on every turn.
The fix: drop a quarter at once
If the agent drops the oldest quarter of the conversation in one go, it has room for many turns before it must drop again. In between, every prompt starts the same way as the one before it, and the server's saved work is good.
In Affordance, how the prompt is laid out is written in a style sheet, the way a web page's layout is written in CSS. The highlighted line is the fix.
history {
max-tokens: 75win;
overflow: drop-start;
drop-step: 25%;}The first two lines say: the history may use 75% of the context window, and when it is over, whole turns are dropped from the oldest end.
This bar is a drawing of the rule, to show the idea. The measurement is below.
What we measured
We ran a conversation of 150 turns, about 600 tokens each, against SGLang serving Qwen3.6-27B on one graphics card. The history was allowed 24,000 tokens, so it filled up after about 45 turns.
| Once the conversation was full | Part of the prompt the server remembered | Time for one request |
|---|---|---|
| dropping a little each turn | 0% | 4,605 ms |
dropping a quarter at once, with drop-step: 25% | 89% | 492 ms |
With the new line, the start of the prompt moved 10 times in 105 turns. Without it, the start moved on every turn, and the server read all 23,878 tokens again each time.
The numbers are averages over the 105 turns after the conversation filled, from one run.