Carryover Effects in Reused KV Caches Across Agent Handoffs and Policy Updates
What does a stale KV cache do to the model that reads it?
Abstract
A carryover effect is what a model shows when it reads a KV cache that another computation wrote: the text or the weights differ from the writer's, and the reader's next-token distribution shifts with them. The systems that serve and train long-horizon agents produce such reads by design, whenever a later stage rebuilds its prompt from text an earlier stage processed, or a pipelined trainer serves rollouts from a cache an earlier policy wrote. We ask what predicts the effect and measure it on two pre-registered records, one for the text axis and one for the weights axis.
On the text axis, the record is 35 coding-agent handoffs whose sender histories run from 35K to 80K tokens. For each, we measure the oracle removal fraction: the share of matched tokens we must remove before the mean key deviation of the remainder falls within a tolerance drawn from a cross-model map's held-out error. Under same-model reuse, the fraction is zero on every one of the 35 handoffs. Its median is zero as well on shorter handoffs, on a second model family, and at two further receivers. Across the four pair cells, the cross-model map must remove 73% to 96% of the matched tokens, and truncating the senders to the short cell's length leaves the deviation above the short cell's level.
On the weights axis, a policy reads a cache that its own earlier weights wrote, up to 32 GRPO steps later. On held-out text, the per-token KL stays inside the registered band on three model families and two seeds. On the policy's own prompts, it sits three to four orders of magnitude higher and is outside the band from the first step, where the stale cache already flips the greedy answer on 71% of GSM8K problems, and on 84% to 90% at later steps. At 32 steps, the stale cache costs the trained policy six of the seven accuracy points it had gained. In the opening windows of the agent trajectories that MLPerf Agentic Inference serves, per-token KL stays within the same band, three orders of magnitude below the policy's own prompts.
At their operating extremes, the two axes land in the same decade: a re-rendered handoff reads at a median of 0.14 nats with 90% greedy agreement, and a policy's own prompts at 32 steps read at 0.09 nats with 90%. The text's relation to the computation that wrote the cache predicts the carryover effect; weight distance, model family, and context length do not.
On the name. In a crossover trial, a carryover effect is the residue of one treatment period on the next. Here the residue is the cache's: a representation of a computation that is not the reader's own, carried into the reader's next token.
The weights axis
A policy at GRPO step g reads the cache its step-0 self wrote. On text the trainer never touched, the stale and fresh readers agree to within the sealed band at every lag; on the prompts the trainer optimises, they part at the first step.
The text axis
At a handoff the receiver re-renders its prompt from text the sender processed, and reads the sender's cache on the tokens that match. The oracle removal fraction is the share of those tokens an oracle sorting by deviation must remove before the mean key deviation of the rest falls within the tolerance; it is a distance from the reader's own computation, not a recomputation budget.
| Cell | Entry | Handoffs | Same-model median f* | Cross-model median f* |
|---|
Benchmarks
Five texts carry the measurements. Each is public, each enters a record through a registered entry, and none was written for this paper: the handoffs are what a leaderboard's agents actually did, the prompts are what the trainer actually optimises, and the benchmark windows are what an inference benchmark actually serves.
| Text | Axis | What it supplies | Size as read | Entries |
|---|---|---|---|---|
| SWE-bench leaderboard trajectories composio SWE-Kit submissions of 16 and 25 October 2024 (astropy and django instances) | text | The recorded coding-agent handoffs: a Claude 3.5 Sonnet solve thread whose text an o1-mini stage then re-renders for summarisation and patch selection. The sender's history and the receiver's rebuilt prompt are the two sides of every handoff. | 68 handoffs observed; 35 long (sender 35K to 80K tokens) and 25 short under the Qwen pair; 28 short and 32 long under the Llama pair | linear-ceiling 0013, 0025, 0029, 0036, 0044, 0051 |
| GSM8K test split | weights | The trainer's own prompts: the GRPO policy trains on GSM8K, so its rollouts are the text a stale cache meets in a pipelined trainer. Also the task endpoint (statistic C): whether the greedy answer changes and what the stale cache costs in accuracy. | 1,319 test prompts, six readers (lags 1 to 32), greedy at a 256-token cap | lag-ladder 0017, 0018, 0027 |
| FineWeb-Edu sample-10BT | weights | Held-out prose the trainer never touched: the text on which stale and fresh readers agree to within the sealed band at every lag. | Ten held-out sequences of 1,024 tokens (reads at 256), shuffle seed 0, drawn once by the instrument | lag-ladder 0005, 0016 |
| The reader's own continuation on-path text | weights | Each reader's greedy continuation from the held-out prefixes: text of the reader's own making that is still not the trainer's, the second held-out reading. | Same ten prefixes, one continuation per reader | lag-ladder 0016, 0019 |
| MLPerf Agentic Inference MLCommons, 613 trajectories: 113 DeepSWE coding traces and 500 Workato workflow conversations | both | What an agent inference benchmark serves. The opening windows are read under the stale-cache readers on the weights axis and registered for the cross-model mapper on the text axis, the same fifty windows on both records. | 50 windows, 25 per suite, the first 1,024 tokens of each trajectory (median first user turn about 1,100 tokens) | lag-ladder 0030, 0036 · linear-ceiling 0064 |