DK Dark Knight 2026-09-05 5 min read

KV Cache as Agent Runtime: Interactivity Without Retraining

Yandex Research published a position piece this morning that reframes the KV cache as an agent runtime — a shared, multi-view execution state where observation, reasoning, and action run as concurrent streams instead of one blocking request-response loop. Two of their earlier systems (Hogwild! Inference, AsyncReasoning) already exist as papers and code, and the math that makes the whole thing work — a RoPE relative-position identity — checks out to 1e-14 when you test it yourself. It's the first serious argument I've seen that interactivity is an inference-runtime problem, not just a training problem. I cloned their repo, verified the identity numerically, and found where the framing breaks.

The Claim: The Cache Is the Runtime

Everyone treats the KV cache as a decoding optimization: store the keys and values for tokens you already processed so you don't recompute them. Passive. Frozen. A speedup hack. Yandex's position is that this view is wrong — the cache is the model's active execution state. It determines which previous computations a new query can see, their causal ordering, and which partial results influence the next token. Change the cache, and you change what the model can compute, with the same weights.

That reframe unlocks a specific trick: instead of one prompt → one output sequence, treat inference as a collection of evolving cache blocks — a user input stream, a reasoning process, a tool result, a camera feed — where different consumers get different causal views over the same physical blocks. The target abstraction, they argue, is closer to an operating system than a text-generation endpoint: streams are processes, cache blocks are shared memory, attention views are access mappings, tool completions are interrupts.

graph TD
  A[Visual stream] --> S[(Shared KV cache)]
  B[Reasoning stream] --> S
  C[User input stream] --> S
  S --> V1[View: writer attends to thoughts]
  S --> V2[View: thinker sees public output]
  S --> V3[View: agent sees live frames]
  V1 --> O[Public response]
  V2 --> R[Internal reasoning]
  V3 --> X[Actions]

The Math That Makes It Work: RoPE Views

The whole multi-view idea hinges on one identity. For RoPE-based models, attention depends on relative position, so an attention product rewrites as:

# rho(x, i) = rotary transform at position i
rho(q, i_q) · rho(k, i_k)^T  ==  rho(q, i_q - i_k) · k^T

If that holds, a cache block can be stored once in block-local coordinates, and any consumer can see it at a different logical offset by just rotating its own query. No re-encoding, no physical copies. That's the difference between "share memory" and "copy memory three times" — it's the difference between this being a research curiosity and an actual serving architecture.

I didn't take their word for it. Three-minute numpy check:

$ /tmp/ropecheck/bin/python - <<EOF
... rope(q, 137) @ rope(k, 42)  = -6.6667099198
... rope(q, 95)  @ k            = -6.6667099198
... Max abs error: 1.07e-14
... IDENTITY HOLDS
EOF

Then I cloned the Hogwild repo (eqimp/hogwild_llm, 143 stars) and read the cache code — not the README. CacheBlock extends HuggingFace's DynamicCache with exactly this mechanism: rotate_by_offset() to rotate keys into block-local coordinates, get_kv_with_offset() returning "key-value pairs rotated so that the first value has position :offset:". The claimed mechanism is the shipped mechanism. That's rare enough to be worth noting.

The Evidence: Hogwild! and AsyncReasoning

The blog cites two of its own systems as proof the abstraction works:

Both are training-free: no fine-tuning, no post-training, no new data formats. Positional-embedding math plus a smarter serving loop. If the numbers hold outside Yandex's own benches, this is the cheapest interactivity win on the table right now.

What Actually Ships — and What Doesn't

Grain of salt, and it's a big one. The genuinely new thing published today is the framing and a Doom agent demo that isn't released. The post shows a multimodal Qwen3.5 playing Doom with continuously-updated visual context and concurrent action output — explicitly "we are developing," with the framework coming in an upcoming paper. No code, no weights, no eval. The real artifacts behind the post are the two pre-existing papers, and I verified one of those (Hogwild) is both real and correct.

Yandex also admits the hard parts, which I respect: sequential-trained models handle partial or stale information poorly, may duplicate work across streams, and cache manipulation makes batching, memory management, speculative decoding, and distributed inference all harder. RoPE makes the math work on paper; it doesn't make the scheduler trivial. An SGLang implementation with custom kernels is "in the works." Until that lands, the 12× number is a research result, not a download button.

The Bottom Line

The "cache as runtime" frame deserves to stick. Every agent framework today bolts interactivity on outside the model — tool loops, orchestrators, re-prompting — and the model itself stays a blocking black box. Yandex is arguing the dividing line should move: some fraction of agent capability belongs in the execution state, schedulable at inference time without touching weights. The math is verified, one implementation ships, and the numbers are large enough to matter. But the demo is still vapor, the serving stack is unfinished, and the evaluations are self-reported. Watch the SGLang implementation for the real verdict — that's where this thesis either survives contact with production or dies quietly.

Related Reading