Every agent builder has stared at the token counter and wondered: do I really need to send every tool's full schema on every request? The answer has always been "well, the model needs to know what tools exist." But that comes at a cost — one that grows linearly with the number of tools in your agent's toolkit.
ReCache (Fang et al., arXiv 2608.19662) is the first systematic answer to that cost. It's a framework that independently caches tool and skill schema representations so they don't need to be re-encoded on every request. The headline numbers are the kind that make you re-examine your inference bill: 92.43% reduction in KV-tensor memory, 3.655× faster time-to-first-token, and essentially zero degradation in tool invocation accuracy (82.3% vs 82.4%).
The Problem: Your Tools Are Repeating Themselves
Here's the architecture that every tool-using LLM agent shares: the system prompt lists available tools, each with a schema describing its purpose, parameters, and return type. Every user request is appended to this context, and the model attends over everything — tool descriptions, previous tool calls, user utterances — to decide what to do next.
Standard prefix caching (PagedAttention, vLLM's automatic prefix caching) can reuse KV states when the exact same prefix repeats. But tool-augmented agents rarely have identical prefixes. The same tools appear in different orders, combined with different subsets of tools depending on the task. One request might use calculator + search + database; the next uses search + email + calendar. Prefix caching fails because the prefix changes every time.
The implicit assumption that tools are independent from one request to the next is wrong — but so is the assumption that the model needs to re-read every tool schema every time. The truth is somewhere in the middle, and ReCache finds it.
The Method: Resource-Wise Attention
ReCache's core insight is elegant: tool schemas don't need to interact with each other during encoding. The model doesn't need to attend from the calculator schema to the email schema. It needs to attend from the user request to the relevant tool schema.
This observation leads to three components:
- Resource-wise attention. Each tool schema attends only to itself and to the user context — not to other tool schemas. This breaks the cross-tool dependencies that prevent standard prefix caching, producing KV blocks that are invariant to tool combination and ordering.
- Contribution-selected routing. Not every layer and KV-head in the model contributes equally to tool invocation. ReCache identifies the subset that matters and restricts resource visibility to those routes, pruning away computation that doesn't affect the output.
- Structural and semantic pruning. Tool schemas contain fields that are structurally required (name, description, parameters) and fields that are semantically critical for invocation (parameter types, required flags, enum values). ReCache prunes the rest — examples, deprecation notices, verbose descriptions — keeping only what the model actually uses to decide which tool to call.
The result is a cache of reusable, pruned KV blocks. When a tool appears in a new request, its cached representation is loaded in place of the full re-encoding, skipping the cross-tool attention computation entirely.
The Numbers: What 92% Less KV Memory Looks Like
The evaluation is run on a composite benchmark assembled from 7 public tool- and skill-use datasets, including resource-disjoint tests where the tools in the request have never appeared together before. This is the hardest case — and it's where prefix caching fails completely.
- Invocation F1: 82.3% (ReCache) vs 82.4% (dense baseline). The difference is noise. The model's ability to pick the right tool is essentially unchanged.
- Time-to-first-token (resource-wise attention): 3.655× faster. The prefill phase — where all schemas are encoded — is the bottleneck. ReCache turns it into a cache lookup.
- KV-tensor memory: 92.43% reduction. From storing full tool schemas for every request to storing compact, pruned KV blocks that are reused across requests.
- End-to-end attention speedup (full framework): 1.423×. The speedup from caching and pruning compounds beyond the prefill phase.
These numbers hold up in the hardest setting — the resource-disjoint test where tools are never seen together during the same context. That's the regime where naive caching strategies collapse, and ReCache's independence assumption pays off.
Limitations
The paper is honest about where it doesn't work yet:
- Static schemas only. ReCache assumes tool schemas don't change between invocations. Dynamic tools — where a schema parameter depends on the user's specific state — break the cache. The paper acknowledges this as future work.
- 7 datasets is a start. The benchmark is well-constructed but doesn't cover multi-turn agent loops where the same tool is called repeatedly with different arguments in the same session. The caching behavior there is less clear.
- Contribution routing is trained, not discovered. The layer and head selection is learned from a specific set of models. It's not clear how well the routing generalizes across model families (the paper tests on LLaMA and Qwen architectures).
- No online deployment results. The evaluation is in controlled inference settings. Real web-serving scenarios with concurrent request batching may reveal interaction effects between cached blocks from different requests.
These are early-stage limitations, not fundamental ones. The architecture is clean enough that most have plausible fixes.
Why Builders Should Care
If you're running an agent that uses more than 5-10 tools, ReCache's 92% memory reduction directly changes your operational economics. Every tool schema you add currently multiplies the KV cache memory for every request. With ReCache, adding a tool adds a fixed-sized KV block that gets cached once and loaded on demand.
The practical implication: the cost of adding tools becomes sublinear. Your 50-tool agent no longer pays 50× the prefill cost per request. It pays a one-time encoding cost per tool, plus a minimal lookup cost per request. That changes how you design agent toolkits — you stop worrying about schema overhead and start thinking about tool quality.
The deeper lesson is about where LLM inference optimization is heading. FlashAttention solved the attention computation bottleneck. PagedAttention solved the KV cache fragmentation bottleneck. ReCache solves the schema repetition bottleneck that neither of those approaches addresses. It's a layer above the attention kernel, and it's the kind of systems-level thinking that makes the difference between a research demo and a production deployment.
The code is available on GitHub, and the architecture is simple enough that a motivated engineer could implement the core idea (resource-wise attention + caching) as a middleware layer in their agent framework. The pruning heuristics can start simple — just exclude description fields and example values — and get more aggressive as you validate against your specific tool set.
- ReCache: Efficient KV Cache Reuse and Compression for Tool-Augmented LLM Agents — Yichu Fang, Sitong Wei, Haozhe Hu, Xiaoyu Shen (20 Aug 2026)
graph TD
subgraph "Standard Agent Loop"
A[User Request] --> B[Tokenize All Tools + Request]
B --> C[Full Cross-Tool Attention]
C --> D[Model Decodes Tool Call]
end
subgraph "ReCache Agent Loop"
E[User Request] --> F[Cache Lookup per Tool]
F --> G[Load Pruned KV Blocks]
G --> H[Resource-Wise Attention\nTool attends only to self + request]
H --> I[Model Decodes Tool Call]
end
subgraph "Savings"
J[3.655x TTFT Speedup]
K[92.43% Less KV Memory]
L[82.3% vs 82.4% F1\nNo accuracy loss]
end
I --> J
I --> K
I --> L