Every production AI system that routes between multiple models faces the same unspoken question: how much should I pay to know which model is best for this query?
The naive answer — try them all — is correct but expensive. The thrifty answer — pick the cheapest one — sacrifices quality. And every intermediate strategy (embedding similarity, a lightweight classifier, a fast approximation) has its own cost-quality tradeoff that nobody formalizes.
Fisch, Trivedi, Huot, Cohen, Kaisers, Lapata, Larson, and Eisenstein — a joint team from Google DeepMind, AWS, and the University of Edinburgh — dropped "Pandora's AI Model Routing Box" on arXiv yesterday. They formalized the problem as an instance of Pandora's Box, the classical optimal search problem with costly inspection. The result is a routing policy that knows when to stop estimating and start answering.
The Problem With Model Routing
Here's the scenario: you have N specialists — a cheap fast model, a capable medium one, an expensive frontier model, maybe a fine-tuned variant with retrieval. A query arrives. Which one should answer?
To decide, you need to estimate each specialist's expected return. But that estimation has a cost:
- Cheap estimators (embedding similarity, a fast classifier) are noisy — they'll route the wrong model a significant fraction of the time.
- Accurate estimators (a fine-tuned model that first retrieves documents or generates partial reasoning traces) give better routing but cost nearly as much as just running the specialist.
Most routing systems pick one estimator and live with the tradeoff. The paper's insight: you don't have to. You can start with a cheap estimate, and only pay for a better one when the value of knowing outweighs the cost of estimating.
graph TB
subgraph "Query Arrives"
Q["Input query"]
end
subgraph "Routing Decision"
Q --> C1["Cheap estimator
(embedding match)"]
C1 --> VOI{"Value of
Information > Cost?"}
VOI -->|No| SM["Selected model A
Answer now"]
VOI -->|Yes| C2["Expensive estimator
(fine-tuned reranker)"]
C2 --> VOI2{"Value of
Information > Cost?"}
VOI2 -->|No| SM2["Selected model B
Answer now"]
VOI2 -->|Yes| C3["Full evaluation
Run all specialists"]
end
style SM fill:#1a3a2a,stroke:#2ecc71
style SM2 fill:#1a3a2a,stroke:#2ecc71
Pandora's Box, Applied
The Pandora's Box problem, formulated by Martin Weitzman in 1979, asks: you have N boxes, each with a known distribution of rewards but a fixed cost to open. You can open boxes one at a time, and once opened, you see the actual reward. You want the best reward at minimum expected cost. The optimal strategy has a closed-form solution: a reservation price for each box that determines whether it's worth opening.
The paper maps model routing onto this framework beautifully:
- Each specialist is a box — you don't know its true value for a query until you run it.
- Each estimator is a way to peek — at a cost, you get a noisy signal of the specialist's value.
- The reservation price is the value of information: if the best estimate you've seen so far exceeds the cost of looking further, stop and answer.
Under a Gaussian signal model, the resulting policies have closed-form value-of-information expressions. No simulation needed. The centralized policy (Pandora's Router) decides, for each specialist and input, whether refining the estimate is worth its cost. The decentralized version (Pandora's Bidder) lets specialists self-assess and bid.
The Numbers
The experiments span three domains, and the results are consistent:
- Multi-LLM benchmark: Pandora's Router matches the routing quality of exhaustive estimation (running the expensive estimator on every specialist) while querying the expensive estimator far less often. The exact savings depend on the domain, but the gap is systematic.
- Retrieval-augmented specialists: When specialists include RAG setups with different retrievers, the value-of-information approach correctly skips expensive estimation for queries where the cheap embedding signal is decisive.
- Variable reasoning depth: For models with controllable reasoning chains (short vs. long chain-of-thought), Pandora's Router learns to request the long version only when the short one's signal is ambiguous — intuitively what you'd want, but formalized as a decision problem with a guarantee.
The decentralized Pandora's Bidder setting adds an interesting wrinkle: when competing estimates are noisy, value-of-information reasoning can actually increase a strategic specialist's utility at others' expense. This is the paper's most provocative result — it suggests that in a market of self-interested models, information asymmetry can be exploited.
Limitations
Three caveats worth noting:
The Gaussian signal assumption. The closed-form results depend on the estimators following a Gaussian noise model. Real estimators may have more complex error structures — fat tails, systematic biases, or adversarial inputs. The paper acknowledges this but doesn't explore robustness to misspecification.
Estimator costs are fixed and known. In practice, estimator costs vary with input complexity and load. A fast classifier on a short query is genuinely cheap; on a 100-page document, it's less so. The model assumes you know each estimator's cost ahead of time.
Only pairwise comparisons. The framework handles one estimator per specialist. Real systems often have a cascade: three increasingly expensive estimators feeding into a routing decision. Extending Pandora's Box to multi-level inspection is left for future work.
Why Builders Should Care
If you run a multi-model system, this paper gives you the math you've been missing. Every routing system implicitly makes a decision about how much to spend on routing versus answering. Pandora's Router tells you the optimal stopping point — and it has a closed form.
It's not just about cost savings. It's about knowing you're at the right point. Most routing systems are tuned empirically: try a threshold, measure quality, adjust. Pandora's Router gives you a principled answer: stop when the value of the next estimate is less than its cost.
The decentralized setting is the dark horse. If you're building a marketplace of models, or an agent that subcontracts to other agents, the Pandora's Bidder results suggest that information asymmetry creates strategic opportunities. Models can over-invest in self-assessment to capture more value — and the system needs countermeasures.
This is the kind of paper that doesn't tell you something new about AI. It tells you something old about decision theory, and shows you it was the right tool all along.
- Pandora's AI Model Routing Box: Efficient Allocation with Costly Value Estimation — Adam Fisch, Shubhendu Trivedi, Fantine Huot, William W. Cohen, Michael Kaisers, Mirella Lapata, Kate Larson, Jacob Eisenstein, Aug 2026