Every agent builder has been here. A tool call returns what looks like valid data. The agent uses it. The pipeline continues. And somewhere downstream, something breaks in a way that makes no sense — because the data was silently wrong.
Tool timeouts are obvious. The agent sees the failure, retries, routes around it. But what about the negative inventory count that arrives in the same format as a positive one? The cached error page that looks like a valid API response? The stale price that gets consumed as fact?
Outcome Monitors (Panthi & Abdelfattah, arXiv 2608.19303) is the first systematic framework for detecting and recovering from these silent tool failures — and the numbers are compelling enough that anyone building production agents should pay attention.
The Problem: Failures That Don't Announce Themselves
The paper draws a clean distinction: a tool call timeout or HTTP 500 is a loud failure — the agent knows something went wrong and can adapt. A silent failure is when the tool returns data that looks structurally valid but violates some property of the real world. Inventory from a cached page shows -3 units in stock. A pricing API returns yesterday's rate. A search endpoint returns results from a different query context.
The agent sees none of this. It consumes the output and reasons over garbage.
This is not hypothetical. The authors ground their motivation in a published incident taxonomy and demonstrate the failure mode across multiple model families and benchmarks. The core insight: the failure isn't in the tool — it's in the contract between the tool's output and the state of the world.
What They Built
An Outcome Monitor sits beside the agent and does three things:
- Detects violations of "outcome contracts" — invariants mined from task-disjoint traces or derived from public schemas. The contract says "inventory should be non-negative" or "price should be current within 24 hours."
- Preserves the raw result (the agent still sees what the tool returned) but appends a nonbinding advisory receipt that names the violated property and lists recovery tools.
- Recovers by leaving the action set intact — the agent can verify, retry, or switch tools based on the advisory.
The key design choice: the monitor does not override the agent. It's advisory. The receipt is nonbinding. The agent remains sovereign — it can ignore the warning, verify independently, or act on the suggested recovery tools. This matters because overridden agents learn helplessness; informed agents learn better tool use.
The contracts themselves are mined from task-disjoint traces — meaning the monitor learns invariants from observing tool use on unrelated tasks, then applies them everywhere. A contract about inventory non-negativity learned while watching a stock-check task applies just as well during a checkout task.
The Numbers
The results are concrete and cleanly controlled:
- ToolMaze (a custom benchmark for tool-use tasks with injected failures): completion rose from 10.9% to 28.1% across four models in two provider families — a clustered mean effect of +17.2 percentage points (p < .00001). The effect replicated independently in a third provider family.
- tau-bench retail (stateful retail agent benchmark): completion improved by +14.0 and +12.0 points on two tiers.
- In controlled ablations, removing the recovery-tool list from the receipt eliminated the entire gain. Restoring it recovered the effect. The diagnostic detail and timing of the advisory made no detectable difference. The recovery tools are the active ingredient.
The gains concentrate where the fault actually blocks completion — exactly where you want them. When the failure doesn't matter, the monitor doesn't slow things down.
Where It Breaks
The limitations are honest and worth knowing:
- Detection outside the mined contract vocabulary falls to 46%. If the failure doesn't match a known pattern, the monitor misses it. The advisory is still delivered (with a generic warning), but it carries no specific recovery guidance.
- The paper only tests frozen, prespecified evaluations. Real-world drift — where failures evolve over time — is not addressed.
- The monitor itself is an additional component in the agent loop. While the paper argues it's lightweight (no LLM calls at inference, just deterministic contract checking), every extra component is another surface for bugs.
This isn't a silver bullet. It's a well-targeted patch for a specific class of failures that current agent architectures are blind to by default.
Why Builders Should Care
If you build agents that call tools — and if you're reading Dispatch, you probably do — you've encountered silent failures. You've probably debugged one that took hours to trace back to a tool returning plausible-looking garbage.
What I find encouraging is the transferability of the framing. Outcome contracts mined from task-disjoint traces mean the monitor doesn't need to understand the task to detect the failure. It needs to know that inventory counts can't be negative. That's a property of the domain, not the task. This makes Outcome Monitors portable across agents and tasks in the same environment.
The control experiment tells the important story: recovery tools carry the gain. The monitor isn't useful because it flags problems — it's useful because it tells the agent what to do next. That's a surprisingly mature design insight for a paper that's been on arXiv for three days. Most failure-detection systems stop at the red flag. This one hands the agent a map.
I'd like to see this extended to dynamic contract learning — where the monitor discovers new invariants as it observes more interactions — and to multi-agent settings where one agent's silent failure can cascade through a team, creating correlated false beliefs across the system. But as a starting point for a class of failures that had no systematic treatment at all, this is solid work.
- Outcome Monitors: Recovery Affordances for Silent Tool Failures — Sugam Panthi, Rabab Abdelfattah (19 Aug 2026)
mindmap
root((Silent Tool Failures))
Problem
Tools return structurally valid garbage
Agent consumes as fact
Downstream corruption
Solution: Outcome Monitors
Detect contract violations
Preserve raw result
Issue advisory receipt
Active Ingredient
Recovery tool list
NOT diagnostic detail
NOT timing
Limits
46% detection outside vocabulary
Frozen eval only
Deterministic contracts