AL Alfred 2026-09-02 4 min read

CordisBench: Agent Lifecycle Reasoning Burns 3,000 Tokens

Dynamic agent harnesses let language models change the software that shapes their own execution — swap a plugin, patch a tool, rewire a dependency. That flexibility comes with a new reasoning burden: a local change propagates through dependency graphs and teardown orders, and the model has to predict what breaks before it breaks. CordisBench, a new 1,200-question benchmark from Damien Sileo and Dimitri Kachler, is the first measurement of this "lifecycle reasoning." The finding is uncomfortable on two levels: models degrade noticeably as component systems grow, and where they do reason, they burn nearly 3,000 reasoning tokens per question — on instances where a deterministic reference semantics answers every scored outcome for free.

The Problem: Self-Modifying Agents Can't See What They're Breaking

Once a harness lets a model mutate its own tooling, correctness stops being a prompt problem and becomes a systems problem. Change one plugin and the blast radius runs through dependencies, initialization, and cleanup: components that depended on the old version, state that must survive teardown, reconfigurations that only succeed under a specific order. Prior agent benchmarks measure planning, tool use, and code repair — none measure whether a model understands the lifecycle of the components it is allowed to rearrange. CordisBench isolates exactly that narrow, increasingly load-bearing skill.

The Method: 1,200 Questions Over the Cordis Runtime

The benchmark pairs a controlled formal setting with programs executed against Cordis, a runtime that manages component dependencies and cleanup. Three efficiency-oriented models are evaluated at low reasoning effort, with 2 to 32 relevant interactions per instance and deterministic task-specific scoring. Four task families cover the failure surface:

flowchart LR
    A[Model edits plugin] --> B[Cordis dependency graph]
    B --> C[Teardown order]
    C --> D[Predict final state]
    C --> E{Condition holds under all/some orders?}
    D --> F[Choose reconfiguration]
    E --> F
    F --> G{Does it succeed when executed?}
    style G fill:#7f1d1d

Of the 1,200 questions, 528 are executable end-to-end — for those, every observation and action outcome used for scoring can be checked against ground truth.

The Results: Degradation at Scale, 3,000 Tokens With No Payoff

Models handle small systems well and "grow less reliable as more interactions become relevant" — the failure concentrates exactly where real harnesses live, at larger component graphs. The weakest spots are predicting final state after teardown and reasoning across teardown orders, both pure dependency logic. Cranking up inference effort recovers "marked gains" for some models, confirming the failure is partly a willingness-to-spend problem. But the cost side is the story: on the 16-interaction subset, GPT-5.6 Luna uses nearly 3,000 reasoning tokens per question at medium effort. And here is the kicker — an independent finite reference semantics agrees with Cordis execution on every observation and action outcome used for scoring across all 528 executable questions. For these instances, the model's expensive reasoning buys exactly nothing that a runtime check doesn't already provide.

Limitations: Controlled Formality, Not Production Chaos

CordisBench is a toy universe by design — formal programs, no network I/O, no real side effects, no third-party code. The reference-semantics result is verified only on the 528 executable questions, not the full 1,200, and the authors don't claim it transfers to production harnesses with messy state and real tools. Three efficiency-oriented models at low effort is a narrow sweep; a reasoning-heavy flagship may behave differently. And there's a definitional caveat worth naming: on instances where a runtime is the source of truth, "lifecycle reasoning" is arguably a systems check wearing a benchmark's clothes — which is exactly why the token bill is such damning evidence.

Why Builders Should Care

If you ship agents with tool loops, plugin loaders, or MCP-adjacent ecosystems, the model is the one enforcing lifecycle correctness today — and this paper says it quietly fails as the component graph grows. Three rules fall out. First, don't let the model be the only thing that knows the dependency graph: push impact analysis and teardown validity into the runtime, where they're deterministic, and let the model decide only what to change. Second, treat reasoning tokens as a line item — a benchmark where a reference implementation is free is a masked cost benchmark, and your inference bill is hiding the same waste. Third, when you build evals for self-modifying agents, include an executable oracle: if a deterministic check can score the task, quote the token spend right next to the accuracy number.

Related reading