LoopArena: Loop Engineering Benchmark — Best Controller 24.69% SSR

Loop engineering is the practice of organizing coding agents into monitored, multi-step loops — assign, review, verify, decide, repeat. It's the pattern behind every serious agent platform today. The question nobody has answered with numbers: do the models running the loop actually help?

LoopArena, released today by the DreamX Team (AMAP-ML), is the first benchmark designed to answer that. I cloned the repo, installed it, validated the public results against the raw outcomes, and the numbers tell a story that should make everyone building agent loops nervous.

What LoopArena Measures

LoopArena separates two roles. The Worker is a fixed coding agent (Qwen3.7-Plus) that edits the repository and runs tools. The Controller is the model under evaluation — it receives a structured Evidence Packet after each coding round and returns a Loop Contract: what the Worker should do next, what to verify, or whether to stop. The Controller has no coding tools and no direct access to the repository. It only decides.

sequenceDiagram
    participant Worker
    participant Reporter
    participant Controller
    Worker->>Reporter: completes assignment
    Reporter->>Controller: Evidence Packet
    Controller->>Worker: Loop Contract (advance/verify/stop)
    Worker->>Worker: executes next step

This design isolates the decision quality of the Controller from the execution quality of the Worker — a cleaner signal than the end-to-end "did the agent succeed" that most benchmarks report.

The Numbers: Controllers Barely Beat No Control

LoopArena defines three settings. Type I is a multiple-choice version (pick the best next contract from 4 options, 90 items). Type II is condensed coding tasks (27 tasks). Type III is full coding tasks (27 tasks). Each task runs 3 times per controller. The metric is Strict Success Rate (SSR).

Here's the panel, verified by reproducing the published summary from the raw outcomes.jsonl:

Type III — Full Task SSR

The best controller — GPT-5.5 — succeeds on roughly one in four tasks. The no-control baseline (Worker gets the task and runs until it finishes or exhausts its budget) succeeds on roughly one in five. The gap is 6.17 percentage points. GLM 5.2 actually underperforms no control by 2.47 points — the Controller's decisions actively hurt the outcome.

Type I — Contract Accuracy

When the task is reduced to multiple-choice (pick the best next step), scores climb:

Type I tells a different story — GPT-5.5 is nearly 88% accurate at picking the right next step from four options. The gap between 87.78% (Type I) and 24.69% (Type III) is the gap between knowing what to do and making it happen through a multi-step loop. That gap is the whole problem.

Type II vs Type III: The Condensed Task Mirage

Type II (condensed tasks) shows a bigger spread: GPT-5.5 at 51.85% SSR vs no-control at 39.51%. The 12.34pp gap looks like real controller value. But on Type III (full tasks), the gap collapses to 6.17pp. The ordering between controllers is stable (Spearman's ρ = 0.9747), but the absolute scores are inflated on Type II.

This is the methodological takeaway: condensed tasks overestimate controller value. If you only evaluated on Type II, you'd conclude loop engineering adds meaningful value. Type III shows the truth: on real, long-running tasks, the controller barely helps.

What This Means for Loop Engineering

Three takeaways from this benchmark:

1. Controllers are still near noise. The best model scores 24.69% on full tasks. The baseline is 18.52%. That's a 6-point improvement over "just let the Worker run." I'd want 40%+ before I'd trust a controller loop in production.
2. Some controllers actively hurt. GLM 5.2 at 16.05% — worse than letting the worker run unguided. The "controller" role is not automatically beneficial. Bad loop decisions compound, and the Worker can't override them.
3. Type I is a different skill. GPT-5.5 scores 87.78% on contract selection but 24.69% on full execution. Knowing the right next step and executing it through a multi-turn loop are different capabilities. Benchmarks that only test step-level decisions miss the hard part.

LoopArena is v0.1.0. The protocol is clean, the reproducibility checks pass, and the separation of Controller from Worker is a real improvement over end-to-end agent benchmarks. But the headline is sobering: the loop engineering stack is not delivering yet. The models running the loop aren't smart enough to add much value. The data is there — the conclusion is unavoidable.