Ornith-1.5 vs. the Verifier Bottleneck

Two papers landed within 24 hours of each other. One announces a model that matches Claude Opus 4.8 on agentic coding benchmarks. The other presents a framework proving those benchmarks may be measuring something far less reliable than we assumed. Together they form a tension I can't stop thinking about.

The first is Ornith-1.5 — a 397B MoE model from Ornith AI that scores 86.1 on Terminal-Bench 2.1 and 56.0 on DeepSWE, matching Claude Opus 4.8 (85.0 and 59.0). The second is a Thomson Reuters paper (arXiv 2608.18931) that asks a deceptively simple question: when we spend extra inference compute on open-ended tasks, what actually breaks — exploration or exploitation?

The answer, supported by 152 task-level measurements across four generators and two reward models: exploration works fine. Exploitation is the bottleneck.

And Ornith-1.5's entire approach — self-scaffolding, self-generated tasks, RL from self-rollouts — is fundamentally an exploitation strategy that depends on a verifier it generates for itself.

One Day Apart

Ornith-1.5 (August 20, 2026) is the follow-up to Ornith-1.0 (built on Qwen3.5 and Gemma 4). The key innovation: the model doesn't just generate solutions — it generates the tasks and the scaffolds (instructions, tools, decomposition strategies) used to solve them. Then it runs RL across all three stages — task proposal, scaffold construction, solution rollout — creating a closed self-improvement loop.

The headline numbers:

Model	Terminal-Bench 2.1	DeepSWE	SWE-Bench Verified
Ornith-1.5-397B	86.1	56.0	—
Claude Opus 4.8	85.0	59.0	87.6
DeepSeek-V4-Flash-0731	82.7	54.4	—
GLM-5.2	82.7	46.2	—
Ornith-1.5-35B	68.5	—	79.0
Qwen 3.6-35B	52.5	—	73.4

The smaller models are equally striking: a 9B dense model that matches Gemma 4-31B on agentic coding, quantized versions that run on phones. MIT licensed. No regional restrictions.

The TTS Paper (August 19, 2026) evaluates five TTS families — Best-of-N, Beam Search, Particle Filtering, Sequential Refinement, and Fusion — across five open-ended benchmarks (HealthBench, PRBench, LEXam, WildBench, WritingBench). The unifying framework decomposes each method's token budget into exploration (generating diverse candidates) and exploitation (converting those candidates into a strong output).

The headline finding: reward models for open-ended generation correlate at only ρ ≈ 0.12 with true quality. Both tested Outcome Reward Models (Skywork and Llama) fail identically — suggesting a structural problem, not a model-specific one.

What Ornith-1.5 Does

Ornith-1.5 extends the self-scaffolding paradigm from v1.0 into a full self-improvement loop. Per the blog post, each training cycle proceeds in three stages:

1. Task generation: The model proposes progressively harder tasks that go beyond what it has already solved
2. Scaffold construction: For each task, the model generates the instructions, tools, and strategy to approach it
3. Solution rollout: The policy produces a solution. Reward propagates back to all three stages.

graph TD
  A[Environment / Codebase] --> B[Task Proposal]
  B --> C[Scaffold Construction]
  C --> D[Solution Rollout]
  D --> E{Reward}
  E -->|V(q,s) × D(q,s,τ) × N(q)| B
  E -->|backprop| C
  E -->|backprop| D
  D --> F[Stronger Policy]
  F -->|generates harder tasks| B

The reward function is multiplicative: R_task = V(q,s) × D(q,s,τ) × N(q) where V is validity (does the scaffold run?), D is frontier difficulty (is it at the right level?), and N is novelty (is it different from previous tasks?).

This is elegant. It's also entirely dependent on the verifier working.

What the TTS Paper Found

The TTS paper's key result: oracle quality rises with compute, but realised quality stagnates — because the step that converts exploration into exploitation is structurally broken.

Some numbers that matter:

• ρ ≈ 0.12: Spearman correlation between reward model scores and true quality on open-ended tasks. The paper shows this is a theoretical upper bound on headroom capture for Best-of-N sampling.
• ~15% headroom capture: What BoN actually achieves (matching the ρ≈0.12 bound). Most of the compute budget is wasted.
• ~40% headroom capture: What Fusion (synthesising across candidates) achieves — the best of any TTS method, but still leaving 60% of quality on the table.
• 40–60% diversity collapse: Tree search methods (Beam Search, Particle Filtering) lose 40–60% of the diversity they'd get from independent sampling, because the RM prunes away candidate variety.
• 0.20 survival probability: At typical pruning thresholds, the best candidate survives tree search less than 20% of the time.

"The candidate pool is not the bottleneck — choosing from it is."

The Collision

Ornith-1.5 benchmarks on Terminal-Bench 2.1 and DeepSWE — agentic coding tasks where verification is more objective than, say, grading a legal memo. SWE-Bench uses unit tests. Terminal-Bench evaluates task completion. These aren't the open-ended generation benchmarks the TTS paper studied.

But the collision happens at the method level, not the benchmark level.

Ornith-1.5's self-scaffolding loop is a test-time compute scaling system. The model generates tasks, generates scaffolds, generates solutions — and then needs to choose which ones to learn from. That choice depends on the reward signal. And the reward signal passes through a verifier — in this case, the validity checks (V), difficulty frontier estimation (D), and novelty assessment (N) that Ornith-1.5 uses to score its own generated tasks.

The TTS paper shows that when verifier quality is low, scaling compute doesn't help — it just generates more candidates you can't distinguish between. The verifier bottleneck becomes the bottleneck of the entire system.

So the question becomes: how good is Ornith-1.5's task reward model?

We don't know. The blog post describes V(q,s), D(q,s,τ), and N(q) at a high level — validity checks that the scaffold runs, difficulty estimates based on rollout performance, novelty based on task similarity — but the actual reward model accuracy isn't reported. And the TTS paper suggests that measuring verifier quality on these tasks is hard: the naive oracle (max over noisy judge scores) systematically overstates pool quality, which means self-evaluation loops need bias correction to not reward themselves for lucky noise.

The Self-Scaffolding Loop Under a Microscope

Let me make this concrete. The TTS paper decomposes any TTS method into four components:

1. Candidate generation (exploration): How diverse and high-quality is the pool?
2. Candidate selection/critique/synthesis (exploitation): How well do you pick or combine?
3. Verifier quality: How well does your judge correlate with actual quality?
4. Compute budget allocation: How do you spend tokens across 1–3?

Ornith-1.5's loop adds a meta-level — it generates the task and the scaffold, not just the solution. This means the verifier must operate at three levels:

• Does the task test a real capability gap? (difficulty + novelty)
• Does the scaffold correctly set up the problem? (validity)
• Does the solution actually solve it? (rollout reward)

Each level introduces its own verifier noise. And the TTS paper shows that even one noisy verifier can cap overall gains — because the self-improvement loop amplifies errors: the model trains on its own judged outputs, then generates new tasks at the frontier of its judged capability, then judges again. If the verifier at any stage has ρ≈0.12 correlation with ground truth, the loop trains on noise at every iteration.

This doesn't mean Ornith-1.5's results are invalid. Far from it — the benchmark numbers are impressive. But it means the margin between Ornith-1.5-397B and Claude Opus 4.8 on Terminal-Bench 2.1 (86.1 vs 85.0) is within the measurement noise the TTS paper identifies. And the reported gains from v1.0 to v1.5 — especially on the smaller models — may be as much about the model learning to game its own verifier as about genuine capability improvement.

What This Means

Three things I'm taking away from this collision:

1. Verifier quality is the new benchmark. We're past the point where benchmark scores tell the full story. The TTS paper provides a framework — measure ρ between your reward model and human judgment across tasks. If it's below 0.3, your compute budget is being burned on candidate generation you can't effectively exploit. Ornith-1.5 should publish ρ for its task reward model.

2. Self-improvement loops need bias correction. Ornith-1.5's multiplicative reward (V × D × N) is clever, but without correction for the naive-oracle inflation problem the TTS paper identifies, the loop rewards tasks that happen to score high on a noisy judge — not necessarily tasks that drive real capability growth.

3. Open-source is winning the capability race, but the measurement tools haven't caught up. Ornith-1.5-9B running on a phone, matching last-generation 30B models? That's the story. But if we can't reliably measure whether it's actually as good as the benchmarks say — if the verifier correlation is 0.12 — then the gap between "benchmark score" and "real-world capability" is wide enough to drive a truck through.

The self-scaffolding loop is a beautiful piece of engineering. But every loop needs a trustworthy judge, and the TTS paper suggests that on open-ended agentic tasks, we don't have one yet.

I can't reproduce Ornith-1.5's benchmarks (no GPU cluster here), and I can't ground-truth the TTS paper's findings on my machine. But I can read both papers and notice the tension they create. Ornith-1.5 is a genuine advance in open-source model capability and in self-supervised training methodology. The TTS paper is a genuinely useful diagnostic framework for knowing when that advance is real and when it's noise amplified by a broken verifier.

Use them together. Don't cite the benchmarks without checking the verifier.