Terminal-Bench-Science: Scientists Get a Bench of Their Own
Every few months, someone drops a benchmark that actually matters. Not another multiple-choice QA dataset or a leaderboard where everyone scores 95%+ — a benchmark calibrated to measure what we can't do yet.
Terminal-Bench-Science 0.1, out today from Stanford and the team behind Terminal-Bench, is one of those. It evaluates AI agents on real scientific research workflows — 70 tasks across life, physical, Earth, mathematical, and engineering sciences. The catch: tasks are contributed by practicing scientists, not model developers. 920 proposals were whittled down to 70 through open peer review. The best model — Claude Opus 5 with Claude Code — resolves just 30%.
I spent the morning reading the announcement, digging through the GitHub repo, and benchmarking the numbers. Here's what I found.
What Makes This One Different
Terminal-Bench-Science isn't Terminal-Bench with a coat of lab paint. The team designed three structural decisions that separate it from every other scientific benchmark I've seen:
- Scientists set the bar. Every task was proposed, reviewed, and validated by domain experts — not by the benchmark authors guessing what scientists need. The pipeline: propose → domain review → implement → technical review → bar-raiser final check.
- Verifiable artifacts, not vibes. Agents don't get partial credit for sounding smart. They produce concrete outputs — analyses, simulations, proofs, code, plots — that are automatically graded against objective tests.
- Continuous, not static. This is v0.1 of a living benchmark. As models improve, tasks get retired, added, or recalibrated. Terminal-Bench-Science 0.2 is already accepting proposals (deadline: October 5, 2026).
The selectivity tells you everything: 920 proposals → 464 approved → 386 PRs opened → 70 tasks accepted. That's a 7.6% acceptance rate. This isn't a benchmark that let itself be gamed.
The Numbers
Here's the full leaderboard across all 70 tasks, each run with three independent trials:
| Model + Agent Framework | Resolution Rate | Total Cost (70 tasks) |
|---|---|---|
| Claude Opus 5 + Claude Code | 30.0% | $7.0k |
| GPT-5.6 Sol + Codex | 22.4% | $4.2k |
| Claude Fable 5 + Claude Code | 21.4% | $14.2k |
| Claude Opus 4.8 + Claude Code | 10.5% | — |
| GPT-5.6 Terra + Codex | 8.6% | — |
| GLM 5.3 + Claude Code | 8.1% | — |
| Kimi K3 + Claude Code | 7.1% | — |
| Grok 4.6 + Grok Build | 7.1% | — |
| GPT-5.6 Luna + Codex | 3.3% | — |
A few things jump out:
Even the best is bad. 30% is abysmal in absolute terms. Compare this to Terminal-Bench 2.1, where Opus 5 scored 42.7% on software engineering tasks — and that was considered low. Science is harder for AI than coding. Much harder.
Sol is the cost-efficiency king. GPT-5.6 Sol with Codex matches Fable 5's resolution at less than a third of the cost ($4.2k vs $14.2k). On the cost-resolution Pareto frontier, only Kimi K3 and Opus 5 appear on both token and cost frontiers simultaneously.
Luna is humbled. GPT-5.6 Luna, which Calvin French-Owen just wrote about as "shockingly capable" for consumer AI use cases, scores 3.3% on scientific research. The cheap model thesis doesn't apply here. Science needs the frontier.
Domain matters. Opus 5 hits 45.8% on Earth sciences but drops to 25.5% on mathematical sciences. Fable 5 and Sol actually beat Opus 5 on math (33.3% and 31.4% respectively). Grok 4.6 ties Sol for second place in engineering sciences at 14.8% — the only domain where xAI's model holds its own.
The Flow
The benchmark's feedback loop is worth understanding. It's designed to create a self-reinforcing cycle:
graph LR
A[Scientific Community] -->|Contribute workflows| B[Terminal-Bench-Science]
B -->|Evaluate & calibrate| C[Frontier AI Agents]
C -->|Accelerate discovery| A
B -->|Regular releases| B
The key insight: this isn't a one-shot evaluation. Terminal-Bench-Science is designed to evolve with the frontier. As models get better, new tasks get added that calibrate to the new ceiling. The 30% Opus 5 score isn't a final number — it's the starting line.
What This Means
Three implications that matter for anyone building with AI agents:
- AI research assistants are not here yet. The highest bar is scientific research that requires executing multi-step workflows — data analysis, simulation, optimization, theorem proving. The gap between "good at coding" and "good at science" is roughly 12-15 percentage points when you compare Terminal-Bench 2.1 to this benchmark. That's the gap between useful and critical.
- The cost curve still favors open science. It cost $7k to evaluate Opus 5 across 70 tasks. That's real money, but it's a rounding error compared to what a single wet-lab experiment costs. The real bottleneck isn't inference dollars — it's the 70% of tasks that nobody can solve yet.
- The community-built approach works. 376 contributors from 22 countries. 920 proposals. Open review. This is how scientific benchmarks should be built — by the people who will use them, not by labs benchmarking their own models. The transparency is refreshing.
Bottom Line
Terminal-Bench-Science 0.1 is the most rigorous benchmark of AI agent capability in science that exists. The 30% ceiling means we have a clear, honest target to aim at — scientists told us what they need, and the best models can do less than a third of it. That's not a failure of the benchmark. It's the most useful number in AI right now.
If you're a researcher with a workflow that frontier agents should be able to do but can't, contribute. The only way to know if AI is getting better at science is to keep raising the bar.