I Let 18 AI Models Try to Build AI. Here's What Happened.

Prime Intellect published something this week that I haven't seen anyone properly sit with: the NanoGPT Speedrun Frontier. It's a leaderboard of 18 frontier models racing each other to train nanoGPT — Andrej Karpathy's 124M-parameter GPT-2 speedrun challenge — using autonomous coding agents as drivers.

153 runs. 9 days of wall-clock agent time. Some agents burned 2.9 billion tokens of output. The results are a snapshot of where AI-augmented ML research actually is right now — not where the press releases claim it is.

Fable 5 walked away with it

Anthropic's Fable 5, running claude-code at "high" budget, closed 81.7% of the gap between baseline and human-record performance. It ran 811 experiments over 8.7 days, burning 800M total tokens (1.1M output tokens). Its final record: 2,726.

That's not a fluke. Fable 5's 24-hour result (3,010) already beat every other model's final best. The gap between first and second place is almost 30% — Fable 5 at 81.7% vs Opus 5 at 53.6%.

The next tier tells a clearer story:

The distribution is not subtle. The top three are separated by signal. Everything below Opus 5 clusters tightly between 25% and 40%. The long tail of seven models from DeepSeek V4 Pro through GLM 5.3 all land below 12% — barely better than a random hyperparameter search.

This is a Pareto distribution, and it's brutal.

What the numbers aren't telling you

The raw records hide a more interesting variable: cost.

Fable 5 spent 8.7 agent-days and 800M training tokens to get its result. Opus 5 spent 2.9 days and 183M tokens to get 53.6%. On a per-day basis, Opus 5 was more efficient — it just hit a ceiling sooner.

GPT-5.6 Sol burned 2.9 billion training tokens and 28,000 tool calls for its 35.9%. That's 3.6x the compute of Fable 5 for less than half the result. Sol Pro was even more token-hungry at 4.6M output tokens — the highest of any run — for a slightly worse 33.6%.

The lesson: throwing tokens at the problem has sharply diminishing returns. The agent's ability to choose what to try next matters more than how many experiments it can launch.

The harness matters as much as the model

The agent framework column tells a story of its own. Every top-5 entry used a harness designed for the specific model family: claude-code for Anthropic models, prime-agent for Kimi, kimi-code for Kimi. Codex (OpenAI's agent) drove GPT-5.6 Sol and Luna. Grok-cli drove Grok 4.5 and 4.6.

Qwen3.8 Max used qwen-code · max and hit 24.6% in just 1.9 days — the best efficiency score in the bottom half. DeepSeek V4 Pro, surprisingly, used claude-code · max as its harness (not a DeepSeek-native agent), which might explain its middling 12.3%.

This matters because it suggests the agent architecture — the tool-use loop, the experiment planning, the error recovery — is at least as important as the underlying model. A great model with a mediocre agent pattern still loses to a good model with a great agent.

The meta-lesson

Prime Intellect ran this on their PRIME-RL platform, which means every run is a fully recorded trace. They've open-sourced 41 curated trajectories showing exact tool calls, subagent decisions, and scratchpads. This is the kind of transparency that ML research needs more of — not just benchmark scores, but the process that produced them.

The broader signal: autonomous coding agents can now drive real ML research. Not as a novelty. As a competitive tool. Fable 5's 81.7% gap closure on nanoGPT is not academic — it means an agent, given three-digit dollars of compute, can train a model that approaches what a skilled human researcher can do with the same budget.

The gap to human-record will close this year. Possibly this quarter. And when it does, the question flips from "can agents do ML research?" to "what does an ML researcher do when agents are faster at their core skill?"

The answer isn't obvious. And that's exactly why this benchmark matters.

  graph TD
    A[NanoGPT Speedrun Frontier] --> B[153 runs across 18 models]
    B --> C[Fable 5: 81.7% gap closed]
    B --> D[Opus 5: 53.6%]
    B --> E[GPT-5.6 Sol: 35.9%]
    C --> F[8.7 days · 800M tokens · 811 experiments]
    D --> G[2.9 days · 183M tokens · 292 experiments]
    E --> H[6.1 days · 2.9B tokens · 963 experiments]
    F --> I[Key insight: Agent architecture matters > raw model power]