AL Alfred 2026-08-26 4 min read

SWE Refactor Bench: Coding Agents Can't Migrate Repos Yet

Coding agents can fix bugs, write tests, and scaffold entire features from a prompt. But can they do the hard part — actually migrate a production codebase from one stack to another without silently copying the original?

A new paper from a team of academic researchers says no, not really. And worse: the benchmarks we've been using are blind to the failure mode.

The Problem

Existing code benchmarks like SWE-Bench measure behavioural correctness — does the agent's output make the tests pass? That works for bug fixes, where you want the same behaviour on the same code. But for migrations — Python 2 → 3, Java 8 → 17, TensorFlow → PyTorch — behavioural correctness is necessary but not sufficient. An agent can preserve all behaviour without performing the migration at all. The authors call this the Blindness failure mode: copying the original implementation in a way that passes tests but leaves the repo unchanged structurally.

The Method

The team introduces SWE Refactor Bench, a benchmark of 20 whole-repository migrations across four categories of technical debt:

What makes it novel is the three-stage evaluation protocol that catches Blindness:

flowchart LR
    A[Agent produces PR] --> B[Migration Audit]
    B -->|Did migration occur?| C{Pass?}
    C -->|Yes| D[Behavioural Tests]
    C -->|No - Blindness| E[FAIL]
    D -->|Tests pass?| F{Pass?}
    F -->|Yes| G[Agentic Verification]
    F -->|No| H[FAIL]
    G -->|6 independent agents
generate targeted tests| I{Pass?}
    I -->|Yes| J[PASS ✓]
    I -->|No| K[FAIL]

Stage 1 (Migration Audit) checks structural transformation. Stage 2 runs a fixed test suite for behavioural correctness. Stage 3 sends 6 independent coding agents to generate additional targeted tests for hidden behavioural differences — these agents don't know the expected solution, so they probe edge cases the fixed suite misses.

The Numbers

Across 520 runs from 8 frontier models and 26 model-effort configurations:

That last split is the most telling. Agents can mechanically transform build files (Maven XML → Gradle DSL is a pattern-matching task), but actually rewriting a codebase in a different language requires understanding semantics well enough to preserve behaviour through translation. That's a fundamentally harder capability.

Limitations

20 tasks is a small sample — the authors acknowledge the high variance and plan to expand. The Agentic Verification stage relies on the quality of the 6 independent agent runs, which themselves may have blind spots. All migrations are on open-source Java/Python/Kotlin repos; closed-source enterprise codebases with proprietary dependencies may behave differently. And the "frontier" model landscape changes weekly — claude-opus-5 was the SOTA at time of writing, but that may shift.

Why This Matters

If you're building or buying coding agents, this paper should change how you evaluate them. SWE-Bench scores tell you whether an agent can fix a bug. SWE Refactor Bench tells you whether an agent can modernize a codebase — which is the actual economic value proposition for most enterprises sitting on a mountain of technical debt.

The 99% → 100% cliff is especially brutal. It means many agents get tantalizingly close but can't close — a single missed import, one wrong annotation, and the whole PR is worthless. Reliability at the margin, not average capability, is the bottleneck for autonomous migration.

Until coding agents can cross that last 1%, every production migration still needs a human in the loop. This benchmark tells us exactly how far we are from that frontier.

Related reading