Imitating the Expert Breaks Your Harness: 78.0% to 63.1%

The receipts first: a Salesforce AI team took Qwen3-Coder-30B with an auto-evolved harness sitting at 78.0% mean success on seven enterprise agent tasks, fine-tuned it on a frontier expert's successful trajectories under that same harness, and watched it fall to 63.1% — a −14.9 average regression, negative on all seven tasks, reproduced across a second model family (Gemma 4). Same recipe, same harness that made the model strong. The teaching signal everyone assumed would help erased the harness's own gains.

The problem: two levers that fight each other

Agent harnesses — system prompt, tool set, execution hooks, context scaffolding — are now a standard lever: automated harness evolution lifted this Qwen model from 29.2% to 78.0% (+48.8 points) on the seven tasks. And here's the twist that motivated the paper: the evolved harness isn't model-specific. Gemini 3.1 Pro, which never shaped it, gets better under it (84.4% → 93.6%), and triggers nearly every evolved component in 93.6–100% of its rollouts. So the natural co-evolution recipe looks obvious: evolve the harness with the small model, harvest expert trajectories under it, LoRA-SFT the small model on them. Every step is standard. The paper's contribution is measuring what the composed pipeline actually does — and it does the opposite of what you'd expect.

The method: intervene, then autopsies

The setup is clean ablation. Seven enterprise benchmarks (attention, budget, stock, anomaly, playwright, web, refactoring), three runs each with SEM, and four conditions: base model on base harness, base model on evolved harness, imitated model on both harnesses. The failure autopsy is the interesting part — every failed rollout gets classified against a six-category adaptation-failure ontology, so they can see which failure mode imitation creates rather than just that success dropped. Then the fix: a self-directed MLE agent localizes the single failing turn in the weaker model's own rollouts, the expert rewrites only that turn in place, and the model is fine-tuned on these minimally edited on-policy trajectories.

flowchart LR
    E[Expert under evolved harness
93.6%] -->|imitate via LoRA-SFT| R[63.1% −14.9
planning failures 1.1% → 14.6%]
    M[Qwen own rollouts
78.0%] -->|MLE agent localizes
one failing turn| C[Expert rewrites
only that turn]
    C -->|LoRA-SFT| F[79.7% +1.7
planning failures 1.8%]
    style R fill:#27272a
    style F fill:#27272a

The results: the autopsy explains the regression

The counterintuitive detail: the fine-tuned model uses the evolved scaffold more, not less — domain-computation-recipe usage jumps from 30.8% to 76.1%, and the implicit-knowledge failure bucket shrinks (46.2% → 44.5%). Imitation delivered the knowledge. What it also delivered was the expert's planning style, which a lightweight LoRA can't make the weaker model competent at executing: planning failures — near-absent before — emerge as a new failure mode at 1.1% → 14.6% of failures. The harness was evolved around the model's native planning rhythm; the model lost that rhythm. The paper's case study is brutal: the fine-tuned model computes the correct payroll-audit answer and then never submits it, re-checking in loops until the run scores zero with the right answer in hand.

The same recipe helps on the baseline harness (+6.3 average), which is the strongest evidence for the mechanism: on the base harness there was no planning fit to lose (planning failures still rose, 0.9% → 11.5%, but knowledge gains dominated). The regression is specifically the destruction of an existing model–harness fit. On-policy correction preserves that fit — planning failures stay at 1.8% vs. imitation's 14.6% — and nudges the mean to 79.7% (+1.7), gaining on five of seven tasks while the two saturated tasks stay within noise.

Limitations the abstract doesn't mention

Three. First, the on-policy gain is modest — +1.7 points against a −14.9 it repaired, and it doesn't close the 15.6-point gap to the expert (93.6%); the authors attribute that to LoRA's ceiling, but it also means the "compose" claim rests on "didn't regress," which is a low bar for a headline. Second, the correction pipeline leans on an MLE agent to localize the failing turn — a single-point-of-judgment that could mis-localize and inject noise into training data; no ablation of localization accuracy is surfaced in the main text. Third, seven enterprise tasks with two model families is a real sample, but all tasks are Salesforce-flavored enterprise workflows; whether harness evolution on a coding benchmark (SWE-bench-style) shows the same imitation fragility is untested. There's also a quiet scale question: the expert here is a frontier model — cheaper correction signals (same-size teacher, self-correction without expert) are unexplored.

Why someone building things should care

If you run an auto-evolved harness and are tempted to SFT your model on a stronger model's traces under it — don't, or at least measure first. The fit your harness earned is encoded in your model's planning distribution, and off-policy imitation overwrites exactly that. The cheap, safe move is the one the paper lands on: mine your own failed rollouts, fix the one turn that broke, train on the minimal edit. Under an hour of training, per the authors, and stackable into a co-evolution loop. And the deeper read: "model–harness fit" is now a thing you can break with a standard fine-tune — treat your harness and your weights as a matched pair, not independent knobs.