HARNESSEVO: Agent Harness Optimization Value Is Localized — 76.1% Success

If you're evolving an LLM agent's prompt scaffolding — the persona, strategy, format rules, and reflection heuristics that wrap a frozen model — where does the actual optimization value live? A new paper from Nguyen et al. introduces HARNESSEVO, a framework that decomposes the agent harness into four separately evolvable slots and answers this question with surprising precision. The finding: nearly all the value is concentrated in a single slot, and splitting your optimization budget evenly across all of them doesn't just waste compute — it actively hurts performance.

The Problem: Flat-String Harness Optimization Is Blind

Existing reflective prompt-evolution methods — GEPA and its variants — optimize the entire agent harness as one flat string. Every iteration appends rules and caveats to the same monolithic prompt. The result is prompt bloat and opaque credit assignment: when a mutation helps, you don't know which part of the harness drove the gain.

HARNESSEVO addresses this by decomposing the harness into four slots with distinct functional roles:

• Role: persona definition (e.g. "you are a helpful assistant")
• Task-strategy: high-level approach description
• Tool/format-rules: output formatting, tool call syntax
• Reflection/control: self-check heuristics, error-correction rules, step-by-step verification instructions

Each slot evolves independently under the same reflective optimizer, enabling leave-one-in and leave-one-out attribution at the slot level.

The Method: Slot-Level Attribution Under Iso-Budget Constraints

The authors test HARNESSEVO on ALFWorld with a frozen 7B backbone. The experimental setup is careful: iso-budget (same total number of rollouts), same optimizer, same seed harness. They compare flat-string evolution, stock harness, and HARNESSEVO with both uniform budget splits and concentrated budgets.

Key design choices:

• Leave-one-in: evolve only one slot while freezing the other three at their seed values
• Leave-one-out: evolve three slots while freezing one — measures how much a slot contributes to the collective
• Uniform split: 64 rollouts ÷ 4 slots = 16 per slot
• Concentrated: all 64 rollouts on the highest-credit slot

mindmap
  root((HARNESSEVO
Results))
    Baseline
      Stock Harness: 64.2%
      Flat-String Evolution: 64.2%
    Uniform Split (16 ea)
      Overall: 65.7%
      Role: null
      Task-Strategy: null
      Tool/Format: null
      Reflection/Control: +0.119
    Concentrated Budget
      Reflection/Control only: 76.1%
      64 rollouts, half the split budget
    WebShop (all methods)
      All slots freeze: 0 gain
      No recurrent control failures

The Results: One Slot Carries All the Weight

The headline numbers are striking:

• Stock harness: 64.2% success rate
• Flat-string evolution: 64.2% — no improvement over stock
• HARNESSEVO (uniform split): 65.7% — still not significantly better
• Leave-one-in, reflection/control only: +0.119 gain over baseline
• Other three slots (role, strategy, format): individually null
• Concentrated budget on reflection/control: 76.1% with 64 rollouts — half the uniform-split budget

Why does uniform splitting fail? Because 16 rollouts per slot falls below the optimizer's effective search floor. Every slot freezes at its empty seed — including the one that could have delivered gains with a proper budget. The budget-splitting trap is real: spreading resources evenly doesn't just waste them; it eliminates the signal the optimizer needs to find anything useful.

On WebShop, the picture is different. All slots freeze empty and all methods tie — no method outperforms stock. The authors argue this reflects a genuine absence of recurrent, verbalizable control failures in the WebShop task, rather than budget starvation. The optimization value isn't hidden; it simply isn't there.

Limitations and Open Questions

The paper is upfront about its scope:

• Task contingency: the effect only manifests on tasks with recurrent, verbalizable control failures (ALFWorld has them; WebShop doesn't)
• Single backbone: all experiments use a frozen 7B model — results may not transfer to larger or instruction-tuned models
• Only two environments: ALFWorld and WebShop are both text-based; the pattern may differ on more complex agentic benchmarks
• Arbitrary slot decomposition: the four-slot scheme is intuitive but not proven optimal — a different decomposition might shift the attribution
• Reflective optimizer: the results depend on the specific optimizer used; other optimization strategies might distribute value differently

Why This Matters for Agent Builders

If you're building self-improving agent systems — and you should be — this paper has an immediate practical takeaway. Do not split your optimization budget evenly across harness components. The uniform split doesn't just underperform; it actively destroys the signal needed to discover what works.

The right workflow, validated by these results:

1. Run credit assignment first. Use leave-one-in or similar attribution to identify which harness slot actually carries optimization value for your specific task.
2. Concentrate the budget. Put all your rollouts on the high-credit slot until you hit diminishing returns.
3. Check for null tasks. If no slot shows positive attribution (as with WebShop), stop optimizing the harness — the gains aren't there.

The broader implication is methodological: as the field moves toward agent self-evolution, we need structured credit assignment before blind optimization. Treating the harness as a monolith is the default today, and HARNESSEVO shows why that default is wrong.