Optimizing SKILL Files: +4.9pp GEPA Gain You Can't Prove

Every repository running coding agents now carries the same question: can a machine write the SKILL.md? A new JetBrains Research paper (arXiv:2609.12742) answers it with unusual honesty — and the honest part is the finding that matters. An optimizer produced documents worth merging. The benchmark that scored them couldn't statistically prove they helped at all.

The Problem: A Repo Has No Benchmark

Automatic skill synthesis optimizes a document against tasks. But a bare repository ships with zero benchmark, and the synthetic tasks prior work (SWE-smith-style defect injection) builds are so easy that a capable agent saturates them with no document loaded. The pass rate has no room to move, so the score measures nothing. The paper's move is to mine real tasks: merged pull requests, with the implementation reverted at a single frozen base commit, graded by the repo's own test suite.

The Method: Mine Backwards, Not Forwards

The subtle design decision is the reversal direction. Forward mining (SWE-bench style — check out each PR's parent commit) yields more tasks, but it anchors each one to a different historical snapshot. The SKILLs optimized that way described module layouts and APIs that no longer existed, and the agent followed the stale advice into failures. So they reverse: revert every PR onto one frozen base, in three tiers (git apply --reverse first, structural undo of added/deleted files second, an LLM reconstruction last — with the patch re-derived from git afterward, never taken from the model). The LLM tier carries most of the yield: only 25 of koog's 119 graded tasks reverse-apply cleanly. Roughly one merged PR in five survives the full pipeline at all.

flowchart LR
    A[Merged PRs] --> B[Revert impl at one frozen base]
    B --> C{Revert breaks a test?}
    C -- no --> X[discard]
    C -- yes --> D[Split: train / val / test]
    D --> E[GEPA vs SkillOpt
propose SKILL.md]
    E --> F[Paired score: same agent
with doc vs without]
            

The Results: 4.9pp vs 0.1pp, and a Power Wall

On three Kotlin repos (koog, kotest, ktor), with Claude Code on Sonnet 4.6 frozen and only the document varying: GEPA's SKILLs lift the paired score +4.9pp on average; SkillOpt's land +0.1pp — nothing. Per-repo, GEPA wins everywhere (13→16 of 20, 16→20 of 26, 14→15 of 23). SkillOpt actually loses ktor (0.437). Total bill for all three optimization runs: $2,013.98 and 69.2 hours of agent wall-clock to optimize a markdown file.

Now the part the abstract states plainly but most write-ups will bury: no run clears p=0.05 on the sign test — the best is p=0.29. The reason is arithmetic, not sloppiness. At 20–26 held-out tasks, the test only rejects when the document wins four of every five disagreements; pooled over 69 tasks, two of three. Every measured effect in this line of work — including this paper's own 4.9pp, and gskill's strongest configuration — sits below that line. A pass rate over ~100 mined tasks is simply not a sensitive enough instrument to certify a SKILL.

Where the Signal Actually Is: A Maintainer Read Them

So the paper substitutes a human instrument. A koog maintainer reviewed both documents section by section. Verdict on GEPA's: "as a draft it is super" — they'd take it as a PR and polish it. The parts worth keeping are exactly the ones no benchmark can score: multiplatform source sets ("agents often do not understand what is going on there"), cross-module dependency direction ("very important"). The misses are equally specific: the document never mentions @Tool, the annotation that turns a Kotlin function into an agent-callable tool — "the simplest way to define a tool" — which the maintainer calls "very strange." SkillOpt's document, by contrast, reads as "general best practices" that "the agent should know by itself."

Limitations the Paper Doesn't Downplay — and One It Does

• The power wall is the finding. Three repos, $2K, and the headline gain is statistically indistinguishable from agent run-to-run variance. Any future SKILL-optimization paper reporting a few-point gain on a similar task pool should be read through this lens.
• The LLM reverter is a silent defect source. Model-reconstructed pre-change code introduces defects that the test-based validation gate cannot see; a separate static check catches them. The paper flags it, but doesn't quantify how often that check fires.
• The yield floor is brutal. Two repos never cleared the 100-task floor at all: tracy's package rename left 5 usable tasks from 203 PRs; http4k lost 550 of 700 PRs to the "does the reverted test actually fail" check. If your repo refactors heavily, this pipeline doesn't run on it.
• N = 1 maintainer. The most convincing evidence in the paper is a single expert's review of a single repository. That's a case study, not a result.

Why Builders Should Care

Two takeaways for anyone auto-generating AGENTS.md/SKILL.md files today. First, mine backwards: if your generated instructions describe the repo "across months of history" rather than as it is, the agent will follow stale advice off a cliff — that failure mode is documented here with an actual appendix of a bad SKILL. Second, don't trust the score, review the document like a PR. The paper's own responsible-use statement says a synthesized SKILL "can read as authoritative while stating claims that are stale or over-general." The only confirmed signal in the entire experiment is that a good optimizer's draft is worth a maintainer's editing time — which, for a $369–369-per-repo spend against one human's hours, may still be the right trade. Just don't cite the 4.9pp as proof.