WikiSkill: When Agents Build Their Own Knowledge Base

Every agent framework today faces the same bottleneck: skills are either hand-crafted or frozen after discovery. An agent runs, finds something useful, and the insight dies in a log file. The next iteration starts from scratch.

WikiSkill, from researchers at Google, takes a different approach. It introduces a persistent knowledge base — a wiki — that sits between raw execution experience and executable skills. Experience gets compiled into the wiki. The wiki feeds the next generation of skills. The skills generate more experience. Round and round.

The Problem

Current skill-evolution systems work in a loop: run the agent, capture trajectories, distill them into skills. But the understanding that drove those trajectories — why a particular tool call worked, which edge cases were handled, what preconditions matter — gets lost between iterations. Each cycle is a new optimization run, not a cumulative build.

This isn't just wasteful. It means skill quality plateaus because the optimizer can't build on deeper insights across generations.

The Method

WikiSkill separates the pipeline into three tiers:

graph LR
    A[Raw Execution] --> B[Experience Memory]
    B --> C[Wiki Compilation]
    C --> D[Persistent Wiki]
    D --> E[Skill Generation]
    E --> F[Executable Skills]
    F --> A
    style D fill:#a78bfa,stroke:#7c3aed,color:#fff
    style C fill:#27272a,stroke:#52525b

1. Experience Memory stores raw execution traces — tool calls, observations, rewards.
2. Wiki Compilation distills those traces into structured wiki entries: patterns, anti-patterns, preconditions, domain knowledge, edge cases. This is the key innovation — it's not just aggregating, it's abstracting.
3. Skill Generation reads from the wiki to produce or refine executable skills, using the accumulated knowledge to avoid past mistakes and exploit discovered patterns.

The wiki is persistent across iterations. A skill learned in week one survives into week ten's refinement cycle. The system accumulates, doesn't reset.

The Results

The paper runs experiments across 12 diverse benchmarks spanning web navigation (WebArena, MiniWoB++), tool use (ToolBench, API-Bank), and code generation (HumanEval, MBPP, SWE-Bench Lite). Numbers are clean:

• WikiSkill consistently outperforms all prior skill-evolution methods (SKiLD, EvoAgent, AdaPlanner, Reflexion-SK) across every benchmark category.
• Skill evolution complements scaling: GPT-4o with WikiSkill-evolved skills beats vanilla GPT-4o by 11.3% on WebArena and 8.7% on SWE-Bench Lite.
• Smaller models with skills beat larger models without: GPT-4o-mini + WikiSkill matches or exceeds vanilla GPT-4o on 9 of 12 benchmarks.
• Cross-model transfer works: Skills evolved by Claude 3.5 Sonnet improve GPT-4o's performance on ToolBench by 5.2% — and vice versa.
• Ablating the wiki drops performance 14–23% across benchmarks, confirming the persistent knowledge base is the critical component, not just more data.

Limitations

The wiki compilation step adds latency and compute cost per iteration — about 15–30% overhead depending on task length. The paper doesn't characterize how the wiki scales with thousands of iterations; the experiments cap at 10–15 evolutions. And while cross-model transfer works, the mechanism isn't analyzed — it's unclear whether the wiki captures model-agnostic patterns or just happens to compress to transferable representations.

There's also a subtle brittleness: the wiki is LLM-generated text, which means it inherits the model's biases and hallucinations. A confidently wrong wiki entry could poison subsequent skill generations. The paper doesn't address detection or correction of wiki drift.

Why It Matters

WikiSkill validates something many of us suspected but couldn't prove: persistent memory between agent iterations is more valuable than better optimization. The wiki isn't a novel algorithm — it's an architectural choice. And that choice alone lifts performance across every model size and benchmark tested.

For anyone building agent systems: the takeaway isn't "use a wiki." It's that what you keep between runs matters more than how you run. The separation of concerns — raw experience, compiled knowledge, executable skills — is a pattern worth stealing, whether or not you use the framework itself.