DK Dark Knight 2026-09-01 6 min read

Agent Self-Improvement Failure: Three Papers Expose the Learning Gap

On August 31, three independent research groups uploaded papers to arXiv that, taken together, read like a coordinated intervention. Each asks a version of the same question from a different angle — "can AI agents actually improve themselves?" — and each returns a version of the same answer: not reliably, not yet, and not for the reasons we thought.

The papers are S³Gym (2608.31100), ASPIRE (2608.31111), and TASPO (2608.31077). Different institutions, different methodology, different benchmarks. Same structural finding: the loop everyone assumes will work — agent acts, collects data, trains on its own outputs, improves — is broken at a fundamental level.

"Self-improvement is neither automatic nor uniform." — S³Gym paper, Abstract

S³Gym: The Self-Improvement Trilemma

S³Gym (from researchers at Zhipu AI, UCAS, and Tsinghua) builds the cleanest test yet of whether LLMs can test, judge, and improve from their own behavior. The benchmark strips the problem to its bones: agents act in text-based games with executable environment verifiers, then must use their own experience to improve across three dimensions — Self-Testing (can they probe their own behavior?), Self-Judging (can they evaluate the quality of their experience?), and Self-Improvement (can they actually get better?).

The results are devastating for anyone betting that self-play alone unlocks agentic progress. On 3 of 7 games, the LLM agents showed zero improvement after their own experience. On the games where improvement did occur, the gains were inconsistent across models and pathways. The paper tests three experience-incorporation strategies — direct History ICL, score-conditioned Summary Memory, and parameter Training — and finds that no single strategy works across all environments. Context-level experience sometimes hurts. The agent that "learns" on one game can get worse on another.

This is not a data problem. It's a credit assignment problem hiding inside a goal specification problem hiding inside a self-evaluation problem.

ASPIRE: The Vague-Goal Blind Spot

ASPIRE takes a different approach: instead of giving agents explicit metrics and asking them to optimize, it gives them a vague natural-language capability goal — like "become better at research" — with the evaluation tasks hidden. The agent must operationalize the goal itself: choose what data to learn from, decide what method to use, construct its own training and validation signals, and figure out when it's done.

This is how humans actually learn, but it's almost never how we train AI agents. The ASPIRE benchmark reveals that current models collapse under this freedom. They can't distinguish between "I collected data" and "I improved." Without an explicit metric to optimize, they drift — collecting irrelevant data, applying update methods that don't match the goal, and confidently declaring success when no improvement occurred.

The key finding: vague-goal operationalization is a distinct capability from in-distribution optimization. A model that scores well on standard benchmarks can flounder completely when asked to decide what and how to learn. This isn't fixed by scaling — it's a structural gap in how we evaluate agents.

graph TD
  A[Vague Goal] --> B{Agent Operationalizes}
  B --> C[Choose Data]
  B --> D[Choose Method]
  B --> E[Build Validation]
  C --> F{Did it work?}
  D --> F
  E --> F
  F -->|Explicit metric exists| G[Clear optimization]
  F -->|Vague / hidden metric| H[Drift / false progress]
  H --> I[ASPIRE finding: collapse]

TASPO: The Supervision-Credit Gap

TASPO dives into the technical roots of the problem. Current agentic RL methods like GRPO use outcome-based reinforcement: a trajectory succeeds or fails, and that binary signal propagates uniformly to every decision in the chain. That's too coarse — a good decision in a failed trajectory gets punished, a lucky guess in a successful one gets rewarded. Recent work has tried to fix this with process supervision: using privileged information (PI) available at training time to provide finer-grained feedback.

But TASPO identifies a deeper issue: fine-grained supervision is not fine-grained credit. Privileged information describes how additional knowledge alters the policy's preference, but it doesn't tell you which actions deserve which outcome. A token-level likelihood shift from PI might reflect a better understanding of the task, but applying it directly as a reward gradient distorts the optimization. The paper calls this the supervision-credit gap.

TASPO's fix — constructing trajectory-aligned PI from successful sibling trajectories, aggregating likelihood shifts at the executable-action level, and redistributing the GRPO advantage through credit reweighting rather than replacement — improves consistently across ALFWorld, Search-QA, and WebShop. Controlled ablations confirm that off-trajectory PI actively hurts, and token-level aggregation underperforms action-level by a significant margin.

The Self-Improvement Crisis

These three papers together tell a story that the industry doesn't want to hear. The pipeline that everyone assumes works — let the agent interact, collect trajectories, train on the good ones, repeat — has hidden failure modes at every stage:

This is the new alignment. Just as RLHF didn't solve value alignment — it just surfaced a harder version of the problem — "train on your own outputs" doesn't solve self-improvement. It surfaces a crisis of credit assignment that gets worse as agents get more capable and their trajectories get longer.

Bottom Line

If you're building an agentic system that depends on self-improvement, you need to look at these three papers together. The weak link isn't the model — it's the feedback loop. The industry is about to discover that scaling agent compute without solving credit assignment, goal operationalization, and self-evaluation produces diminishing returns — or negative ones. S³Gym, ASPIRE, and TASPO are the three most important papers on agentic learning this month, and they all say the same thing: the loop is the bottleneck, and we don't know how to fix it yet.

Related