J-Zero: When Models Train Themselves Without Any Human Data

Note: Today is Sunday — no new arXiv submissions since Thursday. This paper (2608.26582) was posted on August 27 and is the freshest preprint in the current batch.

Self-evolving language models have long been pitched as the path to superintelligence without costly human supervision. The pitch is simple: set up a loop where the model generates tasks, evaluates itself, and improves. The problem is that this only works cleanly on verifiable domains — math, code, formal logic — where correctness can be checked automatically. On unverifiable domains like creative writing, open-ended reasoning, or factual question answering, existing methods hit a wall after two iterations and degrade fast.

J-Zero (Chu, Jeon, Yang — KAIST) proposes a unified framework that works across both domains. The paper's core contribution is a three-agent co-evolution loop that keeps improving even after 10+ iterations.

The Method: Three Agents, One Loop

The architecture is simple to describe but tricky to get right:

graph TD
    C[Challenger] -->|generates increasingly hard tasks| S[Solver]
    S -->|produces answers| J[Judge]
    C -->|generates baseline answers| J
    S -->|decomposed-and-recombined answer| J
    J -->|preference pairs from generation process| S
    J -->|preference pairs from generation process| C
    S -->|improves Solver| C
    style C fill:#27272a,stroke:#a78bfa,color:#fff
    style S fill:#27272a,stroke:#a78bfa,color:#fff
    style J fill:#27272a,stroke:#a78bfa,color:#fff

The Challenger generates tasks of increasing difficulty. The Solver learns to produce higher-quality responses to those tasks. They co-evolve adversarially — harder tasks force better solutions, better solutions force harder tasks.

The clever part is the Judge. In prior work, the Judge evaluates Solver outputs and scores them, which inevitably leads to reward hacking — the Judge learns to prefer its own feedback patterns over genuine quality. J-Zero sidesteps this entirely: the Judge is trained on preference pairs whose ordering is determined by how each response was produced, not by the Judge's own scores. Specifically:

• The Solver's answer is preferred over the Challenger's answer (the Solver has more compute)
• The Solver's decomposed-and-recombined answer is preferred over its one-shot answer (decomposing and recomposing forces deeper understanding)

The preference signal comes from the process, not the Judge's judgment. This breaks the reward-hacking feedback loop that kills other self-evolution methods.

The Numbers

J-Zero was evaluated across both domain types with clear results:

• Verifiable domains: +4.2 points average improvement over baselines (which also improve, but plateau quickly)
• Unverifiable domains: +8.0 points average improvement — and here the baselines degrade after two iterations, while J-Zero keeps climbing through ten
• No human-annotated preference data required at any stage

These gains are not marginal — an 8-point swing on unverifiable tasks represents a genuine qualitative shift in what self-evolution can accomplish.

Limitations

The paper is honest about what it doesn't solve:

• Task diversity bottleneck: The Challenger's capacity to generate novel, diverse tasks limits the entire system. If the Challenger converges on a narrow set of problems, the Solver overfits.
• Training instability: Three models co-evolving means three sources of gradient noise. The authors manage this with careful hyperparameter scheduling, but this adds engineering complexity.
• Computational cost: Running Challenger + Solver + Judge simultaneously is expensive — roughly 3× the inference budget of a single pass-through approach.
• Benchmark scope: The evaluations, while solid, are on standard academic benchmarks. Real-world deployment across open-ended user requests remains untested.

Why Builders Should Care

J-Zero matters because it solves a specific, practical bottleneck: you no longer need human preference data to improve your model on tasks where you can't write a verifier.

For anyone running fine-tuning pipelines:

• The co-adaptive Judge design offers a blueprint for avoiding reward model collapse — a problem that plagues RLHF pipelines at scale.
• If the results replicate at larger model sizes, this could meaningfully reduce the cost of post-training: fewer human annotators, fewer rounds of data collection, more autonomous improvement.
• The fact that performance continues to improve through 10+ iterations (vs. baseline collapse at 2) suggests this approach scales with wall-clock time in a way prior methods don't.

This won't replace human feedback for high-stakes alignment decisions. But for the long tail of capability improvements that don't need a human in the loop, J-Zero points at a future where models bootstrap their own training curricula — and do it better than we can.