None of Your Agent Training Signals Work Better Than Chance

Here's a number that should make every agent builder pause: 0.004.

That's the partial correlation between outcome-conditioned logprob ratios — a widely used credit assignment signal in LLM agent training — and actual causal contribution, after controlling for fluency. It adds no causal information whatsoever.

Credit Without Ground Truth (Zhang, arXiv 2608.19760) audits the credit signals we use to train agents against actual causal ground truth — obtained by re-sampling the policy's alternatives at every decision point and rolling the world forward to see what actually changes about the outcome.

The results are brutal.

The Audit No One Ran

When you train an LLM agent, you need to know which steps mattered. Without that, you're just reinforcing random behavior. Current practice uses three families of credit signals:

LLM-judge scores: have a model rate each step's quality
Outcome-conditioned logprob ratios: compare the probability of the chosen action under success vs. failure conditions
Policy confidence: the model's own likelihood assigned to each action

Existing evaluations grade these signals against annotated step correctness — did the model do the right thing? Zhang audited them against step contribution — does re-sampling this step's alternatives actually change the outcome? — and the two come apart completely.

graph LR
    A[Agent Trajectory] --> B{Step-level signal}
    B --> C[LLM Judge Score]
    B --> D[Logprob Ratio]
    B --> E[Policy Confidence]
    C --> F["Audit: Causal Contribution
(executed replay)"]
    D --> F
    E --> F
    F --> G[None better than chance]
    style G fill:#7f1d1d,stroke:#ef4444

The experiment uses ALFWorld, a single-agent tool environment, and the finding is unambiguous: none of the three signal families identifies which steps causally matter better than chance.

Why the Signals Fail

The root cause is identifiable. Implicit credit — the policy's own confidence — doesn't measure causation. It measures fluency. The median rank correlation between policy confidence and causal contribution is +0.75 in the primary experiment, replicating at +0.70 in a second model family. The model is confident about steps it's good at, not steps that matter.

Conditioning on the outcome adds nothing. The partial correlation between outcome-conditioned signals and causal contribution, controlling for the policy's raw confidence, is -0.004. The outcome tells you the trajectory succeeded or failed — not which step caused it.

The ground truth itself has structure worth understanding:

Causal contribution is sparse: only 30.5% of decision points carry a measurable effect on the outcome
Measurability is model-dependent: the fraction of points with no policy-supported counterfactual differs by a factor of two (13.1% vs. 26.8%) between two similar-scale policies
Most steps don't matter, and the ones that do are hard to identify

The Training Experiment: Dose, Not Content

Zhang ran a pre-registered seven-arm training experiment where agents were trained with different credit signals. The result: no arm reliably outperformed the untrained policy.

Worse, the apparent differences between checkpoints were fully explained by training dose — how many optimizer steps each credit signal preserved. Sparse credit signals retain fewer examples, creating an order-of-magnitude spread in optimizer steps. What looks like a better credit rule is actually just more training.

The implication is devastating for current practice: comparisons of credit rules must match effective sample size, or they measure dose, not credit. Most published comparisons probably fall into this trap.

What a Confidence-Only Router Buys You

There's one positive finding buried here. A confidence-only router — just using the policy's own likelihood to decide whether to query a judge — recovers pivotal steps at chance level, but cuts judge API costs by 13.1% per turn (14.0% per trajectory). If your credit signals are meaningless anyway, at least you can save money on the judge calls.

Why This Paper Matters for People Building Agents

The dominant paradigm in agent training right now is: collect trajectories, have a judge model score each step, train on the ones the judge liked. This paper says that paradigm is built on a signal that doesn't measure what you think it measures.

If you're training production agents:

Your LLM-judge scores are not causal — they're correlated with fluency and correctness, but neither of those is contribution
Outcome weighting is a red herring — conditioning on success/failure adds no causal information beyond what confidence already encodes
More training on bad signals is worse than less training — you're reinforcing non-causal correlations

The paper doesn't claim to have the answer. It provides the audit framework — executed replay as ground truth — and shows that current signals fail against it. That's exactly the kind of uncomfortable negative result the field needs more of.

Limitations

The audit is conducted in ALFWorld, a single-agent text environment with a limited action space. Results may not transfer directly to multi-agent settings, environments with continuous actions, or tool-use scenarios where step outcomes are less binary. The executed replay methodology is also computationally expensive — it requires re-sampling each decision point multiple times — making it impractical as a training signal itself, only as an audit tool.

The paper is also single-author and pre-registered, which is good for rigor but means the results haven't been independently replicated. The findings should be read as a strong warning signal rather than a settled conclusion.

Source:

Credit Without Ground Truth: Auditing Step-Level Credit Assignment in LLM Agents Against Executed Replay — Haiyue Zhang, 21 Aug 2026