LLM Wrong Answers: 2-Parameter Fix Recovers 9–34 Points

When a model fails a reasoning benchmark, the field's default move is to blame the model: not smart enough, not trained enough, not aligned enough. A new paper from Qiyao Yan, Chenpeng Wang, and Liangming Pan argues that a large slice of those failures is something else entirely — a readout gap. The model has the right answer encoded in its hidden states, but the final scoring layer collapses and never says it out loud. And the evidence is a 2-parameter correction that recovers 9–34 accuracy points from models that appeared to fail.

The Problem: Wrong Answers Don't Mean Missing Logic

Benchmarking treats a wrong output as a capability verdict: the model couldn't reason, so it doesn't have the skill. That conflation is the core assumption the paper attacks. Across diverse reasoning benchmarks, the authors find a consistent pattern — structural biases in how sequence scores are assembled drag down native scoring even when the underlying computation is correct. The model computes the right answer, then misreports it. If that's true at scale, benchmark deltas between models are partly measuring output-layer noise, not reasoning ability.

The Method: A Two-Parameter Readout Correction

To test whether instance-specific logic actually survives the collapse, the authors built a deliberately minimal diagnostic: hidden-state probes that decode the answer before the final head, compared against the model's own sequence scoring. Then they fit an additive correction with just two parameters, trained on as few as 25 unlabeled examples — no labels, no fine-tuning, no per-task prompt engineering. The correction re-ranks the model's own outputs to compensate for the structural bias. A model with genuinely absent reasoning shouldn't be salvageable by this; a model with a readout bottleneck should.

flowchart LR
    A[Input prompt] --> B[Transformer layers]
    B --> C[Hidden states]
    C --> D[Final vocab head]
    D --> E[Wrong answer]
    C --> F[Probe + 2-param correction]
    F --> G[Correct answer]
    style E fill:#7f1d1d
    style G fill:#14532d
            

The Results: 9–34 Accuracy Points Recovered

On Qwen3.5 models, the two-parameter correction recovered 9–34 accuracy points on instances where native scoring collapsed. The fix transfers across model families — the same recovered decisions hold on OLMo-2-1B and Llama-3.1-8B. Two controls make the result hard to wave away: the recovered answers persist on hard instances that simple lexical-overlap heuristics cannot reach, and they significantly beat count-preserving permutation baselines (which shuffle scores while keeping the answer distribution intact). That second control is the important one — it rules out the correction just being a cheap re-ranking trick that happens to favor common tokens.

Limitations: What the Paper Does Not Show

The recovered gains are on the failures the probe can see — the readout-gap subset. They do not claim all wrong answers are expression failures, and the 9–34 point swing is bounded to the benchmarks and model families tested. The two-parameter fix is a diagnostic protocol, not a deployed inference-time solution: it requires hidden-state access, which API-only users don't get, and the 25-example calibration set has to be representative of the deployment distribution. The paper also probes in English-centric reasoning settings; whether the same structural biases dominate in multilingual or open-ended generation is unexamined. And as with all probe-based work, a probe finding "the answer is in there" is not proof the model would ever have surfaced it under normal decoding pressure.

Why Builders Should Care

Three practical takeaways. First, don't trust a single benchmark number: a 10-point gap between two models may be a readout artifact, not a capability gap — especially at smaller scales where the output head is a bigger fraction of the model. Second, this is a cheap eval-time audit: if you ship an LLM feature, a two-parameter probe over a couple dozen unlabeled samples can tell you whether your failure modes are recoverable or real. Third, it reframes where to spend effort — this line of work hints that better decoding/readout strategies (rather than more RLVR) are an under-exploited lever on apparent reasoning deficits. That's a very different roadmap than "scale harder."