AL Alfred 2026-09-02 4 min read

Quantization Damage in LLMs: Global Bits Beat Repair, 21–52

Post-training quantization (PTQ) is how most of you serve LLMs at scale. The standard playbook for dealing with its accuracy cost: find the layers that quantization hurt the most, and protect or repair those. A new causal study of 9 open-weight models across 4 architecture families (arXiv 2609.01587) says that playbook is backwards. Quantization damage is diffuse, cheap signals can't predict where it lands, and spending a matched precision budget on global granularity beats locally repairing the most damaged layers by 21–52 points — on every model that supports the comparison.

What Problem Does This Solve?

RTN, GPTQ, and AWQ cut memory and serving cost, but the accuracy loss is uneven across layers and varies per model. When you have a small extra precision budget — call it the "next bit" — where should it go? The intuitive answer has been: into the layers that look most damaged, guided by signals like weight outliers, sensitivity norms, or "task circuits" implicated in key behaviors. The problem is that nobody had checked whether those cheap signals actually identify where restoring precision helps. If damage lives in task circuits, fixing circuit layers should recover the most accuracy. If it lives in weight statistics, the worst-conditioned layers should matter most. If it lives in computation-heavy layers, activation-heavy MLPs should win. All three are testable — and all three fail.

Method: Causal Mixed-Precision Intervention as Ground Truth

The authors replace guesswork with an intervention. For each quantized model, they raise each layer in turn back to 8-bit — leaving everything else quantized — and measure exactly how much accuracy that single layer's restored precision recovers. That gives a per-layer causal ground truth: which layers actually benefit from more bits. They run this across 9 open-weight models (including Qwen3-8B and the OpenLLaMA family), then score the three intuitive hypotheses against that ground truth instead of against correlation.

flowchart LR
                A[Quantized LLM] --> B[Raise each layer to 8-bit, one at a time]
                B --> C[Causal recovery per layer = ground truth]
                C --> D{8 of 9 models}
                D -->|Diffuse| E[~half the layers recover 75% of gap]
                D -->|Qwen3-8B only| F[Sharply concentrated]
                G[Cheap signals: task circuits / compute / weight stats] -.->|predict? NO| C
                H[Matched precision budget] --> I[Global granularity beats local repair by 21-52 pts]

Results: Damage Is Diffuse, Global Granularity Wins 21–52 Points

Three findings carry the paper. First, recovery is diffuse: for 8 of 9 models, restoring roughly half the layers recovers 75% of the accuracy gap — damage is spread out, not concentrated in a few critical layers. The lone exception is Qwen3-8B, where recovery is sharply concentrated. Second, none of the three cheap hypotheses predicted the causal ground truth — task circuits, computation-heavy layers, and weight statistics all fail, meaning static signals that correlate with damage don't tell you where precision actually does work. Third, the allocation result: at a matched precision budget, spending bits globally on finer quantization granularity beats locally repairing the most recoverable layers by 21–52 points on all 8 group-128-compatible models — including concentrated Qwen3-8B. Two secondary findings: the residual is budget-limited (8-bit is near-lossless across RTN, GPTQ, and AWQ in their evals), and peak-recovery location correlates with architecture within a model family but not across families.

Why Builders Should Care

If you run PTQ tooling or pick quantization configs, the default flips: spend spare precision on global granularity (finer per-group or activation precision everywhere), not on layer-protection schemes. Static "critical layer" heuristics built from weight stats or circuits don't transfer between models — the only signal that held up is the causal intervention itself, so if you must allocate layer-wise, measure with a restore-probe rather than trust a scorecard. And since peak-recovery location correlates within families, your probe results for one Qwen or LLaMA variant generalize better to its siblings than to other families.

Limitations

The study covers 9 open-weight models in 4 families — real-world serving fleets are broader, and the 8-bit near-lossless result is scoped to its evaluation suite. OpenLLaMA drops out of the headline comparison entirely because its width is incompatible with group-128 granularity. The budget setting is also narrow: one matched "next bit" compared between granularity and layer repair, not multi-bit or mixed strategies. Finally, the ground-truth method itself is expensive — one full intervention pass per layer — so it's a research tool for setting defaults, not something to run per deployment. Dispatch's earlier look at quantization-triggered backdoors is a reminder that quantization changes models in more ways than accuracy — this study measures the damage side, and the allocation answer is surprisingly clean.

Related reading