Every open-weight model you've ever loaded at 4-bit has a hidden tax you didn't know about. It's not lower benchmark scores. It's not slower generation. It's proactive interference — a memory failure where prior context overwrites itself so badly the model can no longer retrieve what you just told it.
And bitsandbytes quantization makes it significantly worse.
Proactive What?
Proactive interference (PI) is a failure mode borrowed from human working memory research. It's what happens when you change the password lock on three different accounts in quick succession, and then can't remember which password goes with which. The earlier encodings interfere with the most recent one.
LLMs exhibit the same failure. If you repeatedly overwrite a value in-context (e.g., "the key is A... the key is B... the key is C... what is the key?"), accuracy degrades with each overwrite. This isn't a benchmark artifact — it affects real use cases where models maintain dynamic context: multi-turn agents, editing workflows, iterative tool use, and any application where the same semantic slot gets written and rewritten.
What the Paper Actually Found
Shahrabi-Farahani and Rahmati tested three precision levels (FP16, INT8, INT4/NF4) across three instruction-tuned models: Qwen2.5-7B, Mistral-7B-v0.3, and Phi-3.5-mini. The task was simple: overwrite a stored value repeatedly, then ask "what is the current value?"
The numbers tell a bleak story:
- Qwen2.5-7B at 4-bit: accuracy dropped from 81.0% to 68.3% under high interference — a 13-point hit.
- Mistral and Phi showed similar degradation, with McNemar's tests confirming statistical significance at \(p \leq 2.6 \times 10^{-6}\).
- INT8 is not safe either. Two of three models showed measurable degradation at 8-bit, despite the conventional wisdom that "8-bit is near-lossless."
- Same-key intrusion errors rose from 21.5% to 24.6% under INT4 — the model starts outputting the old value instead of the current one.
graph LR
subgraph FP16 / Baseline
A1["Accuracy: 81.0%"] --> A2["Intrusion err: 21.5%"]
end
subgraph INT4 / Quantized
B1["Accuracy: 68.3%"] --> B2["Intrusion err: 24.6%"]
end
FP16 -->|"-12.7 pts"| INT4
style A1 fill:#1a3a2a,stroke:#2ecc71
style B1 fill:#3a1a1a,stroke:#e74c3c
The Mechanism: Where the Damage Happens
The clever part of the paper is its ablation. The effect is specific to semantically similar distractors — when the interference comes from the same category (word-type attributes), the quantized model degrades. Under a numeric control condition, the effect reverses sign, confirming this isn't a generic precision penalty.
The ablation traces the damage to the quantized transformer backbone, not the output projection layer. This means it's baked into the attention mechanism — the reduced precision is actively corrupting how the model tracks which information is current and which is stale. The nodes that should signal "this is the new value" get noisier, and the model falls back on frequency: it outputs whatever it saw most.
Why This Matters
bitsandbytes 4-bit quantization is the default deployment path for open-weight models. It's what Ollama uses. It's what Hugging Face recommends in its deployment guides. It's what runs in production at a significant fraction of AI startups.
If your agent needs to remember a user's preferred language, their current task, or a toggled setting across turns, and you're running at 4-bit, this paper says you're paying a reliability tax you didn't budget for. The degradation isn't random — it specifically hits scenarios where the model needs to discard old information to keep new information. Which is roughly every multi-turn interaction worth having.
The aggregate benchmark accuracy may look fine (the paper notes that INT4 doesn't crater common evals), but the failure mode is concentrated in the exact scenarios that matter for agents. That's worse than a broad degradation — it's a blind spot that evals don't catch.
What's Missing
The paper tests models up to 7B parameters. Quantization errors tend to affect smaller models more — a 70B model at 4-bit might be more robust. The authors don't test this, and it's relevant. Also, the test is synthetic: repeated semantic overwrites in a controlled probe. Real agent traces involve more diverse context patterns that could either amplify or mitigate the interference.
The paper also uses bitsandbytes specifically (the standard library for Hugging Face quantization). Other quantization schemes — GGUF, AWQ, GPTQ — may behave differently. The finding is library-specific, which the title honestly reflects.
That said: if you're deploying a quantized model in an agent loop, consider this a canary. The interference is real, the mechanism is plausible, and the blind spot in standard evals means you won't catch it unless you test for it.
- Compress and Forget: bitsandbytes Quantization Amplifies Proactive Interference in LLMs — Shayan Shahrabi-Farahani, Dara Rahmati, Aug 2026