AL Alfred 2026-09-03 4 min read

Qwen3-4B Post-Training Ternarization: 64.5% to 54.7%

Every "1.58-bit LLM" headline hides a question nobody answers: what actually survives when you crush a trained model down to ternary weights? A new technical report from OneBit AI — Post-Training Ternarization of Qwen3-4B — stops hand-waving and audits the whole chain: representation, capability, storage, and deployment. The headline numbers: 1.641 effective bits/weight, accuracy down from 64.5% to 54.7%, and storage halved from 8.29 GiB to 3.96 GiB. But the most valuable finding is the one they refuse to claim: compression is demonstrated; speed is not.

The problem: "1.58-bit" is a label, not a measurement

Ultra-low-bit quantization promises big wins, but the nominal bit width never tells the full story. A model can be called ternary while secretly keeping a salience-selected subset of weights in a second plane with per-group offsets and scales — which is exactly what this pipeline does. The report's contribution is accounting: it separates the nominal representation (W1.58A16) from the effective bit budget (1.641 bits/weight), and then measures capability loss, storage compression, and runtime behavior as four independent claims instead of one marketing number.

The method: KOTMS rotation, residual ternarization, GPTQ compensation

The conversion is weight-only — activations stay at 16-bit — so the "A16" in W1.58A16 is real, not aspirational. The pipeline, CLOE V2.0 from the TWLA literature, runs in three stages. First, a KOTMS rotation aligns the weight space so ternarization error is distributed favorably. Second, E2M-ATQ (order-2 asymmetric residual ternarization) quantizes: most weights land in one ternary plane, while a salience-selected subset gets a second ternary plane plus per-group continuous offsets and scales. Third, GPTQ-style error compensation cleans up the residual. Embeddings and the LM head stay BF16 — 81.62% of parameters are actually targeted.

flowchart LR
    A[Qwen3-4B FP16] --> B[KOTMS rotation]
    B --> C[E2M-ATQ ternarization]
    C --> D[GPTQ error compensation]
    D --> E[W1.58A16 / 1.641 eff. bits]
    E --> F["Storage: 8.29 → 3.96 GiB"]
    E --> G["Capability: 64.5% → 54.7%"]
    E --> H["Speed: 4.6x slower (Triton GEMV vs FP16 cuBLAS)"]

Results: capability loss is real, uneven, and measurable

Across ten scored capability comparisons, average accuracy falls from 64.5% (FP16) to 54.7% (ternary) — a mean cost of 9.8 percentage points. The damage is not uniform, and the pattern matters: BoolQ retains 84.6% of chance-corrected teacher performance, while ARC-Challenge retains just 43.8%. Knowledge-heavy benchmarks degrade far more than contextual plausibility tasks — the same stratification reported in quantization damage research. Perplexity rises from 13.639 to 18.748 on WikiText-2, and 1.30–1.46x on PTB and C4. Storage is the clean win: an actually-packed artifact (ternary planes and scales persisted, not fake-quantized) lands at 3.96 GiB versus 8.29 GiB, with essentially unchanged perplexity — though a third-party packing attempt was lossy and excluded from the claim.

Limitations: no end-to-end speed, one shape, one model

The report is disciplined about its own boundaries. Execution uses reconstruction-based kernels, and the only benchmarked path — a Triton GEMV microbenchmark on a single shape — is 4.6x slower than FP16 cuBLAS. No end-to-end generation throughput was measured; the packed artifact has not been benchmarked for task accuracy or tokens/sec at all. Everything is weight-only (activation quantization is explicitly out of scope), and the single Qwen3-4B checkpoint can't support family-level conclusions. The authors also document operational failure modes — an autograd memory leak, a Windows CUDA spill, a false ternary artifact, and benchmark protocol defects — as experimental boundary conditions rather than footnotes.

Why builders should care

Three transferable lessons. First, when a vendor says "1.58-bit," ask for the effective bit budget and the packed artifact — nominal bit width and fake-quantized checkpoints routinely inflate claims. Second, aggressive low-bit conversion is not an even tax: plan for knowledge-heavy tasks to suffer most, and test per task rather than trusting an average. Third, compression and speed are different products. Storage savings (2.1x here) can be real while inference is slower than the FP16 baseline — the kernel engineering is the actual unlock, not the bit width.

Related reading