LLM Geometry of Ignorance: Unigram Prior in Unembedding Matrix

Every large language model has a secret floor it falls through when the context runs thin. Not a hallucination — something deeper. A geometric structure in the unembedding matrix that encodes exactly what the model "thinks" before it has any evidence at all.

The Geometry of Ignorance: LLMs Know When to Temper Bayesian Priors (Liu et al., 2026) identifies a single direction in the unembedding matrix — present in every model family tested from 0.4B to 405B parameters — that encodes the unigram distribution of the training corpus. It's the model's Bayesian prior, geometrically instantiated. And it's causally active: tweak it, and you steer predictions toward or away from common tokens.

The Problem: What Does an LLM Predict With No Context?

When a language model sees "The capital of France is ___", the answer is obvious. But what about "The ___"? With no context, the model must fall back on something. The obvious guess is the most frequent token in the training data — "the", "a", punctuation — but how does the model actually compute this fallback? And does it know when to rely on it vs. when to trust the context?

Prior work on logit lens techniques showed that intermediate representations can be decoded into vocabulary distributions, suggesting the model maintains something like a prior throughout its forward pass. But nobody had located a geometric structure that cleanly separates prior from evidence — or shown that this structure is consistent across architectures, sizes, and training runs.

The Method: Finding the Direction of Ignorance

The authors discovered that a single direction in the unembedding matrix encodes the unigram prior. They call it the direction of ignorance — the first principal component of the unembedding matrix rows — and it appears in all four families examined: Llama, Qwen, Gemma, and Pythia, at every scale from 0.4B up to 405B parameters.

Here's the core decomposition. For any final hidden state h, the projection onto this direction yields a scalar λ (the prior loading factor). The hidden state decomposes into two orthogonal components:

graph TD
    H[Final Hidden State h] --> P[Project onto Direction of Ignorance]
    P --> L[Prior Loading Factor λ]
    P --> O[Orthogonal Component]
    L --> U[Unigram Prior p⁽λ⁾]
    O --> C[Context-Driven Likelihood]
    U --> R[Prediction = Tempered Bayesian Update]
    C --> R

This isn't just a mathematical curiosity. The decomposition corresponds exactly to a tempered Bayesian update: the unigram prior raised to the exponent λ, multiplied by a context-driven likelihood. When λ is high, the model leans heavily on token frequency. When λ drops, context takes over.

Critically, λ is calibrated — it means the same thing across model sizes. A λ of 0.3 on a 7B Llama model represents the same prior reliance as λ=0.3 on a 405B model. This is rare in mechanistic interpretability and enables direct cross-family comparisons.

The Results: Larger Models Are Less Gullible

The key empirical finding: λ declines steadily as context becomes more informative, across all model families. The model doesn't blindly use its prior — it tempers it based on available evidence.

But the rate of tempering differs by scale:

• Larger models (70B, 405B) exhibit significantly lower prior reliance in the high-context limit. They learn to trust evidence more aggressively.
• Smaller models (0.4B–7B) maintain higher λ even with substantial context, defaulting to frequency-based guesses longer.
• The direction of ignorance is causally active: artificially increasing λ at the final hidden state shifts predictions toward the unigram prior (measured in KL divergence). Decreasing λ pushes predictions away from it.

The authors tested this by intervening on the residual stream at the final layer, demonstrating that the effect is not just correlational. You can literally turn up the model's "ignorance" and watch it predict more common tokens.

Limitations

The paper is clean but leaves open questions. The decomposition was verified on English-dominant models — whether the direction of ignorance exists and behaves identically in multilingual or code-trained models isn't established. The causal steering experiments show λ shifts predictions, but the effect size is modest and not yet explored for multi-token steering. The paper also doesn't address whether finetuning (RLHF, instruction tuning) modifies the direction of ignorance or only the context-driven likelihood component.

Most importantly: the direction of ignorance is a descriptive structure. The paper shows it exists and is causally relevant, but doesn't yet explain how the model computes λ — what circuits determine when to trust the prior vs. the context.

Why This Matters for Building

For anyone shipping LLM-based products, this paper has immediate implications:

1. Calibration debugging. If your model defaults to common completions even with rich context, its λ isn't dropping enough. The direction of ignorance gives you a direct diagnostic: measure λ on your task and compare to the expected decay curve for your model size.
2. Steerable priors. The causal steering result suggests you could manipulate λ at inference time to control the model's conservatism — making it more adventurous (lower λ) or more conservative (higher λ) without retraining.
3. Architecture comparison. The calibrated λ lets you compare how different model families handle uncertainty. If you're choosing between architectures, this is a meaningful axis beyond benchmark scores.
4. Token-level confidence. Per-token λ values could serve as a built-in confidence measure — low λ means the model is certain; high λ means it's guessing based on frequency. No need for external confidence calibration.

The unembedding matrix has been treated as a boring linear projection. This paper shows it's doing something far more interesting: it's the model's built-in prior, geometrically encoded and dynamically tempered. Every LLM carries a measure of its own ignorance, and it knows exactly when to use it.