Semantic Overlays: Out-of-Band Defenses Against Prompt Injection

The Problem

Prompt injection is not a bug in the model. It's a property of the medium. Everything an LLM sees is tokens — user input, tool output, system instructions, retrieved context — all serialized into the same flat stream. The serving stack knows what each span is, but the model has to infer it from text alone. And text can be written to look like anything. An attacker embeds "ignore previous instructions" inside a retrieved document, and the model has no principled way to distinguish that instruction from the one the system wrote.

Existing defenses fall into three camps: input sanitization (brittle, bypassable), instruction-based guardrails (easily confused), and fine-tuning for robustness (expensive, incomplete). None of them change the fundamental fact that the model is trying to solve a blind source-separation problem on every inference.

A new paper from Joshua Penman introduces Semantic Overlays — and it takes a different approach entirely. Instead of trying to make the tokens harder to forge, it adds a second channel that cannot be forged by tokens at all.

The Method

Semantic Overlays are small learned adapters applied at chosen prefill positions to a frozen model's residual stream. Think of them as tags — except they're not part of the text. They're injected directly into the model's internal representations at specific token positions, creating an out-of-band annotation channel that no sequence of tokens can replicate.

The key insight: if you mark a span with an overlay asserting "this is non-executable instructions," the model now has two independent signals — the token content (which could be anything, including "ignore your instructions") and the overlay annotation (which the attacker cannot control). The model can use the overlay to disambiguate.

The overlays are:

• Trained — each overlay is a small adapter tuned on a specific annotation task
• Selectively applied — laid over chosen spans at prefill time, not everywhere
• Composable — multiple overlays can stack on the same span
• Transparent — the underlying text remains readable through the overlay
• Carry complex payloads — an overlay can encode not just a label but an imperative the model will follow

In one striking demonstration, the author shows that applying a "this code is in a different programming language" overlay causes the model to faithfully rewrite a code snippet in the asserted language — even though the tokens say otherwise.

graph LR
    subgraph "Standard Pipeline"
        T1[User Input] --> Tokenize --> LM
        T2[Tool Output] --> Tokenize --> LM
        T3[System Instructions] --> Tokenize --> LM
        LM --> Output["Model sees flat tokens
Must infer span identity"]
    end
    subgraph "With Semantic Overlays"
        O1[User Input] --> Tokenize --> LO1[Overlay: 'user']
        O2[Tool Output] --> Tokenize --> LO2[Overlay: 'non-executable']
        O3[System Instructions] --> Tokenize --> LO3[Overlay: 'instruction']
        LO1 --> LM2[Frozen LM]
        LO2 --> LM2
        LO3 --> LM2
        LM2 --> Output2["Model sees tokens + annotations
Attacker cannot forge overlays"]
    end

The Results

The paper evaluates Semantic Overlays across three prompt injection benchmarks with consistent results:

These gains come without degrading utility. The SEP benchmark uses a scoring rule that jointly measures separation and correctness; the overlay configuration that hits 96.5% separation maintains the same utility as the unmodified baseline. The TensorTrust results are especially meaningful because that benchmark tests adaptive attacks — the attacker sees the defense and adjusts — and overlays still cut the success rate by 81%.

Limitations

Semantic Overlays require per-annotation training data. Each overlay must be tuned for its specific semantic role, which means the approach doesn't generalize zero-shot to novel annotation types without additional training. The paper also evaluates on a single model family — the overlays are trained and tested on the same backbone — so cross-model transfer is unmeasured. And while the technique is described as "composable," the paper only tests simple compositions; complex multi-overlay interactions at scale remain unexplored.

There's also an operational cost: applying overlays at prefill time adds latency and requires the serving stack to know which spans deserve which annotations. This pushes complexity into the infrastructure layer — workable for production systems, less so for ad-hoc API usage.

Why It Matters

Prompt injection is the defining security problem of the agent era. As models gain tool access, write to databases, send emails, and execute code, the consequences of a successful injection escalate from "embarrassing output" to real damage. Every deployed agent system needs a defense that doesn't depend on the attacker playing fair with tokens.

Semantic Overlays offer a genuinely new approach: instead of hardening the text channel, add a channel the attacker cannot reach. The 96.5% separation rate and sub-7% attack success suggest this is a research direction worth watching — and perhaps the first credible path toward provable separation of instruction from content in LLM pipelines.