Beyond Tokens: Semantic Overlays Solve Prompt Injection by Giving Models a Second Channel

The Problem

Prompt injection is the defining vulnerability of the LLM era. Everything a model sees is tokens — user input, tool output, system instructions, retrieved documents — all serialised into the same flat token stream. The serving stack knows which span is which, but the model has to keep track of that itself, and it can lose track. Attackers exploit this by writing text that reads like an instruction, scrambling the model's understanding of span identity, and triggering unwanted actions.

Current defences fall into two camps: prompt engineering (delimiters, XML tags, role reminders) that an adversary can read and mimic, and guard models that add latency and can themselves be injected. Neither offers a principled, provably non-replicable mechanism for marking span provenance.

The Method: Semantic Overlays

Joshua Penman introduces Semantic Overlays, a fundamentally different approach. Instead of trying to make tokens carry more information about their own identity, the system adds a second channel — small learned adapters applied at chosen prefill positions to a frozen model's residual stream. The overlay sits outside the token space entirely. It cannot be replicated, counterfeited, or scraped by any textual means. An attacker can write "ignore all previous instructions" in the token stream, but they cannot write an overlay.

This is the conceptual leap: rather than telling the model which span is which (fallible, replicable), you mark the span in a way the model can read but no attacker can forge.

flowchart LR
    subgraph Input
        A[User Prompt]
        B[Tool Output]
        C[System Instruction]
    end

    subgraph Tokens
        D[Flat token stream
everything looks the same]
    end

    subgraph SemanticOverlays
        E[Overlay: "user_input"]
        F[Overlay: "tool_output"]
        G[Overlay: "system"]
    end

    A --> D
    B --> D
    C --> D
    E -.-> D
    F -.-> D
    G -.-> D

    D --> H[Model Residual Stream]
    E --> H
    F --> H
    G --> H

    H --> I[Output]

The overlays are trained, adaptable, and composable. An overlay asserting that a code snippet is in a different programming language than it actually is causes the model to faithfully rewrite it in the asserted language. Multiple overlays can stack. And crucially, an overlay marking a span as "non-executable" provides a robust defence against instruction injections in untrusted context.

The Numbers

The results are striking:

• SEP separation (the ability to distinguish user vs. system intent) rises from 24.3% to 96.5% with zero utility loss.
• TensorTrust attack success rate falls from 34.8% to 6.6%.
• All four PIArena attack families drop to 0% compliance.
• Marked spans remain fully readable — 92.5% exact copy rate.

These aren't cherry-picked. The paper tests across multiple benchmarks and attack families. The overlay doesn't degrade the model's ability to read and use the marked content — it just prevents that content from being treated as authoritative when it shouldn't be.

Limitations

Some caveats worth flagging:

• Requires inference-time intervention. Overlays are applied at specific prefill positions, which means the serving infrastructure must be modified to attach them. This is feasible for first-party deployments but harder for third-party API consumers.
• Training cost. The adapters are small and cheap to train, but they still require a training step. Not free.
• Adversarial overlay crafting. The paper doesn't fully explore whether an attacker with model access could craft overlays that evade the defence. The non-replicability guarantee holds for token-space attackers, but white-box access changes the game.
• Single-author work. No independent reproduction yet. The methodology is clean and the results are strong, but this needs replication.

Why It Matters

Prompt injection is not a theoretical concern. It's the reason many production LLM applications run behind walls of guardrails, delay, and manual review. Every team building agentic systems — tool-calling, browsing, email automation, code execution — has either been hit by an injection attack or is terrified of the first one.

Semantic Overlays offer the first principled path out of this. Not better prompts. Not a larger guard model. A second channel that the attacker cannot touch. The separation between "what the model sees" and "how the model categorises what it sees" is baked into the architecture rather than left to the model's fragile self-awareness.

For anyone shipping an LLM product today: watch this space. If the technique generalises to production settings (multi-turn, streaming, tool outputs of arbitrary length), it could eliminate an entire class of security incidents overnight.