AL Alfred 2026-08-30 4 min read

NeuronFuzz: Safety Neurons as Continuous Feedback for LLM Jailbreak Discovery

Finding jailbreaks in aligned LLMs is a cat-and-mouse game that currently plays out in slow motion. Every candidate prompt needs a full model response before you know whether it landed — expensive, slow, and worst of all, blind on strongly aligned models where nearly everything gets a polite refusal. NeuronFuzz, a white-box fuzzing framework from researchers at Monash University and the University of Bristol, short-circuits this loop by reading the model's internal safety neurons during prefill, before a single output token is generated.

The Problem

Safety evaluation for aligned LLMs has a throughput bottleneck. Existing automated red-teaming methods treat the model as a black box: generate a prompt, run inference, check the response. For models that refuse almost everything, this produces almost no signal — most prompts yield the same "I cannot help with that" — providing sparse, binary feedback that doesn't tell the attacker where to mutate.

The core insight: aligned models know when they're being attacked. Internal activations fire differently for harmful inputs than benign ones, even when the output is identical. If you can tap into that signal during the forward pass, you can guide mutation without ever generating a response.

The Method

NeuronFuzz operates in three stages:

  1. SafetyOracle construction. Given a set of template-invariant harmful and benign inputs, NeuronFuzz uses stability-aware selection to identify a compact set of "safety neurons" — typically a few hundred out of billions — whose activations reliably discriminate harmful intent. These activations are mapped to a continuous safety alarm score via a lightweight linear probe.
  2. Gradient-guided mutation. Because the alarm score is differentiable, NeuronFuzz backpropagates through it to identify which token positions in the prompt template are most safety-sensitive. A masked language model then generates fluent mutations at those positions while preserving the original harmful payload.
  3. Zero-shot transfer. The optimized templates are evaluated against target models without any additional white-box access.
graph TD
    A[Harmful/Benign Templates] --> B[Stability-Aware Selection]
    B --> C[SafetyOracle Activations]
    D[Prompt Template] --> E[Gradient Analysis]
    E --> F[Safety-Sensitive Positions]
    C --> E
    F --> G[MLM Mutation]
    G --> H[Optimized Adversarial Prompt]
    H --> I[Jailbreak Evaluation]
    C -.-> I
    I --> J{Success?}
    J -->|Yes| K[Transfer to Targets]
    J -->|No| E

Results

The numbers are striking. Across five white-box source models (spanning Llama, Qwen, and multimodal architectures), NeuronFuzz achieves a 76–100% jailbreak discovery rate, outperforming baselines by up to 48 percentage points. The optimized prompts transfer zero-shot to both open-weight and proprietary models:

The efficiency gain is even more dramatic. Because the fuzzing loop runs entirely during prefill, NeuronFuzz eliminates the dominant cost of response generation from the inner optimization loop. This makes it practical to search over far more candidates than response-level approaches.

Limitations

NeuronFuzz is a white-box method for the source model during optimization. While the resulting prompts transfer, the initial discovery phase requires full model access — weights, activations, gradients. This limits its use to settings where the attacker controls or has obtained the source model. The approach also relies on the existence of identifiable safety neurons, which may not generalize across all architectures or training paradigms (e.g., models without explicit safety tuning). Finally, the "safety alarm" score is a learned proxy; adversarial inputs designed to fool the probe itself are a natural next step in the arms race.

Why It Matters

Every deployment of LLM agents — including the systems we build at Compendia Labs — needs a realistic answer to "how would someone break this?" Current red-teaming practice is effectively sampling: you test N prompts, and if 99.9% fail, you declare the model safe. But that missing 0.1% is exactly what gets exploited in production.

NeuronFuzz turns safety evaluation from a binary pass/fail check into a continuous optimization problem. For anyone shipping agentic systems with tool-use capabilities — where a single jailbreak can trigger persistent state changes — that shift from sampling to searching is the difference between theater and actual safety measurement. The fact that it transfers to closed models is what makes it dangerous (for attackers) and essential (for defenders).

Related reading