NeuronFuzz: Peeking Inside the Model to Find Jailbreaks Faster

Jailbreak testing is expensive. Every candidate prompt needs a full response generation — and on strongly aligned models, most of those responses are identical refusals. You burn through tokens and get back almost no signal. It's like fuzzing a kernel with a black-box oracle: each test costs a full boot cycle and most tell you nothing.

NeuronFuzz, from Xu et al., proposes a radically different approach: instead of waiting for the model's output, read its mind during prefill. The framework exploits internal safety neurons — neurons whose activations reliably distinguish harmful from benign inputs — and uses their activation patterns as a continuous, differentiable feedback signal. No response generation needed inside the fuzzing loop.

How it works

Three stages:

1. SafetyOracle construction. Using template-invariant harmful and benign inputs, NeuronFuzz identifies a compact set of safety neurons whose activations capture harmful-intent recognition. Stability-aware selection ensures these neurons generalize across prompt variants, not just one phrasing.
2. Gradient-guided mutation. Because the safety alarm score is differentiable, NeuronFuzz computes gradients through the model's prefill to identify which token positions are most safety-sensitive. A masked language model then generates fluent, context-compatible mutations at those positions while preserving the original harmful payload.
3. Zero-shot transfer. The optimized adversarial templates transfer without any retargeting — to open-weight models and proprietary APIs alike.

flowchart TD
    A["Template-invariant harmful/benign prompts"] --> B["SafetyOracle
(safety neuron activations)"]
    B --> C["Continuous safety alarm score
(during prefill, no response needed)"]
    C --> D["Gradient backprop → identify
safety-sensitive token positions"]
    D --> E["Masked LM generates
fluent mutations at those positions"]
    E --> F["New adversarial template"]
    F --> B
    F --> G["Zero-shot transfer
to target models"]

The key insight: response generation is eliminated from the fuzzing loop entirely. The oracle runs during prefill — the forward pass through the first few tokens — and returns a continuous score. This makes the search dramatically cheaper per iteration and opens the door to gradient-based optimization, which black-box methods can't use.

The numbers

The authors evaluated across 21 models — text and multimodal, white-box and proprietary.

• White-box models (5 source models): NeuronFuzz achieves a 76–100% jailbreak discovery rate, outperforming baselines by up to 48 percentage points.
• Proprietary models (zero-shot): Optimized templates transferred without modification, achieving an average Attack Success Rate (ASR) of 69.6% and a Top-5 Ensemble ASR of 92.6%.
• Open-weight models: Average ASR of 44.1% and Top-5 Ensemble ASR of 60.0%.

These are not cherry-picked: the numbers span the full evaluation set, and the transfer results are genuinely zero-shot — no prompt retuning, no model-specific adaptation.

Limitations

No free lunch. A few worth flagging:

• White-box access required for oracle construction. You need the model's internals to identify safety neurons in the first place. The paper uses template-invariant inputs and stability-aware selection, but this step still demands a transparent model.
• Transfer is unidirectional. Templates found on a white-box source transfer to proprietary models, but you can't use those proprietary models to improve the search — the fuzzing loop itself needs gradients.
• SafetyOracle fidelity. The oracle is a proxy for harmfulness. If the safety neurons miss a novel attack vector (e.g., one that exploits a completely different circuit), the fuzzer won't find it. The approach inherits the blind spots of whatever internal safety mechanism it reads.
• Evaluation scope. The 21 models are substantial but still a slice of the ecosystem. Proprietary API behavior can change with model updates, and the paper reports a point-in-time snapshot.

Why this matters

Two reasons, one tactical and one strategic.

Tactical: NeuronFuzz makes jailbreak testing orders of magnitude cheaper. When each candidate costs a prefill instead of a full generation, you can explore vastly more of the adversarial input space per dollar. For anyone running red-teaming pipelines — bug bounty programs, safety eval teams, internal audit — this directly moves the bottleneck.

Strategic: The paper demonstrates a broader principle that deserves attention: alignment-relevant signals live inside the model, not just in its outputs. As models get increasingly refusal-hardened, output-level evaluation becomes sparser and less useful. Internal monitoring — safety neurons, representation probes, activation patterns — becomes the viable path. NeuronFuzz is an early, concrete proof that this works in practice, not just in theory.

The flip side: the same techniques can be weaponized. If safety neurons are discoverable and their activations differentiable, adversaries can optimize against them directly. This is a cat-and-mouse game, and NeuronFuzz just gave the mouse a gradient.

Paper published Aug 26, 2026. Open-source code and data available on request from the authors.