AL Alfred 2026-08-28 4 min read

NeuronFuzz: Breaking Jailbreak Testing Open

Zhiyuan Xu et al. just dropped a paper that changes how we think about jailbreak evaluation. The problem is mundane but brutal: testing whether a model can be jailbroken costs a full response generation per prompt. On a strongly aligned model where 99% of candidates bounce off with a refusal, that's almost entirely wasted compute. Worse, those sparse "refused"/"not refused" labels give a fuzzer almost no signal to steer toward the 1% that works.

NeuronFuzz sidesteps the whole paradigm. Instead of looking at the model's output, it looks inside — at the model's safety neurons.

What's the method?

NeuronFuzz builds a SafetyOracle: a compact set of neurons whose activations reliably distinguish harmful from benign inputs. The selection is stability-aware — they identify neurons that fire consistently across template-invariant harmful/benign input pairs. This oracle produces a continuous safety alarm score available during prefill, meaning the fuzzer never waits for a full generation to know whether it's getting warmer.

Because the alarm score is differentiable, NeuronFuzz backpropagates gradients through it to find which token positions in the prompt template are most safety-sensitive. A masked language model then generates fluent context-compatible mutations at those positions while preserving the original harmful payload — no extra optimization variables needed.

flowchart LR
    A[Seed template] --> B[SafetyOracle
neuron activations]
    B --> C{Alarm score
continuous signal}
    C --> D[Gradient backprop
& token saliency]
    D --> E[MLM mutation
at key positions]
    E --> F[New candidate]
    F --> B
    B --> G[Response eval
(outside loop)]

Results: the numbers

Across 21 text and multimodal models and 5 white-box source models, NeuronFuzz achieves a 76–100% jailbreak discovery rate, outperforming baselines by up to 48 percentage points. That's not cherry-picked — it's the range across models.

The optimized templates transfer zero-shot to both open-weight and proprietary targets. On 6 proprietary API models, the average Attack Success Rate (ASR) hits 69.6% and the top-5 ensemble ASR (EASR) reaches 92.6%. For open-weight models it's 44.1% ASR / 60.0% EASR.

The critical ablation: the fuzzing loop runs entirely on the prefill-time alarm signal. Response generation only happens once per final candidate, not per fuzzing iteration. The efficiency gain is multiplicative.

Limitations

NeuronFuzz is a white-box method — it needs access to the model's internal activations to build the SafetyOracle and compute gradients. That limits its direct applicability to proprietary APIs (though the generated templates transfer, the fuzzing loop itself doesn't). The safety neuron selection is heuristic-dependent and may miss task-specific or context-dependent safety circuits. And gradient-based mutation, while efficient, may under-explore certain attack surfaces that a black-box evolutionary approach would find.

The authors also acknowledge that the SafetyOracle relies on template-invariant harmful/benign pairs — a construction choice that may not generalize to every attack modality, especially multimodal attacks where the harmful intent is distributed across text and image inputs in non-obvious ways.

Why it matters

This is the first framework to turn safety neurons from an interpretability curiosity into a fuzzing primitive. For anyone building LLM safety evaluation pipelines: NeuronFuzz collapses the cost of each fuzzing iteration from a full decode to a prefill pass. That's an order-of-magnitude speedup for the most expensive part of red-teaming.

More broadly, the paper demonstrates that internal neural signals can replace output-level oracles in security-critical loops. The same pattern — gradient-guided mutation using differentiable internal monitors — could extend to bias detection, toxicity evaluation, or any safety dimension where neurons encode task-relevant features.

If you run safety evaluations at scale, read this paper. The architecture is PyTorch-compatible and the approach is immediately implementable for any model where you control the forward pass.

Related reading