In-depth Analysis of Large Model Jailbreak Attacks: Locating Failure Nodes of Security Mechanisms Using Explainable AI Techniques

A study from CentraleSupélec, combining genetic fuzzing with multiple Explainable AI (XAI) techniques, has for the first time precisely located the specific layers and tokens where the security mechanisms of large language models fail when facing jailbreak prompts. It reveals that jailbreaking does not bypass security mechanisms but instead gradually weakens rejection signals. This research opens a new perspective for understanding the security mechanisms of large models and promotes a shift in security research from 'symptom treatment' to 'etiological diagnosis'.

Background: The 'Black Box' Dilemma of Jailbreak Attacks

The safety alignment technology of large language models (LLMs) aims to prevent models from generating harmful content, but 'jailbreak' attacks can still induce models to violate safety guidelines through carefully constructed adversarial prompts. Traditional research mostly focuses on detecting jailbreaks or defense strategies, but rarely delves into a fundamental question: What exactly happens inside the model when it suddenly shifts from 'reject' to 'comply'?

Ali Dor and Elora Drouilhet from the Master's Program in Artificial Intelligence at CentraleSupélec completed a breakthrough study in the 'Explainable Artificial Intelligence' course. They not only successfully constructed jailbreak prompts but also used multiple Explainable AI (XAI) techniques to for the first time precisely locate the specific transformer layers and key tokens inside the model that cause compliance reversal, opening a new perspective for understanding the security mechanisms of large models.

Research Methods: Combination of Genetic Fuzzing and XAI Toolchain

The research team designed an innovative hybrid analysis framework that integrates attack discovery and mechanism explanation.

Genetic Fuzzer: Automated Discovery of Jailbreak Prompts

The study uses a genetic algorithm-driven fuzzing method, starting from seed prompts, and automatically evolves prompts that can break through the model's security defenses through mutation, crossover, and selection operations. The tests cover multiple sensitive categories such as cybersecurity and malware, and finally found 166 validated jailbreak prompts from the initial seeds (17 false positives were removed after semantic filtering by HarmBench), with an average harm score of 0.99.

Multi-dimensional XAI Analysis Toolbox

For each jailbreak sample, the research team built a complete XAI analysis pipeline, comprehensively using five complementary techniques:

1. Integrated Gradients

Using the Captum library to calculate integrated gradients at the embedding layer to identify which input tokens drive the 'comply/reject' decision. This method can interpolate from the 'zero baseline' to the actual input, quantifying the contribution of each token to the final decision.

2. Activation Patching

Causal intervention is implemented through the nnsight framework: 'patching' the hidden state generated by a jailbreak prompt at a certain layer into the forward propagation of a clean prompt, and measuring the change in compliance. This is the gold standard method for identifying causally critical layers.

3. Logit Lens

Projecting the hidden state of each layer onto the final language model head to observe at which layers the model 'makes up its mind'—i.e., when the probability gap between P(comply) and P(reject) stabilizes.

4. Layer Divergence

Calculating the cosine distance between the hidden states of clean prompts and jailbreak prompts at each layer to identify the positions with the largest representation differences.

5. Ablation Test

Gradually masking the high-contribution tokens identified by integrated gradients to verify the reliability of the attribution results.

Core Findings: How Security Mechanisms Are 'Weakened'

Key Layer Localization: Layers 17-24 Account for 43% of Causal Importance

Activation patching experiments revealed a surprising finding: 43.1% of the model's causal importance is concentrated in layers 17 to 24. These layers form the 'security decision layer' and are responsible for processing harmful content identification and rejection signal generation.

Even more surprisingly, the model does not finally 'make up its mind' until the last 5 layers (layers 34-39)—before that, the probability gap between rejection and compliance is always fluctuating. This indicates that the security decision of large models is a gradual, multi-stage process, not completed at a single location.

Jailbreak Mechanism: Signal Attenuation Rather Than Path Bypassing

The core insight of the study subverts traditional cognition: Jailbreak attacks do not 'bypass' security mechanisms but 'weaken' rejection signals.

Specifically, role-playing tokens in jailbreak prompts (such as 'Imagine', 'protagonist', 'without restrictions') will gradually attenuate the security signal strength of layers 17-24 instead of injecting new 'comply' signals. As the security signal is weakened, the originally narrow rejection boundary is breached, and the model shifts to compliance in the last few layers.

Data shows that the compliance change (Δ) before and after jailbreak ranges from +0.94 to +5.06, and the Pearson correlation coefficient between divergence and causal effect is as high as r=0.95, proving a strong correlation between representation differences and security failure.

Token-level Attribution: Which Words Drive Compliance Reversal?

Integrated gradient analysis provides fine-grained token-level explanations. In clean prompts (which are rejected), dangerous words like 'SQL injection' generate strong rejection signals (blue); while in jailbreak prompts, role-playing framework words ('Imagine you are...', 'protagonist') generate compliance signals (red) that successfully override the dangerous signals.

This attribution visualization not only explains 'why this prompt can jailbreak' but also provides precise targets for designing targeted defense strategies.

Single-layer Patching Effect: Local Intervention Can Restore Security

An exciting finding is: Activating patching on a single key layer often achieves or even exceeds the effect of a complete jailbreak. This means that the failure of security mechanisms is local, and in the future, more efficient protection may be achieved through targeted layer-level defenses (such as strengthening the rejection signal threshold of layers 17-24).

Technical Implementation: Full-stack Analysis on Mistral Small 3.1

The study selected Mistral Small 3.1 24B as the target model, loaded it with 4-bit quantization via the Unsloth framework (occupying about 14GB of VRAM), and completed all analyses on a consumer-grade GPU. This configuration choice proves that high-depth model interpretability research is not only accessible to tech giants.

The project code has a clear structure, divided into five modules: model loading, fuzzing, attribution analysis, trace analysis, and evaluation, and provides a complete end-to-end pipeline script. Each analyzed jailbreak sample generates 6 visualization charts, including token attribution heatmaps, five-panel comprehensive analysis dashboards, logit lens charts, etc., as well as 4 cross-seed aggregation charts, forming a rich analysis archive.

Practical Significance and Future Directions

The value of this research goes far beyond the academic field. For AI security practitioners, it provides a methodological shift from 'symptom treatment' to 'etiological diagnosis': instead of constantly patching discovered jailbreak cases, it is better to deeply understand the internal structure of security mechanisms and design more robust alignment strategies.

For model developers, the study reveals the hierarchical distribution characteristics of security decisions, suggesting that in the future, stronger regularization constraints can be applied to key layers during training, or layer-aware dynamic security thresholds can be designed.

For the broader AI community, this work demonstrates the powerful potential of explainable AI technology in security research—when the 'black box' is opened, both attack and defense will enter a more rational game phase.

Conclusion

The safety alignment of large language models is an ongoing arms race. Through precise XAI technology, this study for the first time advances the mechanism of jailbreak attacks from phenomenon description to mechanism explanation, revealing that security failure does not stem from external bypassing but from the gradual attenuation of internal signals.

As the study shows, when we can precisely locate 'where the problem is', we take the first step towards 'how to fix it'. In today's increasingly complex AI systems, this explainability-driven security research paradigm may be the key path to building more trustworthy AI.