Attention Vulnerabilities in Large Reasoning Models: A New Paradigm of Reinforcement Learning-based Jailbreak Attacks

Large Reasoning Models (LRMs) such as OpenAI o1/o3 and DeepSeek-R1 demonstrate strong reasoning capabilities through chain-of-thought mechanisms, but exposing their reasoning process introduces new security risks—they are more vulnerable to jailbreak attacks than standard LLMs. The study finds that successful jailbreaks are closely related to attention distribution: harmful tokens receive low attention in the input layer and high attention in the reasoning layer. Based on this, the proposed attention-guided reinforcement learning attack method significantly outperforms existing solutions in success rate, efficiency, and transferability, while also providing new directions for LRM security defense.

Large Reasoning Models (LRMs) outperform traditional LLMs on complex tasks by generating structured step-by-step reasoning content (chain-of-thought). However, the design of exposing internal reasoning processes makes LRMs more susceptible to jailbreak attacks—being induced to generate harmful content.

The study finds that the attention pattern of successful jailbreak attacks has dual characteristics: 1. Input layer attention suppression (harmful tokens have low attention weights in input prompts); 2. Reasoning layer attention enhancement (the same harmful tokens have high attention in reasoning content). This reveals blind spots in LRM security mechanisms and provides new dimensions for attack design, defense improvement, and model architecture reflection.

Based on the attention findings, the study proposes a novel jailbreak method, whose core is integrating attention signals into the reinforcement learning reward function:

1. Attention-aware reward function: Minimize input attention + Maximize reasoning attention;
2. Diverse persuasion strategy space: Role-playing, scenario construction, logical confusion, progressive induction;
3. Strategy optimization and transfer: Learn transferable strategies via the PPO algorithm; strategies trained on open-source models can be transferred to closed-source models.

Experiments were validated on 3 evaluation benchmarks and 5 models:

• Attack Success Rate (ASR): 15-25% higher than gradient methods, 30-40% higher than template methods, and 10-15% higher than pure RL methods;
• Efficiency: Fewer average queries, faster convergence, and controllable computational overhead;
• Transferability: Effective from open-source to open-source/closed-source/cross-architecture models, indicating that LRMs share attention vulnerabilities.

Existing security mechanisms (such as RLHF) do not fully consider the attack surface exposed by the reasoning process. Potential defense directions:

1. Attention monitoring: Identify harmful tokens with abnormally low attention in the input layer;
2. Reasoning process review: Set up security checkpoints during the reasoning phase;
3. Adversarial training: Incorporate attention-guided attacks to improve robustness;
4. Reasoning process isolation: Separate internal states from user outputs or filter reasoning content.

The study follows the principle of responsible disclosure (communicating with vendors, for research purposes only) to enhance security awareness. Industry trends:

• Trade-off between capability and security: Improved chain-of-thought capabilities come with security costs;
• Attack evolution: From prompt engineering to adaptive reinforcement learning attacks;
• Security gap between open-source and closed-source models: Vulnerabilities in open-source models easily affect closed-source models, highlighting the value of security research in the open-source community.