Self-ReSET: Enabling Large Language Models to Self-Recover from Dangerous Reasoning Trajectories

Self-ReSET is a pure reinforcement learning framework whose core innovation is enabling models to learn recovery capabilities from their own safety error trajectories, significantly enhancing robustness against adversarial attacks (especially out-of-distribution jailbreak prompts) while preserving general capabilities. This framework uses dynamic on-policy reasoning trajectory training to eliminate the gap between static data and dynamic behavior, providing a new direction for the safety alignment of reasoning models.

Large language reasoning models (LRMs) have self-correction capabilities in general domains, but struggle to recover from unsafe reasoning trajectories when facing adversarial attacks. Attackers can induce models into dangerous paths via carefully designed prompts. Existing alignment methods (e.g., fine-tuning with expert data) have a gap between static training data and dynamic reasoning trajectories, making it hard to cover the model's vast generation space and unable to let models recover from their own failures.

Core Ideas

Self-ReSET (Self-Recovering from Error States via Reinforcement Learning Training) is a pure reinforcement learning framework that enables models to learn recovery capabilities from their own safety error trajectories. Workflow:

1. Generate error trajectories: Produce unsafe reasoning trajectories during training
2. Trajectory reuse: Use error trajectories as initial states for reinforcement learning
3. Recovery learning: Learn to return to safe paths from error states

Technical Details

• Error state recognition: Identify harmful content generation, successful jailbreaks, outputs violating safety guidelines
• Trajectory as initial state: Implement mid-flight correction
• Pure reinforcement learning training: No reliance on expert data or supervised fine-tuning

Comparison with Existing Methods

Method Type	Training Data	Main Limitations
Supervised Fine-tuning (SFT)	Static expert data	Mismatch between data and model behavior distribution
Adversarial Training	Pre-generated adversarial samples	Hard to cover all attack variants
Self-ReSET	Dynamic on-policy trajectories	Requires more training steps

Enhanced Adversarial Robustness

Self-ReSET significantly enhances robustness against adversarial attacks, especially defense against out-of-distribution (OOD) jailbreak prompts, using general recovery strategies to handle unknown attacks.

Preservation of General Capabilities

While improving safety benchmarks, it maintains performance on general tasks without obvious over-defense phenomena.

Data Utilization Efficiency

Training data comes from the model's own reasoning process; each round generates new policy-consistent data, forming a positive cycle.

Self-ReSET cultivates the model's self-recovery mode:

1. Error recognition: Identify potential problem states during reasoning
2. Path correction: Proactively adjust reasoning direction after detecting danger signals
3. Safe regression: Return to safe paths

This ability is similar to human metacognition (monitoring and regulating one's own thinking). Additionally, the model can recognize and recover from unsafe intermediate error states—even early deviations can be corrected later.

Self-ReSET provides new ideas for safety alignment of reasoning models:

• From passive defense to active recovery: Endow models with the ability to actively recover from errors
• Dynamic adaptation instead of static rules: Learn adaptive safety strategies via reinforcement learning
• Scalable training paradigm: Pure reinforcement learning method is easy to extend to new model architectures and safety scenarios

Self-ReSET still has areas for exploration:

• Training stability: Pure reinforcement learning requires more refined hyperparameter tuning
• Error state definition: How to more accurately define and identify unsafe reasoning states
• Multilingual and cross-cultural: Effectiveness in different language and cultural backgrounds needs verification