Rt-LRM: A Red Teaming Framework for Large Reasoning Models

The Rt-LRM (Red Teaming Large Reasoning Models) project, jointly launched by East China Normal University, Tsinghua University Shenzhen International Graduate School, and other institutions, provides a comprehensive red teaming test toolkit for large reasoning models, covering three key dimensions: authenticity, security, and efficiency, to help researchers systematically evaluate model performance in adversarial scenarios.

As the reasoning capabilities of large language models improve, evaluating their robustness against malicious attacks has become a critical topic in AI security. Large Reasoning Models (LRMs) enhance complex task-solving abilities through chain-of-thought, but they also introduce new security risks: harmful outputs under adversarial prompts, sensitive information leakage, or sudden efficiency drops. Traditional security assessment methods focus on a single dimension and struggle to fully capture model vulnerabilities. Rt-LRM fills this gap by constructing a systematic evaluation framework covering three core dimensions: authenticity, security, and efficiency.

Rt-LRM adopts a modular batch inference and evaluation process, supporting local Hugging Face model deployment and remote OpenAI-compatible API inference modes. It focuses on three types of attacks:

Authenticity Dimension

• CPT (Contrastive Prompt Testing): Tests factual consistency
• TruthEval Dataset: Comprehensive factual evaluation

Security Dimension

• H-CoT (Harmful Chain-of-Thought): Tests harmful reasoning paths
• Attack_600: 600 multi-turn dialogue attack samples simulating progressive induction attacks

Efficiency Dimension

• CatAttack: Tests computational resource consumption
• Overthinking Attack: Evaluates response delays caused by excessive reasoning

These tests help identify model weaknesses in knowledge boundaries, security protection, resource efficiency, and other areas.

Rt-LRM provides multi-dimensional quantitative metrics:

• Attack Success Rate (ASR): Determines attack success via LLM-based security audits, identifying security violations, information leakage, or harmful content generation
• Toxicity Score: Integrates Perspective API to detect output toxicity, quantifying offensiveness, hate speech, etc.
• Accuracy Evaluation: Compares model outputs with standard answers in CPT/TruthEval tests
• Overthinking Rate: Identifies unnecessary excessive reasoning by analyzing token counts
• Timeout Statistics: Records the proportion of queries with response times exceeding thresholds (e.g., 180 seconds) to evaluate stability

These metrics help researchers comprehensively measure model performance in adversarial scenarios.

Rt-LRM supports Python 3.10 environments and depends on core libraries such as pandas, openpyxl, tqdm, openai, torch, and transformers. The framework design considers multiple deployment scenarios:

• Local Inference: Loads models via Hugging Face Transformers, supporting NPU acceleration
• Remote API: Compatible with OpenAI-format chat APIs, facilitating testing of closed-source commercial models

Evaluation scripts use a layered design, with independent evaluation modules for each dimension that can be flexibly combined.

Rt-LRM provides important infrastructure for the AI security community:

• Model Developers: A systematic diagnostic tool to identify potential risks before release
• Security Researchers: A standardized evaluation benchmark for fair comparisons between models
• Policy Makers: Quantitative metrics provide technical basis for AI governance

The project reminds users to pay attention to ethical and legal boundaries: some datasets contain sensitive requests and can only be used in legal and ethical research environments.

Rt-LRM is an important advancement in the field of large reasoning model security assessment. It integrates the three dimensions of authenticity, security, and efficiency into a unified testing framework, providing a technical foundation for building more reliable and secure AI systems. As the capabilities of reasoning models expand, such red teaming tools will play an increasingly important role in the AI security ecosystem.