AegisRT: A Native Python Security Testing and Auditing Framework for Large Language Models

AegisRT is a native Python framework designed specifically for Large Language Model (LLM) security. It systematically covers the OWASP LLM Top 10 risks and provides developers and security teams with scalable model security testing and auditing capabilities. It aims to address the shortcomings of traditional software security testing tools in LLM scenarios and fill the industry gap.

The widespread application of LLMs brings security risks such as prompt injection and sensitive information leakage. Traditional static analysis cannot capture dynamic interactions, and dynamic testing lacks systematic methods targeting LLM-specific attack surfaces. The OWASP LLM Top 10 risk list provides a reference, but how to translate it into executable test cases remains a challenge.

AegisRT is a native Python framework that seamlessly integrates with the existing Python ecosystem. Its core design philosophy is "auditability": it not only identifies issues but also provides audit trails to help understand the root cause, impact, and remediation path, making it suitable for enterprise-level compliance auditing and risk management needs.

AegisRT systematically covers the OWASP LLM Top 10 risks:

• Prompt Injection: Provides direct/indirect injection and jailbreak attack templates to evaluate defense capabilities;
• Sensitive Information Leakage: Evaluated through information extraction attacks and membership inference tests;
• Supply Chain Contamination: Audits the security of dependencies and detects malicious code injection;
• Data Poisoning and Model Theft: Evaluates the robustness against adversarial examples and the vulnerability of model extraction;
• Over-Agency: Tests the model's ability to reject unauthorized requests.

AegisRT adopts a modular and scalable design:

• Attack Vector Library: Built-in rich attack templates from real scenarios;
• Evaluation Engine: Supports custom scoring criteria and threshold adjustments;
• Report Generator: Automatically generates multi-format reports containing vulnerabilities, risk levels, reproduction steps, and remediation suggestions;
• Plugin System: Open architecture allows the community to contribute new attack vectors and evaluation methods.

AegisRT is applicable in the following scenarios:

• Security Shift-Left in Development Phase: Baseline testing before integration to detect vulnerabilities early and reduce costs;
• CI/CD Integration: Trigger security scans in automated pipelines;
• Third-Party Model Evaluation: Verify the security commitments of external models/APIs;
• Red Team Exercises: Simulate attackers to test defense effectiveness.

Possible future development directions for AegisRT: Combine with model interpretability technology to deepen vulnerability root cause analysis; support multi-modal model security testing; establish industry-standard LLM security benchmark test sets.