DeepSeek Security Audit System: An Automated Security Testing Framework for Large Language Models

Researchers from Moscow State Linguistic University have developed an automated security audit system for Large Language Models (LLMs), using DeepSeek as a case study. This system covers 27 attack vectors, over 80 test prompts, integrates advanced features like semantic analysis, sentiment analysis, hallucination detection, and Bayesian risk assessment, adopts a modular design, supports multilingual testing, provides adaptive testing strategies and visual reports, and offers a comprehensive automated solution for LLM security assessment.

With the widespread application of LLMs in various industries, their security issues have become increasingly prominent (e.g., harmful outputs, information leakage, malicious exploitation). Traditional security testing methods struggle to handle the complexity and uncertainty of LLMs, so researchers developed this automated security audit system specifically for LLMs.

System Architecture

The audit system is a comprehensive intelligent testing platform with a modular design including:

• Attack Vector Library: Covers 27 attack vectors, spanning the entire lifecycle of ML systems
• Test Prompt Set: Over 80 carefully designed test prompts
• STRIDE-AI Classification Framework: Structured threat classification analysis

Core Functions

• Semantic Analysis: Identifies refusal responses, information leakage, and evasive behaviors
• Sentiment Analysis: Quantitatively evaluates output sentiment tendency (from -1 to 1)
• Multilingual Support: Russian, English, Chinese, French, German
• Intelligent Detection: ML pre-detection, hallucination detector (identifies factual errors/logical contradictions), time-series analysis

Adaptive Testing Strategy

• Entropy-based Threat Selection: Uses Shannon entropy to adaptively select the next test threat, maximizing coverage and avoiding redundancy
• Intelligent Stop Mechanism: Automatically stops when test results stabilize (based on statistical confidence)

Risk Assessment

Uses Bayesian methods combining prior probabilities and observed data to dynamically assess vulnerability risks, which better reflects the real security situation than binary judgments.

Anti-Detection Measures

• Token Pool Management: 6 tokens, handling 900 requests per day
• Automatic Switching: Automatically switches tokens when restricted
• User-Agent Rotation: 14 identifiers
• Behavior Simulation: Simulates real users with delays like "coffee breaks"

Statistical Confidence

Uses Wilson interval method to calculate 95% confidence intervals, combined with Bayesian estimation to ensure reliable results.

Visualization System

• 7 Interactive Charts: Vulnerability distribution heatmap, attack success rate trend, risk level matrix, multilingual comparison, etc.
• Real-time Dashboard: Live Dashboard monitors test progress

Reports and Notifications

• Automatically generates academically standardized reports (including dynamic risk assessment tables)
• Telegram Notifications: Instantly pushes alerts for critical security issues

Application Value

• Research Community: Open-source tool fills the gap in automated security testing, modular and easy to extend
• Industry: Enterprise self-assessment tool to identify and fix vulnerabilities, reducing compliance risks
• Methodology: Combines traditional cybersecurity with ML, proposes a generative AI security assessment framework

Limitations and Future

• Currently only targets DeepSeek; future expansion to more LLM platforms
• Attack vector library needs to be updated with adversarial technologies
• Hallucination detection module needs improvement in complex reasoning chains

The DeepSeek Security Audit System integrates multiple analysis technologies, adaptive strategies, and visual reports to provide a comprehensive automated solution for LLM security assessment. As an open-source project, it has important reference value for AI security developers and researchers, representing a key advancement in the field of LLM security testing.