Pen-Strategist: An LLM Reasoning Framework for Penetration Testing with 87% Improvement in Strategy Generation Performance

Researchers propose the Pen-Strategist framework, which uses a domain-specific reasoning model and a semantic classifier to improve the performance of LLMs in penetration testing strategy generation tasks by 87% and increase subtask completion rate by 47.5%. This framework addresses issues such as insufficient strategy formulation and domain reasoning in existing LLM penetration testing tools, providing a new solution for automated security testing.

There is a severe global shortage of cybersecurity talents, and traditional defense systems struggle to cope with complex threats. Existing LLM penetration testing frameworks (e.g., PentestGPT) face issues like insufficient strategy formulation, domain-specific reasoning, and tool selection. The general knowledge of LLMs cannot meet the deep reasoning requirements of penetration testing, leading to superficial generated strategies.

The framework includes two core modules:

1. Domain-Specific Reasoning Model: Based on Qwen-3-14B, fine-tuned via reinforcement learning to understand penetration testing contexts and generate logically consistent attack strategies;
2. Semantic Classifier: A CNN architecture that converts high-level strategies into executable steps, solving the "last mile" problem from strategy to execution.

A penetration testing reasoning dataset was constructed, including logical explanations of strategy derivation (complete reasoning chains) and logical explanations of step selection (decision-making basis). Qwen-3-14B was fine-tuned using reinforcement learning, with the reward mechanism considering dimensions such as strategy completeness, feasibility, and security.

• Strategy generation performance: 87% improvement over the baseline;
• Subtask completion rate: 47.5% increase after integration into existing frameworks, exceeding the GPT-5 baseline;
• CTFKnow benchmark: 18% performance improvement;
• Step prediction: CNN classifier accuracy is 28% higher than commercial LLMs;
• Human evaluation: Strategy quality is better than Claude-4.6-Sonnet.

1. Domain-specific reasoning: General LLMs need domain training to enhance their ability for professional tasks;
2. Separation of strategy and execution: Separating high-level strategies from specific steps improves reliability and interpretability;
3. Value of reinforcement learning: Helps models learn deep reasoning capabilities beyond simple pattern matching.

Extend the architecture to security domains such as vulnerability discovery and malware analysis; integrate multimodal technologies like network traffic analysis and system log understanding to further enhance the intelligence level of automated penetration testing.