VCAO: A Game Theory-Based Verifier-Centered Agent Orchestration System for Strategic Discovery of OS Vulnerabilities

This paper proposes the VCAO (Verifier-Centered Agentic Orchestration) framework, which models OS vulnerability discovery as a Bayesian Stackelberg search game. It dynamically allocates analysis budgets via a large reasoning model orchestrator, combining verifiers like static analysis, fuzz testing, and memory detectors to achieve efficient automated vulnerability mining. The system performs excellently in Linux kernel vulnerability discovery, significantly outperforming existing methods.

The OS kernel is the cornerstone of computing systems, but its large code size (e.g., Linux kernel has over 30 million lines) makes vulnerability discovery difficult. Traditional methods (static analysis, fuzz testing, symbolic execution) each have limitations: static analysis has high false positives, fuzz testing has limited coverage, and symbolic execution is constrained by path explosion. The rise of large language models brings new possibilities for vulnerability discovery, but it requires intelligent orchestration of heterogeneous tools to work together—thus VCAO was born.

VCAO models vulnerability discovery as a repeated Bayesian Stackelberg search game: the defender (system) allocates budgets, the attacker looks for missed vulnerabilities, and Bayesian beliefs are updated over multiple rounds to adjust strategies. The system uses a six-layer architecture: Surface Mapping Layer (code structure analysis), Intra-Kernel Attack Graph Construction Layer (data flow/control flow dependencies), Game-Theoretic File/Function Ranking Layer (core decision-making, MILP for optimal allocation), Parallel Executor Agent Layer (manages verifier tool instances), Cascaded Verification Layer (cross-validation to reduce false positives), and Security Governance Layer (system monitoring).

Experiments were conducted on 5 Linux kernel subsystems: 1. Historical CVE Reproduction: Rediscovered 847 known CVEs; 2. Real-Time Discovery: Found 0-day vulnerabilities in the latest kernel snapshot. Performance comparison: Per unit budget, it found 2.7x more valid vulnerabilities than coverage-guided fuzz testing, 1.9x more than static analysis, and 1.4x more than non-game-theoretic multi-agent pipelines; false positive rate was reduced by 68%.

Key factors for VCAO's success: 1. Dynamic Adaptability: Adjust strategies based on real-time results; 2. Heterogeneous Tool Collaboration: Optimize complementary tool combinations; 3. Strategic Prioritization: Focus on high-risk components via attack graphs and Bayesian beliefs, ensuring resources are used where they matter most.

The research team open-sourced the simulation framework, synthetic attack graph generator, and evaluation toolchain to facilitate reproduction, extension, and standardized evaluation. Limitations: Currently focused on Linux kernel, attack graphs rely on static analysis, and game parameters need calibration. Future directions: Extend to user-space applications, introduce more verifier tools, and explore collaboration between multiple VCAO instances.

VCAO combines game theory, Bayesian reasoning, and large language models to build an intelligent adaptive vulnerability discovery system. In today's era of severe cybersecurity threats, such automated tools can help security teams quickly locate risks and fix vulnerabilities proactively. AI will play an increasingly important role in the software security field.