Application of Large Language Models in Software Vulnerability Detection: A Systematic Literature Review

This article is a systematic literature review project "Awesome-LLM4SVD" published in TOSEM'26, which comprehensively summarizes the current research status of large language models (LLMs) in the field of software vulnerability detection. The core content includes the limitations of traditional vulnerability detection methods, the unique advantages of LLMs, technical routes of existing research, evaluation challenges, key issues, and future directions, providing an important reference for security researchers and practitioners.

Limitations of Traditional Methods

Traditional vulnerability detection tools rely on predefined rules/pattern matching and have the following problems:

• High cost of rule maintenance (need continuous updates to address new vulnerabilities)
• High false positive rate, consuming analysts' energy
• Difficulty in detecting complex vulnerabilities (cross-file/function calls)
• Lack of contextual semantic understanding

Unique Advantages of LLMs

The capabilities of LLMs in code tasks provide ideas to solve the above problems:

• Deep semantic understanding: can grasp code intent rather than just syntax matching
• Strong generalization ability: identify unseen vulnerability patterns through pre-training
• Natural language interaction: lowers the threshold for security analysts to use
• Context awareness: analyze code relationships across functions/files

Literature Screening and Classification

Follow strict systematic literature review (SLR) methodology, establish clear inclusion/exclusion criteria to ensure comprehensiveness and objectivity.

Analysis Dimensions

Classify and analyze existing research from 5 dimensions:

1. Model Architecture: Types of LLMs used (e.g., GPT series, CodeBERT, GraphCodeBERT, etc.)
2. Application Scenarios: Target programming languages, vulnerability types, code scale
3. Technical Methods: Fine-tuning strategies, prompt engineering, multimodal fusion, etc.
4. Evaluation Methods: Datasets, evaluation metrics, comparison with baseline methods
5. Practical Deployment: Computational resource requirements, inference latency, interpretability

Diversified Technical Routes

Existing research adopts multiple technical routes:

• Code Representation Learning-based: Convert code into graph structures (abstract syntax trees, control flow graphs, etc.), combine graph neural networks with LLMs to identify vulnerabilities
• Sequence-based: Treat code as text sequences and process directly with pre-trained LLMs (simple to implement but may lose structural information)
• Hybrid Methods: Rough screening by traditional tools + fine inspection by LLMs, balancing efficiency and accuracy

Challenges in Evaluation

• Dataset Limitations: Mostly use synthetic data, which differs from real vulnerability distribution
• Ununified Evaluation Metrics: Difficult to compare horizontally
• Inappropriate Baseline Selection: Some comparison baselines are simple, failing to reflect the real gains of LLMs
• Reproducibility Issues: Code/data not fully open-sourced

Technical Challenges

• Large-scale Code Processing: How to efficiently process real codebases with millions of lines
• Vulnerability Localization Accuracy: Need more precise localization of vulnerability positions and types
• Adversarial Robustness: Insufficient research on defending against malicious obfuscation/adversarial examples
• Interpretability Requirements: The "black-box" nature of LLMs makes it difficult to meet analysts' understanding of vulnerability causes

Practical Application Barriers

• False Positive Rate Control: Need to balance false positives and false negatives in safety-critical scenarios
• Computational Resource Requirements: Large LLMs have high inference costs, making deployment in constrained environments difficult
• Integration with Existing Workflows: Need to solve engineering problems in CI/CD and security audit processes

Collaboration Needs of the Research Community

• Establish standardized evaluation benchmarks and datasets
• Open-source more high-quality research code
• Strengthen collaboration between academia and industry
• Focus on engineering challenges of practical deployment (Note: The project has established the "Awesome-LLM4SVD" resource list, which collects relevant papers/datasets/tools.)

Recommendations for Security Practitioners

1. Progressive Introduction: Start with assisting analysts and reducing false positives, do not directly replace traditional tools
2. Continuous Monitoring: Establish model performance monitoring mechanisms to detect drift/degradation in time
3. Human-Machine Collaboration: Design efficient interfaces for analysts to review and confirm model outputs
4. Security Considerations: Evaluate the security of LLM tools themselves to prevent supply chain attacks or data leaks

"Awesome-LLM4SVD" project presents a panoramic view of LLMs in the field of vulnerability detection through a systematic review, showing both great potential and current limitations and challenges.

For researchers: This is a field full of opportunities, and it is necessary to improve model capabilities and the maturity of evaluation methods. For practitioners: It is necessary to rationally understand the technical boundaries and gradually explore and verify in practice to successfully apply LLM technology.