From Cloud to Edge: A Privacy-First Approach for Automated Software Vulnerability Detection Using Large Language Models

This article presents a graduation project by an Indian student team. Addressing the limitations of traditional SAST tools and the privacy risks of using LLMs in the cloud, the team proposes a multi-stage framework that balances detection capability and privacy protection. By comparing Google Gemini's cloud API with a locally quantized Llama 3 model and optimizing with prompt engineering, it achieves local vulnerability detection with a 96% recall rate while ensuring code privacy. The project also includes an interactive Streamlit interface, providing a practical solution for enterprises and learners.

Traditional SAST tools have issues like high false positive rates and lack of semantic understanding, making it difficult to detect complex logical vulnerabilities. While LLMs can identify subtle vulnerability patterns, using them in the cloud poses risks of code privacy and intellectual property leakage. Core question: Can LLMs be run on local hardware to balance privacy and detection capability?

The three-stage framework includes: 1. Cloud Baseline (Google Gemini 2.5 Flash API zero-shot inference to establish performance benchmarks); 2. Local Deployment (Meta Llama3 8B model, 4-bit quantization, run on NVIDIA RTX3060 12GB via Ollama); 3. Prompt Engineering Optimization (zero-shot, role-playing, few-shot prompts—few-shot being the most effective).

The dataset is built based on CodeXGLUE, covering web application vulnerabilities (SQLi, XSS in Python/PHP) and system-level vulnerabilities (buffer overflow, memory leak in C/C++). Key results: Local model achieves a 96% recall rate, protects code privacy, has no API costs, and low latency suitable for CI/CD.

The interactive Streamlit interface supports pasting code, selecting prompt strategies, viewing analysis reports, and comparing results. Engineering details: Hardware requirement of NVIDIA GPU with 12GB VRAM, dependency management (requirements.txt), and modular design for easy experimentation.

Limitations include hardware barriers (RTX3060 is not widely available), high false positive rates, limited coverage of vulnerability types, and manual model updates. Future directions: More aggressive quantization (e.g., GGUF Q4_K_M), secondary filtering mechanisms, expanding vulnerability types, and establishing version management processes.

Implications: Enterprises can balance AI capability and privacy through local deployment + prompt engineering; prompt engineering can bridge the gap in model size; the project demonstrates a complete research process and has educational value.