Empirical Study on Large Language Models in Vulnerability Analysis of Automotive Binary Programs

This paper conducts an empirical study on the application of large language models (LLMs) in vulnerability analysis of automotive binary programs, exploring their capabilities, limitations, and practical application prospects in the field of automotive software security. The study finds that while LLMs show potential in vulnerability detection, they have problems such as insufficient cross-architecture generalization and high false positive rates; integrating with traditional static analysis tools can improve detection coverage and accuracy, providing a new path for automotive software security testing.

Research Background and Significance

With the development of intelligent connected vehicles, the security of in-vehicle software has become a focus. Modern cars have a large number of ECUs and complex embedded software; traditional binary vulnerability analysis relies on expert experience and static tools, facing challenges such as low efficiency, high false positives, and difficulty in handling complex code. LLMs have great potential in code understanding and security detection, and their application in automotive binary vulnerability analysis is expected to break through traditional bottlenecks.

Unique Challenges of Automotive Software Security

Automotive embedded systems have characteristics such as heterogeneous architectures (ARM, PowerPC, etc.), many closed-source binary components, special communication protocols (CAN/LIN/FlexRay), and resource constraints. General-purpose tools have limited effectiveness, putting forward special requirements for the application of LLMs.

This study constructs a test dataset for automotive binary programs (open-source firmware + actual vulnerability cases) and designs a multi-dimensional evaluation framework (accuracy, false positive rate, efficiency, cross-architecture generalization, etc.). The experiment selects general code models and security-fine-tuned models; the evaluation tasks cover common vulnerabilities such as buffer overflow and integer overflow, and examines the robustness of binaries with different optimization levels, obfuscation/packing.

The research results show that: security-fine-tuned models perform better than general models in identifying typical memory vulnerabilities; however, their performance declines when handling highly optimized binaries, as compiler optimizations interfere with pattern recognition; cross-architecture generalization ability is limited, with insufficient analysis of rare automotive processors; understanding of automotive-specific protocols and state machine logic is weak, requiring domain knowledge verification.

Main false positive scenarios: misjudging normal boundary checks as vulnerabilities, over-alerting on complex pointer operations, misunderstanding compiler protection code as attack payloads. In terms of interpretability, LLMs can generate natural language reasoning, but there is a "hallucination" phenomenon (explanations do not match code logic), so the credibility of reasoning needs to be improved.

Traditional static tools have clear rules and low false positives, but are difficult to handle unknown vulnerabilities and complex code. LLMs have strong generalization ability and can identify uncovered vulnerability variants, but have high false positives. The hybrid process uses LLMs as a supplementary layer: static tools initially screen for clear vulnerabilities, and LLMs focus on high-complexity suspicious areas. Experiments show that this method can maintain a low false positive rate and improve detection coverage.

Deployment needs to address issues such as computing resource costs, model update and maintenance (to adapt to the evolution of technology stacks), data privacy and intellectual property protection. Future directions: developing lightweight dedicated models, establishing standardized evaluation benchmarks, and exploring deep integration with formal verification.