Can Large Language Models Deobfuscate Binary Code? A Systematic Analysis and the BinDeObfBench Benchmark

This paper systematically evaluates the performance of Large Language Models (LLMs) on binary deobfuscation tasks by constructing the BinDeObfBench benchmark. Key findings include: reasoning ability and domain expertise are more important than model size; supervised fine-tuning (SFT) for deobfuscation tasks outperforms general pre-training; models with reasoning capabilities exhibit stronger robustness in heavily obfuscated scenarios and better cross-architecture generalization. The release of BinDeObfBench provides a standardized evaluation foundation for LLM-assisted deobfuscation research.

Binary deobfuscation is a core challenge in software security reverse engineering. Traditional methods (rules, pattern matching, symbolic execution) are insufficient against new obfuscation techniques. The ability of LLMs in code understanding and other fields has raised questions about whether they can solve deobfuscation problems, but existing research has limitations: it only focuses on specific obfuscation types or model architectures, lacking systematic comparisons; existing benchmarks cover limited scenarios and are difficult to reflect real-world capability boundaries.

BinDeObfBench is the first comprehensive benchmark for LLM binary deobfuscation. Its design features include: 1. Multi-stage obfuscation coverage: pre-compilation (source code layer), compilation time (compiler IR layer), post-compilation (binary layer); 2. Cross-architecture and optimization levels: covering instruction sets such as x86, ARM, RISC-V, and optimization levels from O0 to O3, ensuring evaluation universality.

Experimental evaluations yielded key findings: 1. Reasoning ability outperforms size: medium-sized but well-trained models may outperform untrained large models; 2. Task fine-tuning is better: SFT models consistently outperform general pre-trained models; 3. Reasoning models are robust: chain-of-thought-based models perform better in heavily obfuscated scenarios and have outstanding cross-architecture generalization; 4. Context learning effect varies: significant improvement for standard models, but limited gain for reasoning models.

Practical recommendations based on the findings: 1. Prioritize domain-specific training: fine-tuning models on deobfuscation datasets, although additional annotation is required, the performance improvement is significant; 2. Emphasize reasoning ability cultivation: use chain-of-thought data, multi-step reasoning supervision, etc., to enhance model reasoning ability; 3. Establish a continuous evaluation mechanism: use BinDeObfBench to regularly track the performance of new models/technologies to cope with the development of obfuscation techniques.

Limitations of the current benchmark: it mainly focuses on pseudocode-level deobfuscation and does not involve original source code recovery; it does not include dynamic obfuscation (such as runtime self-modifying code). Future directions: expand to more architectures (GPU kernels, embedded firmware); add dynamic analysis dimensions; explore human-machine collaboration models; develop more efficient fine-tuning strategies to reduce domain adaptation costs.