Cross-Language Code Clone Detection via Knowledge Distillation from DeepSeek-R1: Empowering Small Models with Large Model Reasoning Capabilities

This study transfers the strong reasoning capabilities of DeepSeek-R1 to small open-source models like Phi3 and Qwen-Coder via knowledge distillation. Combined with LoRA fine-tuning and response stabilization techniques, it significantly enhances the reliability and prediction performance of small models in cross-language code clone detection tasks such as Python-Java and Rust-Java.

Challenges of Cross-Language Clone Detection

• Syntax differences: Vastly different syntax structures across languages (e.g., Python list comprehensions vs Java Stream API)
• Idiom differences: Unique programming idioms in each language (e.g., C++ pointer operations vs Rust ownership system)
• Standard library differences: Large variations in design and API style, making surface matching ineffective

Limitations of Existing Methods

• Traditional AST/PDG methods: Require building parsers for each language, leading to high maintenance costs
• Large Language Models (LLMs): High cost, poor reproducibility, privacy risks, and unstable output formats
• Small open-source models: Weak ability to follow reasoning-oriented prompts, with outputs hard to map to binary clone labels

Core Ideas

• Teacher model: DeepSeek-R1 (strong code understanding and reasoning capabilities)
• Student models: Phi3, Qwen-Coder (small open-source, suitable for local deployment)
• Transfer method: LoRA efficient fine-tuning (only train a small number of low-rank matrix parameters)

Response Stabilization Techniques

1. Forced conclusion prompting: Explicitly require the model to output conclusions in a fixed format (e.g., "Conclusion: Is clone / Not clone")
2. Binary classification head: Add a classification head to the model, converting generation tasks into classification tasks for controllable output and faster reasoning
3. Contrastive classification head: Introduce contrastive learning to maximize similarity between clone pairs and minimize similarity between non-clone pairs

Technical Details

• LoRA configuration: Rank 8/16, alpha 16/32, target modules are attention layer Q/K/V/O projection matrices, dropout 0.05-0.1
• Training strategy: Two stages (warm-up: fine-tuning on general code understanding data; distillation: training on DeepSeek-R1 reasoning data)
• Loss functions: Cross-entropy for generative tasks, binary cross-entropy for classification heads, InfoNCE loss for contrastive variants

Testing Scenarios

Covers 4 cross-language code pairs: Python↔Java, Rust↔Java, Rust↔Python, Rust↔Ruby

Key Findings

• Reliability improvement: Distilled small models achieve near-100% response rate (stable output)
• Performance improvement: Significant increase in prediction accuracy, especially in distribution shift scenarios
• Efficiency optimization: Classification head variants have drastically reduced reasoning time, suitable for deployment

Comparative Analysis

• Knowledge distillation outperforms pure prompt engineering
• Classification head output balances efficiency and effectiveness better than generative output
• Multi-language joint training yields better performance

1. Reduce review costs: Support scenarios like code migration, cross-language plagiarism detection, and vulnerability propagation analysis
2. Protect code privacy: Local deployment of small models avoids sensitive code leakage, meeting compliance requirements
3. Customizability: Open-source models can be further fine-tuned for specific domains (e.g., company code style, industry norms)

Current Limitations

• Limited language coverage (only Python, Java, Rust, Ruby)
• Weak detection capability for complex near-miss clones
• Reasoning efficiency in ultra-large codebases needs optimization

Future Directions

• Expand to mainstream languages like C++, Go, and JavaScript
• Combine static program analysis techniques to enhance semantic signals
• Integrate code text and program graphs (AST, CFG, PDG) for multi-modal detection
• Design active learning strategies to reduce distillation costs