DiffCodeGen: A New Test-Time Expansion Method for Code Generation Without Extra LLM Calls

DiffCodeGen proposes a test-time expansion method based on coverage-guided difference analysis. It selects optimal code candidates via dynamic behavior clustering, enabling efficient code generation selection without additional LLM calls, significantly reducing token consumption and inference time. This addresses the core pain points of existing test-time expansion methods, which rely on public test cases and expensive LLM inference.

In recent years, large language models (LLMs) have performed exceptionally well in code generation tasks. However, test-time expansion (exploring large solution spaces during inference to improve generation quality) faces two major challenges: 1. Reliance on public test cases, which often do not exist in real-world scenarios; 2. Some methods require additional LLM calls to evaluate candidates, leading to high token consumption and time delays that limit scalability. These pain points have spurred the innovative solution of DiffCodeGen.

The core innovation is to eliminate the need for additional LLM calls by adopting a dynamic behavior similarity clustering paradigm. The process consists of four stages: 1. Candidate Generation: Generate a large number of candidates via diverse sampling and prompt variants, supporting parallel execution; 2. Input Synthesis: Automatically synthesize inputs using coverage-guided fuzz testing, without pre-stored test cases or LLMs; 3. Dynamic Behavior Capture: Execute candidates on synthesized inputs and capture behavioral features such as execution paths and outputs; 4. Behavior Clustering and Selection: Cluster candidates based on behavioral similarity and select the center point of the largest cluster as the final output (based on the assumption that mainstream behavioral patterns are more likely to be correct).

1. Zero Additional LLM Calls: Candidate selection is based on execution behavior and clustering algorithms, significantly reducing token consumption; 2. Fully Asynchronous Architecture: Parallel candidate generation, independent input synthesis and execution, and batch clustering processing, adapting to the trend of Agentic Coding; 3. Model-Agnostic: Can be paired with any code generation model, and performance is further improved when combined with inference models.

Evaluated across four LLMs of different scales, DiffCodeGen delivers consistent performance improvements; compared with SOTA methods: performance is comparable or even better, while completing tasks with minimal time and token consumption, showing a significant performance-efficiency trade-off advantage.

It opens up a new path for test-time expansion in code generation, proving that high-quality code selection does not require expensive model inference; for developers: lower API call costs, faster response times, and easy integration into development toolchains; in the future, it is expected to become an important component of code agents under the Agentic Coding paradigm, promoting the development of automated programming towards practical and economical directions.