RealBench: Bringing Code Generation Evaluation Back to Real Software Development Scenarios

The new benchmark RealBench introduces UML design diagrams and natural language requirements, bridging the gap between existing code generation benchmarks and real enterprise-level development scenarios, and revealing the capabilities and limitations of LLMs in real software development. Keywords: code generation, LLM, benchmark, software development, UML, enterprise application, AI programming assistant.

Code generation is one of the most compelling applications of large language models (LLMs). However, existing classic benchmarks (such as HumanEval and EvoCodeBench) only require models to generate code based on natural language descriptions, which has a clear gap with the actual workflow in enterprise-level development that is based on structured system design documents or UML diagrams. This leads to current evaluation scores failing to accurately reflect the actual value of code generation technology for software development.

To bridge the gap between existing benchmarks and real scenarios, the research team launched RealBench—a code generation benchmark aligned with real industrial software development practices. Its core innovations include: 1. Dual-input design: Each test case includes both natural language requirements and UML diagrams as system design; 2. Repository-level generation: Requires generating an entire code repository, including multiple interrelated modules; 3. Emphasis on the ability to understand structured inputs such as UML class diagrams and sequence diagrams.

Through systematic evaluation of advanced LLMs, the key capabilities and limitations of current models in real scenarios are revealed: 1. Significant decline in repository-level generation performance: All LLMs show a clear drop in performance when handling repository-level tasks, and the gap between models widens; 2. Strong module recognition ability but poor implementation quality: LLMs can accurately identify modules in UML diagrams and create corresponding files, but the generated code has many syntax errors and logical flaws; 3. Choice of generation strategy is crucial: Small repositories are suitable for the overall generation strategy, while complex repositories are suitable for the module-by-module generation strategy.

The research results of RealBench have implications for AI-assisted software development: 1. New dimension of requirement understanding: Future AI programming assistants need to have the ability to parse UML diagrams and architecture documents; 2. Quality assurance mechanisms: Need to integrate automatic verification, test generation, and code review mechanisms to address errors in generated code; 3. Progressive generation strategy: Choosing the appropriate generation strategy based on project complexity can significantly improve output quality.

The launch of RealBench marks a new stage in code generation evaluation—moving from "toy problems" to "real scenarios". It is of great significance to enterprise users (providing reliable model selection basis), researchers (pointing out the direction of structured input understanding and large-scale project generation), and tool developers (suggesting optimization space for product design). Future research may focus on enhancing the model's ability to parse design documents such as UML, developing specialized training methods for repository-level generation, and building a progressive code generation workflow for human-machine collaboration.

RealBench is not only a new benchmark but also an important reflection on the field of code generation. It reminds us that the ultimate goal of evaluating AI capabilities is to enable it to create real value in the real world. Only when benchmarks align with actual needs can technological progress truly translate into productivity improvement.