LLM-testing: A Systematic Evaluation Methodology for Large Language Models in Practical Software Development

LLM-testing is an open-source evaluation framework focused on assessing the performance of large language models (LLMs) in real-world software development scenarios. It aims to establish an evaluation system that is close to software engineering practices, helping developers understand the strengths and weaknesses of different models in real work scenarios, providing a reference for selecting and optimizing AI coding assistants, and addressing the significant gap between existing lab evaluation scores and actual user experience.

Current LLM evaluations have a significant gap between lab environments and real-world development scenarios: Academic benchmarks (such as GLUE and HumanEval) use carefully cleaned datasets with clear problem boundaries, but real project requirements are ambiguous, change frequently, and rely on large amounts of context; Evaluations only focus on code correctness, ignoring engineering dimensions like maintainability, performance, and security; Moreover, evaluations are one-time generation tasks, while actual development is an iterative process (including debugging, refactoring, etc.). The LLM-testing project was thus born to bridge this gap.

LLM-testing is based on the design philosophy of "from practice, to practice", and designs evaluation tasks for key challenges in software development:

1. Requirement Understanding and Clarification: Evaluate the model's ability to identify ambiguities, propose hypotheses, and proactively clarify vague requirements;
2. Code Generation and Context Integration: Test the model's ability to generate code that maintains consistent architecture and style within an existing codebase;
3. Debugging and Bug Fixing: Assess the model's ability to locate the root cause of bugs, propose fixes, and verify their effectiveness;
4. Code Refactoring and Optimization: Test the model's ability to improve code structure, performance, and maintainability;
5. Security and Best Practices: Check whether generated code has common vulnerabilities (e.g., SQL injection) and adheres to language best practices.

The technical implementation of LLM-testing includes:

1. Test Case Collection: Mix real issues/PRs from open-source projects (desensitized and simplified) with manually constructed cases; each case includes requirements, output standards, and automated evaluation scripts;
2. Objectification of Evaluation Criteria: Verify correctness via unit tests, check norms/complexity using static analysis tools, establish quality benchmarks through blind human reviews, and use scoring models for automatic evaluation in some dimensions;
3. Standardization of Model Interfaces: A unified API supports calls to multiple models (OpenAI API, local deployment, etc.), and generation configurations are controlled to reduce randomness;
4. Result Visualization: Generate detailed reports including score comparisons, case studies, and statistical tests.

Key patterns revealed by LLM-testing:

1. Non-linear Relationship with Model Size: Medium-sized models (7B-13B) are close to large models in basic tasks, but large models have obvious advantages in complex reasoning/long context tasks;
2. Significant Impact of Prompt Engineering: Clear context, output format requirements, and few-shot examples can greatly improve model performance;
3. Value of Domain-Specific Fine-Tuning: General-purpose models perform worse than specially fine-tuned models in specific tech stacks;
4. Iterative Interaction is Better: Allowing models to iteratively modify based on feedback is more effective than one-time generation.

LLM-testing provides references for different roles:

• Individual Developers: Choose the appropriate AI coding assistant based on your tech stack and tasks;
• Technical Teams: Use the evaluation framework for due diligence before introducing AI tools, and estimate the model's performance and risks in your own scenarios;
• Model Developers: Take real engineering scenarios as optimization targets, and avoid overfitting to academic evaluations.

Limitations of LLM-testing: It does not cover the entire lifecycle stages such as requirement analysis and architecture design, and evaluation cases are limited by public data. Future directions: Expand to more languages/paradigms, introduce human-machine collaboration evaluations, establish continuously updated benchmarks, and explore multi-modal evaluations (UI design, database schema, etc.).