LeetGPTSolver: An Open-Source Benchmark for Systematically Evaluating Large Language Models' Algorithmic Problem-Solving Capabilities

LeetGPTSolver is an open-source benchmark project focused on evaluating the performance of large language models (LLMs) in LeetCode algorithm challenges. Through standardized testing processes, it assesses LLMs' code generation, debugging, and problem-solving capabilities, aiming to provide objective data support for technical teams in model selection, researchers in understanding model capability boundaries, and job seekers in evaluating the feasibility of AI-assisted learning. This project focuses on algorithm competition scenarios, placing higher demands on model reasoning ability and code accuracy.

Large language models have demonstrated remarkable capabilities in code generation, and AI programming assistants are changing the software development model. However, in algorithm interview scenarios, the differences in success rates, code quality, and time complexity performance of different models when solving LeetCode problems are not yet clear. These issues have important reference value for technical teams in selecting AI tools, researchers in understanding model boundaries, and job seekers in evaluating the feasibility of AI-assisted learning—thus the LeetGPTSolver project was born.

LeetGPTSolver is an open-source benchmark testing framework focused on evaluating LLMs' performance on LeetCode algorithm problems. Its evaluation framework design includes:

1. Problem Library Construction: Covers classic algorithm categories such as arrays and strings, with difficulty levels from Easy to Hard, and is equipped with standard test cases (including boundary conditions and extreme inputs);
2. Model Calling and Code Generation: Supports integration with GPT, Claude, Gemini, and open-source models (e.g., Llama), with a unified API interface and optimized prompts (including few-shot examples);
3. Automated Test Execution: Automatically compiles and executes generated code, checks correctness, execution time, and memory usage, and also analyzes code quality (lines of code, cyclomatic complexity, etc.);
4. Result Statistics and Visualization: Generates detailed reports (overall pass rate, performance across different difficulty levels/algorithm categories) and supports output in JSON, Markdown tables, and visual charts.

Core evaluation dimensions include:

1. Problem-Solving Success Rate: Counts the pass rate of various types of problems, broken down by difficulty level and algorithm type, to reveal model strengths and weaknesses;
2. Code Execution Efficiency: Records the running time of passing solutions and compares it with the theoretical complexity of the optimal solution;
3. Code Quality and Readability: Evaluates code style, comment quality, variable naming, etc., through static analysis;
4. Prompt Sensitivity: Compares performance differences between zero-shot and few-shot prompts, as well as between detailed and concise prompts.

Practical application value is reflected in:

1. Model Selection Reference: Provides objective basis for technical teams, reflecting the true level of models' code reasoning ability;
2. Model Capability Research: Helps researchers analyze LLMs' code understanding and generation mechanisms, and discover capability boundaries and improvement directions;
3. Interview Preparation Assistance: Job seekers can understand the boundaries of AI-assisted problem-solving and allocate learning time efficiently;
4. Educational Scenario Application: Teachers can design reasonable homework and exam formats to ensure students master algorithmic thinking.

Technical Implementation Highlights: Developed in Python, uses Docker for code execution environment isolation, pytest as the testing framework, and matplotlib/pandas for visualization; plugin-based design—adding new models only requires implementing standard interfaces, and the evaluation process is highly configurable. Limitations: LeetCode algorithm problems only represent one aspect of programming ability and cannot cover aspects such as maintainability and architecture design in real-world development.

Future plans include expanding the evaluation scope to cover more work-relevant scenarios such as system design problems, code review tasks, and bug fixes in real open-source projects. We also welcome the community to contribute more problems and model support to jointly improve this open-source benchmark.