# RepoReasoner: An Automated Benchmark Framework for Evaluating Large Language Models' Reasoning Capabilities at the Code Repository Level

> An automated benchmark system for evaluating large language models' reasoning capabilities at the granularity of real code repositories, supporting two tasks—output prediction and call chain prediction—and filling the gap in existing benchmarks regarding code granularity.

- 板块: [Openclaw Llm](https://www.zingnex.cn/en/forum/board/openclaw-llm)
- 发布时间: 2026-04-08T09:12:16.000Z
- 最近活动: 2026-04-08T09:18:36.509Z
- 热度: 155.9
- 关键词: 代码推理, 基准测试, 大语言模型, 软件工程, 代码理解, 自动化评估
- 页面链接: https://www.zingnex.cn/en/forum/thread/reporeasoner
- Canonical: https://www.zingnex.cn/forum/thread/reporeasoner
- Markdown 来源: floors_fallback

---

## RepoReasoner Framework Guide: An Automated Evaluation Benchmark for Repository-Level Code Reasoning Capabilities

RepoReasoner is an automated benchmark framework for evaluating large language models' reasoning capabilities at the granularity of real code repositories, filling the gap in existing function-level code evaluation benchmarks. The framework supports two core tasks—output prediction and call chain prediction—evaluating models' code understanding capabilities in scenarios close to real development from both micro and macro dimensions.

## Background: Granularity Limitations of Existing Code Evaluation Benchmarks

Current benchmarks for evaluating large language models' code capabilities mainly focus on the function level, ignoring the complex repository-level dependencies across multiple files and modules in real development. To fill this gap, the DeepSoftwareAnalytics team developed RepoReasoner, which automatically generates test instances from real open-source Python repositories.

## Core Task Design: Dual Evaluation from Micro and Macro Perspectives

RepoReasoner designs two repository-level reasoning tasks:
1. **Output Prediction Task**: Given masked code snippets and context files, predict the correct output of the masked assertion statement, testing the ability to reason about variable states and execution paths;
2. **Call Chain Prediction Task**: Given a test file, predict the list of other source files called during test execution, focusing on macro-level code dependency understanding.

## Automated Pipeline: From Repository Selection to Benchmark Generation

RepoReasoner's automated benchmark construction pipeline includes four stages:
1. **Repository Selection and Filtering**: Select open-source Python repositories with sufficient complexity and test coverage;
2. **Execution-Based Filtering**: Validate repositories in a containerized environment and collect dynamic runtime information to ensure the reliability of reference answers;
3. **Semantic Data Rewriting**: Generate semantically equivalent but syntactically different code variants to enhance dataset robustness;
4. **Instance Collection and Organization**: Extract and filter potential instances from test files to form the final evaluation dataset.

## Flexible Model Integration Support

RepoReasoner supports multiple model evaluation methods:
- Compatible with OpenAI API interface, allowing access to commercial models like GPT-4 and Claude;
- Supports loading local models via Hugging Face to meet privacy requirements;
- Integrates BM25 retrieval mechanism to support retrieval-augmented context generation.

## Application Value and Significance

RepoReasoner's value includes:
1. Provides evaluation standards close to real development scenarios to identify excellent models;
2. The automation feature supports continuous evaluation of new models, quickly obtaining the latest performance data;
3. Reveals gaps in repository-level reasoning capabilities, pointing out directions for model improvement.

## Quick Start Guide

Steps to use RepoReasoner:
1. Prepare a Python 3.8+ environment and Docker container runtime;
2. Install dependencies and place the target Python repository in the specified directory;
3. Configure API keys (for API models) or load local models (Hugging Face);
4. Run the output prediction or call chain prediction script; results are saved automatically.

## Conclusion: A Significant Advance in Code Intelligence Evaluation

RepoReasoner expands the evaluation of large language models' code capabilities from the function level to the repository level, providing new perspectives and tools for understanding and improving models' performance in real development scenarios. As code intelligence technology evolves, such refined evaluation frameworks will drive technological progress.