DABench-RLM-Eval: A Framework for Evaluating Data Analysis Capabilities of DSPy Recursive Language Models

DABench-RLM-Eval is a benchmark framework specifically designed to evaluate the performance of DSPy Recursive Language Models (RLMs) on data analysis tasks. It supports automated scoring and iterative code evaluation, helping developers quantify RLMs' capabilities in tabular data processing scenarios. This framework addresses key challenges in RLM evaluation, including diverse iterative execution paths, dependencies on code execution environments, complex result validation, and high reproducibility requirements, providing a complete evaluation pipeline.

With the breakthroughs of large language models in code generation, Recursive Language Models (RLMs) adopt an iterative generate-execute-feedback loop, enabling them to handle complex logic and multi-step tasks. DSPy is a declarative programming framework launched by Stanford, which optimizes RLMs' performance in multi-turn reasoning and tool calling scenarios (e.g., data analysis). However, evaluating RLMs faces four major challenges:

1. Diverse iterative execution paths
2. Dependence on secure sandbox environments for code execution
3. Complex result validation (numerical tolerance, table structure matching)
4. High reproducibility requirements

Core Capabilities

1. Integrates diverse data analysis tasks from DABench
2. Optimized specifically for DSPy RLMs
3. Intelligent automated scoring system
4. Supports multi-round iterative evaluation
5. Native Windows support

Technical Architecture

1. Task Design: Covers 6 types of tasks including table query, statistical analysis, data cleaning, etc. Each task includes datasets, problem descriptions, scoring criteria, and reference solutions
2. Recursive Evaluation Mechanism: Load task → Generate code → Sandbox execution → Feedback correction → Repeat until success or maximum iterations. Scoring dimensions include result correctness (40%), iteration efficiency (25%), code quality (20%), and execution efficiency (15%)
3. Secure Environment: Sandbox isolation, timeout control, resource limits, network isolation
4. Automated Scoring: Multi-strategy scoring for numerical values (exact/tolerance/range), tables (rows/columns/structure), and code (syntax/library usage)

Environment Requirements

Windows 10/11 or Linux/macOS (source code execution), 4GB+ RAM, Python3.9+ (API usage)

Quick Start

Windows users can download .exe/.zip files, unzip and run; source code users need to configure the Python environment

Typical Workflow

Open the application → Select task set → Configure model → Set parameters → Start evaluation → View results

Application Scenarios

• Model development: Verify version improvements, identify weaknesses, compare architectures
• Prompt engineering: Test prompt strategies, optimize DSPy modules
• Production deployment: Evaluate reliability before launch, establish baselines
• Academic research: Standardized benchmarks, reproducible experiments

Result Interpretation

Reports include task status, overall score, iteration statistics, error classification, and detailed logs

1. Native Support for Iterative Evaluation: Records state changes per round, analyzes error correction patterns, evaluates self-improvement efficiency
2. Diverse Scoring Strategies: Understands data semantics, tolerates reasonable format differences, detects partially correct cases
3. Out-of-the-Box Experience: Windows executable files do not require a Python environment, lowering the entry barrier

Current Limitations

• Mainly for Windows users, limited cross-platform support
• Task set coverage needs expansion
• Advanced visual evaluation is not perfect

Future Plans

• Expand data source types (SQL, API)
• Add multi-language support (R, Julia)
• Integrate continuous testing framework
• Support distributed evaluation acceleration

Tool	Features	Application Scenarios
DABench-RLM-Eval	Focuses on RLMs, data analysis, iterative evaluation	DSPy developers, RLM research
BigCode Evaluation Harness	General code evaluation, multi-language support	General code model evaluation
HumanEval/MBPP	Classic programming benchmarks, one-time generation	Basic code capability testing
DS-1000	Data science tasks, Python-focused	Data science model evaluation

The uniqueness of DABench-RLM-Eval lies in its focus on the intersection of Recursive Language Models × Data Analysis Tasks.

As AI programming assistants evolve toward complex tasks, evaluating RLMs' ability to handle multi-step data analysis is crucial. DABench-RLM-Eval provides a professional automated evaluation framework, helping developers and researchers quantify RLM performance, track iterative improvement effects, and establish decision-making basis for production deployment. For teams using or researching DSPy RLMs, it is a practical framework worth including in the toolchain.