simple-evals-mm: A Multimodal Evaluation Framework for Vision-Language Models, Facilitating Standardization of VLM Performance Assessment

simple-evals-mm is an open-source project developed by the llm-jp team, extended from OpenAI simple-evals, specifically designed to provide a standardized evaluation solution for Vision-Language Models (VLMs). This framework supports over 20 authoritative benchmark tests, covering multimodal datasets such as AI2D, MMMU, and ScienceQA. It is also an important component of the JAMMEval evaluation project, aiming to address the lack of objectivity and comprehensiveness in VLM assessments.

With the rapid development of VLMs like GPT-4V, Gemini, and Qwen-VL, traditional text model evaluation frameworks can no longer meet the needs of multimodal evaluation, as existing tools lack uniformity and scalability. Against this backdrop, the llm-jp team launched simple-evals-mm as a multimodal extended version of OpenAI simple-evals, providing systematic support for VLM performance evaluation.

Multimodal Benchmark Datasets

Integrates over 20 authoritative English datasets such as ChartQA (Chart Question Answering), AI2D (Scientific Diagram Understanding), and MMMU (Multidisciplinary Multimodal Understanding), covering dimensions like chart/document comprehension, scientific reasoning, fine-grained recognition, and real-world scenarios.

Japanese Scenario Support

Integrates Japanese benchmarks from the JAMMEval series such as CC-OCR, JDocQA, and JMMMU, filling the gap in Japanese VLM evaluation.

Text Capability Preservation

Retains classic text tests like GPQA, MATH, and MMLU to comprehensively assess the model's basic language capabilities.

Multi-Backend Model Compatibility

Supports OpenAI (GPT-4o, GPT-5.1), Google Gemini, and open-source models (InternVL, Qwen-VL, etc.), enabling fair comparison of different VLMs.

Modern Environment Management

Uses uv (a high-speed package manager written in Rust), with uv sync for quick environment configuration and uv run to execute scripts ensuring consistency.

Result Analysis Tools

Built-in visualization scripts generate comparison charts; an interactive web viewer supports side-by-side viewing of model outputs and images, facilitating error pattern analysis.

CLI Tool Workflow

1. List available models: uv run python src/simple_evals_mm/simple_evals.py --list-models
2. List evaluation tasks: uv run python src/simple_evals_mm/simple_evals.py --list-evals
3. Execute evaluation: Specify the model and benchmark; supports repeated runs to obtain statistical significance.

Dataset Management

Most benchmarks are automatically downloaded from HuggingFace; preparation guides are provided for special datasets.

Result Format

Saves three layers of results in JSONL format: single-sample detailed output, aggregated scores, and statistical summaries (mean, standard deviation, etc.).

The project has published a related paper (arXiv:2604.00909) that elaborates on the JAMMEval benchmark construction concept and evaluation methodology. It is open-sourced under the MIT license and provides CONTRIBUTING.md to guide community contributions. It also points out limitations: the flexibility constraints in model output evaluation may lead to underestimation of strong models' performance, reflecting academic rigor.

simple-evals-mm is an important step toward the standardization and systematization of VLM evaluation, providing reliable infrastructure for VLM research and development. In the future, it will further expand coverage of emerging evaluation sets, support more model backends, and continuously innovate evaluation methodologies. It is an open-source project worth attention for professionals in VLM research, development, and application.