M_Judger: Advancing Multimodal Evaluation Models via Competency-Oriented Benchmarks and MCTS-Driven Data Generation

This article introduces the M_Judger project, which systematically enhances the evaluation and training capabilities of multimodal evaluation models by constructing a competency-oriented evaluation benchmark M-JudgeBench and a Monte Carlo Tree Search (MCTS)-driven data generation method.

With the rapid development of Large Multimodal Models (LMMs) in tasks such as visual understanding and image-text generation, accurately evaluating the output quality of these models has become a key challenge. Traditional evaluation methods often rely on manual annotation or simple rule matching, which struggle to capture the complexity and nuances of multimodal tasks. The M_Judger project addresses this issue by proposing a systematic solution, aiming to advance the development of multimodal evaluation models through more refined competency division and smarter data generation strategies.

M_Judger is an open-source project developed by researchers including Chen Zhiyuan. Its core contributions consist of two main parts: first, constructing the competency-oriented multimodal evaluation benchmark M-JudgeBench; second, developing a Monte Carlo Tree Search (MCTS)-based data generation method. The project has been open-sourced on GitHub and accompanied by an arXiv paper (arXiv:2603.00546), providing the research community with complete reproducibility resources and experimental data.

The design philosophy of M-JudgeBench is to decompose multimodal evaluation capabilities into multiple fine-grained dimensions instead of using a single comprehensive score. This competency-oriented evaluation method can more accurately identify the strengths and weaknesses of models.

In terms of data construction, M-JudgeBench adopts two innovative error sample generation strategies. The first is the construction of Result-error pairs: by having different models perform inference under varying temperature parameters and inference length settings, diverse output results are collected, and sample pairs containing errors are filtered out. This method can cover various error patterns that models may exhibit under different inference strategies.

The second is the generation of Process-error data: through controlled noise injection techniques, errors in the inference process are intentionally introduced while keeping the final answer correct. This type of data is particularly important for training models to recognize the subtle scenario of "correct answer but wrong reasoning", which is key to improving the robustness of evaluation models.

Monte Carlo Tree Search (MCTS), a classic decision optimization algorithm, is innovatively applied to the data generation process in M_Judger. Traditional data generation often uses random sampling or greedy search, which makes it difficult to systematically explore the space of high-quality training samples.

The MCTS method builds a search tree structure and balances Exploration and Exploitation in each iteration, gradually converging to high-value generation paths. In the multimodal evaluation scenario, this means that "edge cases"—samples that models easily confuse and find hard to evaluate accurately—can be generated more targeted. These edge cases have extremely high training value for improving the discriminative ability of evaluation models.

The M_Judger project provides complete code implementation and datasets, including data construction pipelines, evaluation scripts, and pre-trained models. The project's modular design allows researchers to easily reproduce the paper's results or migrate its methods to other multimodal tasks.

Notably, the project team has publicly released detailed descriptions of the data construction process, including how to extract error patterns from the inference outputs of different models and how to design noise injection strategies to generate process-error data. This transparency is of great significance for promoting method reproduction and comparative research in the field.

The research results of M_Judger have multiple practical values for the multimodal AI field. First, M-JudgeBench provides model developers with more refined diagnostic tools to help identify the shortcomings of models in specific competency dimensions. Second, the MCTS-driven data generation method can significantly improve the quality of training data and reduce the waste of training resources on low-value samples.

From a broader perspective, as multimodal models are increasingly applied in key fields such as autonomous driving, medical image analysis, and educational assistance, reliable automatic evaluation capabilities will become an important guarantee for ensuring system safety and effectiveness. The methodology proposed by M_Judger provides a technical foundation for these application scenarios.

The M_Judger project opens up new directions for the research of multimodal evaluation models through competency-oriented evaluation design and intelligent data generation strategies. Its open-source implementation and detailed documentation lower the threshold for subsequent research and are expected to inspire more innovations in evaluation model architectures, training strategies, and application scenarios. As multimodal AI technology continues to evolve, foundational work like M_Judger will play an increasingly important role in building more reliable and controllable AI systems.