B&J Benchmark: A Comprehensive Evaluation Framework for Medical Multimodal Models Targeting Musculoskeletal Diseases

B&J Benchmark is a comprehensive evaluation framework specifically designed for musculoskeletal diseases, aiming to systematically assess the performance of large language models (LLMs) and vision-language models (VLMs) across various stages of clinical reasoning. This framework fills the gap in existing medical AI evaluation benchmarks for the musculoskeletal specialty, covering the complete process from basic medical knowledge to complex clinical decision-making. It has systematically evaluated mainstream multimodal and pure-text models, providing important support for medical AI research and development, clinical application, and industry standardization.

As LLMs and VLMs are increasingly applied in the medical field, accurately evaluating the real clinical performance of models has become a key issue. Existing medical AI evaluation benchmarks mostly focus on general medical knowledge or specific imaging modalities, lacking a dedicated evaluation framework for the musculoskeletal system. Diagnosis of musculoskeletal diseases requires integrating multi-dimensional data such as image interpretation and medical history collection, so B&J Benchmark was created to fill this gap.

Core Components of the Evaluation Framework

• Medical knowledge recall: Assess the mastery of basic medical knowledge of the musculoskeletal system
• Clinical case interpretation: Evaluate the ability to understand and analyze text information in medical records
• Medical image interpretation: Test the ability to recognize and analyze images such as X-rays, CT, and MRI
• Diagnosis generation and reasoning: Verify the ability to make accurate diagnoses and explanations based on multi-source information
• Treatment plan planning and justification: Evaluate the ability to formulate treatment plans and clarify clinical basis

Dataset Design

A mixed question type of multiple-choice questions + open-ended questions is adopted, referencing authoritative medical textbooks and clinical guidelines. It includes questions of different difficulty levels, balancing the assessment of knowledge reserve and clinical reasoning ability.

Vision-Language Models

General models: GLM-4V-9B, Qwen2-VL-7B, MiniCPM-V2.6, Llama-3.2-Vision-11B, GPT-4o, Claude 3.5 Sonnet, DeepSeek-VL2 Medical-specific models: Med-Flamingo, LLaVA-Med, MedVInT, MiniGPT-Med

Pure-Text Large Models

General models: DeepSeek-R1, Qwen2.5-32B, GLM-4-9B Medical-specialized models: MedGPT, MedFound, Baichuan-M2

The diverse selection of models ensures the reference value of the evaluation results, helping to understand the advantages and disadvantages of different technical routes.

1. Provide optimization directions for medical multimodal model research and development: Identify model knowledge blind spots, reasoning chain gaps, and clinical expression deficiencies through error analysis, and make targeted improvements to architectures and training strategies
2. Provide objective basis for medical institutions to select AI-assisted systems: Choose suitable solutions based on differences in model evaluation indicators
3. Promote the standardization process of medical AI: Unify evaluation standards and compare public results to promote industry consensus, accelerating technology iteration and application implementation

B&J Benchmark is released as open-source, with clear code and dataset structures, including Python evaluation code, standard question sets, original model outputs, and scoring results. The evaluation code implements standardized model calling interfaces and scoring logic, supporting batch evaluation of mainstream models; the question sets are classified by evaluation dimensions, with correct answers and scoring standards labeled, ensuring the fairness and credibility of the evaluation process.

Limitations

The current dataset is based on static questions and standard answers, which has a gap compared to the dynamic and open diagnosis and treatment process in real clinical settings

Future Directions

• Expand the dataset scale to cover more rare diseases and complex cases
• Introduce multi-round interactive evaluation to simulate real consultation processes
• Establish human-machine comparison benchmarks to evaluate the actual gain of AI assistance on clinical decision-making
• Explore deep capability dimensions such as model interpretability evaluation