VisualPRM Medical Reasoning Pipeline: A Process Reward Model Dataset Construction Tool for Multimodal Visual Question Answering

VisualPRM-Medical-PRM is a multi-step Process Reward Model (PRM) dataset construction tool for medical Visual Question Answering (VQA) tasks, designed to address the interpretability challenges of medical AI. This tool supports multiple backends such as OpenAI, Gemini, and local open-source models, provides a visual annotation interface and multiple training data export formats, filling the gap of PRM tools in the medical field.

In the field of medical AI, model interpretability is crucial for clinical applications. However, existing medical VQA models are mostly trained as end-to-end black boxes, lacking supervision of intermediate reasoning steps. PRM technology provides a new approach to solve this problem by assigning reward signals to each reasoning step, guiding the model to learn structured and interpretable reasoning chains.

Traditional Outcome Reward Model (ORM) only rewards at the final answer level, while PRM scores each intermediate reasoning step. Its advantages include: 1. More precise error localization, able to identify specific error steps; 2. Optimized reasoning quality, generating coherent and logically rigorous reasoning chains. For example, in medical image analysis, the model can show a complete chain from observing abnormalities to considering diseases and then suggesting examinations.

VisualPRM provides a complete dataset construction process:

1. Multi-candidate reasoning generation: Use the sampling capability of large models to generate diverse reasoning candidates;
2. Monte Carlo Rollout scoring: For each candidate step prefix, complete the remaining reasoning through multiple random samplings, and use the final answer accuracy rate as the PRM score;
3. Manual confirmation and override: Provide a visual web interface, allowing experts to review and modify automatic labels;
4. Multi-format export: Support formats such as raw JSON, VisualPRM-specific JSON, step-level training JSON/JSONL, etc., to adapt to different training frameworks.

VisualPRM supports four operation modes to meet different needs:

Mode	Application Scenario	Configuration Example
Commercial	Production environment (pursuing optimal performance)	OpenAI GPT-4o-mini
Gemini	Google Cloud ecosystem users	Gemini 2.5 Flash
Open Model	Data privacy-sensitive scenarios (local deployment)	Qwen2.5-VL-7B-Instruct
Demo	Conference demonstration (offline preview)	Pre-generated results (zero API cost)
This design allows seamless switching from cloud APIs to local models, expanding the tool's application scope.

The application scenarios of VisualPRM include:

• Medical education: Build high-quality reasoning datasets to train medical students' clinical thinking;
• AI-assisted diagnosis: Develop interpretable medical AI systems, providing diagnostic suggestions with reasoning processes;
• Medical knowledge graph construction: Extract structured reasoning patterns from question-answer pairs;
• Model evaluation benchmark: Establish a fine-grained evaluation system to assess the quality of reasoning processes rather than just the final answer.

Limitations of the current version: It mainly focuses on VQA-format medical question-answering, with insufficient support for complex clinical decision-making scenarios (such as multi-turn consultations); the Monte Carlo Rollout calculation cost is relatively high. Future directions:

• Support PRM construction for multi-turn conversational medical consultations;
• Integrate more medical image modalities (CT, MRI, pathological slices, etc.);
• Develop active learning strategies to reduce manual annotation;
• Deep integration with open-source medical large models (such as MedLlama, Huatuo).