LLM Internal Medicine Monitoring Toolkit: A Professional Evaluation Framework for Medical Large Language Models

The LLM Internal Medicine Monitoring Toolkit (llm-internal-medicine) is an open-source project developed by the bo-ke team, focusing on providing systematic evaluation and monitoring capabilities for large language models in internal medicine scenarios. This toolset aims to address the issue that general evaluation benchmarks cannot meet the high accuracy, low error tolerance, and strict regulatory requirements of medical scenarios. Through a standardized test case library, automated evaluation pipeline, and multi-dimensional evaluation metrics, it helps researchers, developers, and medical institutions verify the reliability of medical large language models, suitable for product development, academic research, and technology selection scenarios.

Medical AI applications have far higher requirements for accuracy, error tolerance, and regulatory compliance than general scenarios. While general evaluation benchmarks (such as MMLU, HumanEval) can measure basic capabilities, they struggle to capture the nuances of medical scenarios—for example, general models may make fatal errors in complex internal medicine cases. This capability gap highlights the necessity of domain-specific evaluation tools.

llm-internal-medicine does not build medical large models; instead, it provides an evaluation toolset. Its core objectives include: 1) Establishing a standardized test case library for common internal medicine diseases; 2) Providing an automated model performance monitoring mechanism; 3) Supporting multi-dimensional evaluation metric collection and analysis. This tool can be used with various basic models and has wide applicability.

Internal Medicine Disease Test Case Library

Covers major internal medicine branches such as cardiovascular, respiratory, and digestive systems. The cases are reviewed by medical professionals, including real and complete case descriptions (chief complaints, present illness history, etc.) and multi-dimensional questions (diagnostic reasoning, treatment plans, etc.), with difficulty levels forming an evaluation gradient.

Automated Evaluation Pipeline

Supports three modes: single-model in-depth evaluation, multi-model comparison, and continuous monitoring. After users configure the model API or local path, the system automatically runs tests, collects outputs, compares results, and generates reports.

Multi-Dimensional Evaluation Metrics

In addition to accuracy, it includes diagnostic accuracy, completeness of differential diagnosis, rationality of treatment plans, coverage of risk prompts, etc., to fully reflect the clinical value of the model.

Modular Design

The core evaluation engine is independent of the test case library, analysis module, and report module, facilitating expansion and community contributions (medical experts can focus on adding cases without needing to understand technical details).

Extensible Model Interfaces

Supports access to cloud APIs (OpenAI-compatible) and local open-source models (such as Llama, Mistral), providing sample configurations for quick integration with medical-specific models.

Result Visualization and Export

Evaluation results can be exported in HTML (for manual review), JSON (for system integration), and CSV (for data analysis) formats, supporting CI/CD processes.

Medical AI Product R&D

Assists in model selection (comparing capabilities of different basic models), fine-tuning optimization (tracking iterative performance), and generating evaluation reports before release.

Academic Research Benchmark

As a standardized evaluation benchmark, it enhances experimental reproducibility and result comparability, encouraging the community to contribute test cases to improve the open benchmark.

Medical Institution Technology Selection

Helps medical institutions objectively evaluate candidate AI products, verify vendor technical indicators, and make informed procurement decisions.

Current Limitations

• Test case coverage still needs expansion;
• Does not cover multi-modal medical data (imaging, test reports) processing capabilities;
• Automated scoring of complex cases requires manual review.

Future Outlook

Relies on community collaboration: Calls on medical professionals to contribute real cases and annotations, and technical personnel to improve evaluation algorithms and functional modules, with the goal of becoming an important infrastructure in the field of medical large model evaluation.

llm-internal-medicine represents the development direction of specialized evaluation tools in the medical AI field. Against the backdrop of rapid iteration of medical large models, this domain-specific evaluation framework is of great significance for ensuring the safety and effectiveness of AI systems, and is worth exploring and using by medical AI researchers and practitioners.