MADE: A Dynamic Benchmark for Multi-Label Classification and Uncertainty Quantification of Medical Device Adverse Events

MADE is a dynamic multi-label classification benchmark for medical device adverse events. It prevents data contamination via strict temporal partitioning, systematically evaluates the predictive performance and uncertainty quantification (UQ) methods of over 20 models, and reveals the complex trade-off between model size and UQ quality. Its core innovation lies in the continuous update mechanism, which addresses the issues of saturation and data contamination in existing benchmarks, providing a real and reliable evaluation platform for medical AI.

Multi-label text classification (MLTC) in medical AI faces challenges such as label imbalance, label dependency, and combinatorial complexity. Existing MLTC benchmarks have issues of saturation (performance reaching a ceiling) and data contamination (models memorizing test data). Due to sensitive data and rapid knowledge evolution in the medical field, static benchmarks are even harder to meet the demands.

MADE adopts a dynamic update mechanism (automatically incorporating new reports), a hierarchical long-tail label system (reflecting medical classification ontologies and real distributions), and strict temporal partitioning (training/validation/test sets divided by release time). These measures fundamentally prevent data contamination and ensure real evaluation and generalization ability testing.

Evaluating over 20 models revealed: Small discriminative fine-tuned decoders have the strongest overall accuracy; generative fine-tuned models have the most reliable UQ; large reasoning models perform well on rare labels but have weak UQ. Comparison of UQ methods: The entropy method is simple and efficient but has limitations; the consistency method is reliable; the gap between self-verbalized confidence and actual accuracy is significant, making it unreliable.

Model selection needs to balance scenarios: Use large reasoning models for rare cases, generative fine-tuned models for reliable UQ, and small discriminative models for resource-constrained situations. For UQ design, prioritize entropy or consistency methods and tune thresholds. Continuous monitoring and updating of models are needed to adapt to the evolution of the medical field.

Limitations: Only covers English reports, the label system needs refinement, and automated updates require quality control. Future directions: Explore active learning, use hierarchical labels to improve reasoning, and develop medical-specific UQ methods.