S3 Dataset: A Significant Breakthrough in Multimodal Large Models for Medical Video Understanding

Seizure-Semiology-Suite (S3) is the first multimodal dataset and benchmark for understanding seizure semiology, containing 438 seizure videos and over 35,000 dense annotations covering 20 ILAE-defined semiological features. This study reveals the systemic weaknesses of current multimodal large language models (MLLMs) in medical video understanding and proposes improvement solutions, providing key benchmarks and development directions for the medical AI field.

Multimodal large language models have made significant progress in general video understanding tasks, but face huge challenges in safety-critical fields such as medicine. Seizure semiology requires understanding involuntary, spatiotemporally evolving pathological motor behaviors, which places extremely high demands on models' temporal reasoning capabilities and medical expertise. Existing models lack reliability in high-risk, high-precision medical fields and struggle to handle complex clinical dimensions such as spatiotemporal patterns of symptoms and lateral localization.

S3 is the first large-scale clinical dataset for seizure semiology, containing 438 seizure videos and over 35,000 dense annotations covering 20 semiological features defined by the International League Against Epilepsy (ILAE). Annotations are completed by professional neurologists, including clinical in-depth information such as symptom onset time, left-right distribution, and evolution sequence, providing a solid foundation for model training and evaluation.

The study designed a seven-layer hierarchical evaluation framework to comprehensively examine model capabilities from low-level visual perception to high-level clinical reasoning: 1. Low-level visual perception; 2. Temporal localization; 3. Left-right reasoning; 4. Symptom sequence understanding; 5. Narrative report generation; 6. Seizure vs. non-seizure differentiation; 7. Comprehensive diagnostic reasoning. Meanwhile, the Seizure-RQI metric is proposed to evaluate the clinical utility of reports from dimensions such as symptom completeness, temporal accuracy, and lateral correctness, making up for the deficiencies of traditional automatic evaluation metrics.

Evaluation of 11 open-source multimodal large language models revealed key weaknesses: 1. Insufficient left-right reasoning ability (affecting epileptogenic focus localization); 2. Limited temporal localization accuracy; 3. Weak symptom sequence understanding; 4. Lack of clinical fidelity (non-standard reports or missing key information).

Domain-specific fine-tuning for the epilepsy field can significantly improve model performance. The two-stage neuro-symbolic framework proposed in the study achieved an F1 score of 0.96 in seizure vs. non-seizure classification tasks. This framework first uses neural networks to extract video symptom features, then integrates these features through a symbolic reasoning layer for clinical judgment, combining the perceptual capabilities of deep learning with the interpretability of symbolic reasoning.

The S3 dataset fills the gap in the evaluation of multimodal large models for medical video understanding, providing researchers with strict benchmarks and improvement directions. For medical AI teams, S3 is a valuable resource (high-quality data, comprehensive evaluation benchmarks, validated improvement pathways). Future research based on S3 is expected, especially in directions such as medical knowledge injection, temporal reasoning enhancement, and neuro-symbolic fusion, to promote the safe and effective application of multimodal intelligence in the medical field.