EHRStruct: The Touchstone of Medical AI—A New Benchmark for Evaluating Large Models on Structured Electronic Health Records

This article interprets the AAAI 2026 Oral paper EHRStruct, a medical large model evaluation framework for structured Electronic Health Record (EHR) tasks. It includes 11 clinical tasks and 2200 standardized samples, aiming to address the objective and systematic issues in medical AI evaluation and provide an important tool for assessing its reliability and practicality.

Large language models are widely used in the medical field, but traditional evaluations focus on single tasks (e.g., image classification accuracy) and cannot reflect the ability to handle complex structured EHRs in real clinical settings. The EHRStruct framework developed by the team from Nanyang Technological University, Singapore, has received the AAAI 2026 Oral honor, opening a new path for systematic evaluation.

EHRStruct covers 11 clinical tasks, divided into 6 major categories (data understanding, data reasoning, knowledge understanding, knowledge reasoning, etc.). The dataset comes from Synthea synthetic data (no privacy risks, scalable) and eICU real clinical data (requires authenticated access). The team provides preprocessing code and data.

EHRStruct supports four input formats: plain text, LaTeX, hypergraph, and natural language generation; it adopts a standardized process of clinical expert review and multiple validations; and supports zero/few-shot evaluation. The team also developed the EHRMaster baseline model, which optimizes table encoding, injects medical knowledge, and conducts multi-task joint training.

Experiments compare general and medical models: general models excel at data understanding, medical models are strong in knowledge reasoning, and there is a non-linear relationship between scale and performance; task difficulty gradients are obvious (data filtering is easy, while terminology standardization and medication reasoning are difficult); model performance is significantly affected by input formats.

Since its release in November 2025, EHRStruct has received attention from media such as AI_Era. In December 2025, a Codabench challenge was launched, and the open-source license supports academic use. Implications: Evaluation drives innovation (e.g., ImageNet promoted computer vision), structured data processing capabilities need optimization, and deep medical knowledge integration still faces challenges.

Usage requires an environment such as Python 3.9+. You can choose to preprocess Synthea data or apply for eICU data. An example command is python run.py --llm Qwen72B --task aggregation --type txt --k 0. Limitations: Does not cover multimodality, is limited to English, and uses static data; future plans include expanding tasks, multi-language support, interactive evaluation, etc.