EHRStruct: A New Benchmark for Medical AI, Enabling Large Models to Truly Understand Electronic Health Records

EHRStruct, developed by a team from Nanyang Technological University (NTU) in Singapore, is the first comprehensive benchmark for systematically evaluating the performance of large language models (LLMs) on structured electronic health record (EHR) tasks. Accepted as an Oral paper at AAAI 2026, it provides a standardized evaluation system for the medical AI field.

• Original Authors/Maintainers: Xiao Yang, Xuejiao Zhao, Zhiqi Shen (LILY Research Center, Nanyang Technological University)
• Source Platform: GitHub
• Original Title: EHRStruct: A Comprehensive Benchmark Framework for Evaluating Large Language Models on Structured Electronic Health Record Tasks
• Original Link: https://github.com/YXNTU/EHRStruct
• Release Date: May 24, 2026
• Paper Publication: AAAI 2026 Oral

Electronic Health Records (EHRs) are the digital cornerstone of modern medicine, containing the full range of patient information from admission to discharge—diagnosis records, medication lists, test results, vital sign monitoring data, etc. However, this data is often stored in highly structured table formats, and understanding these tables is not an easy task for large language models. Existing medical AI benchmarks mostly focus on medical Q&A or clinical dialogue, lacking systematic evaluation of structured EHR data. This leads to an awkward situation: models perform well on medical exam questions but frequently make mistakes in real-world EHR analysis tasks. The medical AI field urgently needs a standardized evaluation framework that can comprehensively test models' ability to handle structured medical data.

EHRStruct was born to address this pain point. As the first comprehensive evaluation benchmark for LLMs specifically targeting structured EHR tasks, it defines 11 clinically meaningful evaluation tasks covering 6 major categories, providing a reliable "touchstone" for medical AI research and development. The core contribution of this project lies in establishing a complete evaluation pipeline: starting from clinical demand induction and task distillation, building a classification system based on clinical scenarios and reasoning levels, then extracting standardized samples from real and synthetic EHR data, and finally forming a reproducible model evaluation process.

EHRStruct's 11 evaluation tasks are divided into 6 major categories, each corresponding to practical application scenarios in clinical work:

Filtering Tasks (D-U1/U2): Test the model's ability to screen patients who meet specific criteria from a large number of EHR records, e.g., "Find all hypertensive patients who use a certain drug".

Aggregation Tasks (D-R1/R2/R3): Evaluate the model's ability to perform summary analysis on multi-source data, e.g., "Calculate the average length of stay in a department over the past month".

Arithmetic Tasks (D-R4/R5): Test the model's numerical calculation ability in medical scenarios, e.g., "Calculate the total daily intake based on the patient's weight and drug dosage".

SNOMED Task (K-U1): Test the model's understanding of the medical ontology SNOMED CT, including conceptual hierarchical relationships and semantic reasoning.

Mortality Prediction (K-R1): Evaluate the model's ability to predict a patient's mortality risk based on EHR data.

Disease Diagnosis (K-R2): Test the model's ability to infer diseases based on symptoms and test results.

Medication Recommendation (K-R3): Test the model's ability to provide reasonable medication recommendations considering the patient's medical history and current medications.

EHRStruct uses two complementary data sources to build evaluation samples: