FHIRBench: A Systematic Benchmarking Framework for Clinical Data Serialization Strategies

FHIRBench is an open-source benchmarking framework for clinical data serialization strategies in the healthcare domain. It systematically evaluates 6 serialization formats, 4 large language models (LLMs), and 3 types of clinical tasks. Its goal is to provide a scientific basis for data format selection in healthcare AI applications, filling the gap of a lack of unified evaluation framework in this field.

In the healthcare AI field, FHIR has become the standard for clinical data exchange. However, when LLMs process FHIR data, developers face the problem of how to serialize it effectively. Different strategies affect model understanding ability, reasoning accuracy, and computational efficiency. Currently, the industry lacks a unified and systematic evaluation framework to compare their advantages and disadvantages.

FHIRBench has designed a comprehensive testing matrix covering three key dimensions:

1. Serialization Formats: Evaluate 6 commonly used formats such as JSON, XML, YAML, and LLM-optimized textual representations;
2. Large Language Models: Cover 4 mainstream models (e.g., GPT series, Claude, open-source models, etc.);
3. Clinical Tasks: Include three typical scenarios: clinical question answering, information extraction, and decision support.

Synthetic Data Generation

Use Synthea to generate FHIR R4 standard synthetic data, which protects privacy while providing diverse testing scenarios.

Serializer Implementation

The serializers/ directory contains implementations for various formats, ensuring semantic integrity and hierarchical relationships are preserved during conversion.

Evaluation Framework

The evaluation/ module provides standardized metrics: accuracy (matching degree with standard answers), efficiency (processing time/resource consumption), and robustness (stability under data complexity).

For Developers

1. Choose the optimal serialization strategy;
2. Optimize prompt engineering;
3. Evaluate model adaptability.

For the Ecosystem

The open-source nature supports community reproduction and verification, contribution of new formats/tasks, and development of better solutions, promoting deep integration of FHIR and AI.

Core Modules:

• data/synthea/: Synthetic data management;
• serializers/: Serialization implementation;
• evaluation/: Evaluation tools;
• tasks/: Clinical task definitions;
• specs/: Configuration files;
• docs/: Documentation. It uses the MIT license, and dependencies are managed via requirements.txt for easy deployment.

FHIRBench fills the gap of a lack of systematic serialization evaluation standards in the healthcare AI field, providing a scientific basis for application development. In the future, it will expand to support more serialization formats, models, and clinical tasks, becoming an important part of healthcare AI infrastructure.