FairMedQA: A Benchmark Dataset and Empirical Study for Evaluating the Fairness of Medical AI

This article introduces FairMedQA—an open-source benchmark dataset for evaluating the fairness of large language models (LLMs) in medical question-answering tasks. Through counterfactual samples and adversarial testing, this study reveals bias issues in current medical AI systems across dimensions such as race, gender, and socioeconomic status, providing standardized tools and empirical evidence for building more fair medical AI.

Artificial intelligence is widely applied in the medical field, but the fairness issues of LLMs in medical question-answering are becoming increasingly prominent. The medical system itself has inherent inequalities; if AI learns biases from historical data, it may amplify rather than mitigate these gaps. The FairMedQA project aims to create a standardized benchmark to evaluate performance differences of medical AI across different demographic groups, supporting the construction of fair medical AI.

FairMedQA uses a counterfactual approach to construct test samples: paired cases only change demographic characteristics (race, gender, SES) while keeping clinical information consistent—if the AI gives different answers, it indicates bias. Data sources include MedQA (USMLE-style questions) and expert-reviewed clinical cases. The dataset structure includes original questions, variant questions (demographic characteristic variants), neutralized versions, and adversarial samples. The sample generation process is: GPT-4/DeepSeek generate cases → expert review → variant generation → quality control.

Core fairness metrics include accuracy difference (correct rate difference across groups), consistency test (McNemar test for consistency of paired samples), and fairness heatmap (visualizing performance differences across groups). It can detect bias types such as explicit, implicit, representational, and annotation biases. The evaluation framework adopts multi-agent collaboration: GPT-Agent generates answers and evaluations, DeepSeek-Agent performs comparative verification, and human experts conduct sampling reviews.

The study reveals that mainstream medical LLMs have significant biases: In terms of race, some models have lower accuracy when handling cases of Black patients compared to White patients; in terms of gender, there are stereotypes in gynecology and mental health fields; in terms of SES, the accuracy of cases involving low-income patients is lower. Sources of bias include training data deviation (insufficient group representation), model architecture limitations (lack of fairness constraints), and evaluation method issues (ignoring group differences). Model comparisons show that closed-source models (such as GPT-4) are overall better but still have biases, while open-source models (such as Llama) have more serious fairness issues.

Academically, FairMedQA provides the first medical fairness benchmark, empirical evidence, and methodological innovations. Practically, it provides guidance for developers on fairness training and deployment, and offers evaluation tools for regulators. Policy-wise, it suggests that medical AI needs to pass fairness evaluations before being launched, promoting the formulation of industry standards and resource investment in fairness research.

Current limitations: Geography (mainly U.S. scenarios), disease coverage (insufficient coverage of rare diseases), bias dimensions (inadequate coverage of age, etc.), and evaluation methods (automatic evaluation errors). Future directions: Expand the dataset (geography, diseases, demographic dimensions), improve methods (fine-grained bias detection, causal inference), intervention research (effectiveness of debiasing techniques), and policy research (evaluation of regulatory strategies).