TreeDDx: Evaluating Large Language Models' Differential Diagnosis Reasoning Ability Using Structured Clinical Decision Trees

TreeDDx is a benchmark framework for large language models (LLMs) on clinical differential diagnosis tasks, evaluating models' reasoning ability and diagnostic accuracy using structured decision trees. It addresses the problem that existing medical benchmarks struggle to assess complex clinical decision-making reasoning chains, making the reasoning process of LLMs traceable and quantifiable.

LLMs perform well in medical question answering and knowledge retrieval tasks, but in real clinical scenarios, differential diagnosis requires models to have structured reasoning ability (screening the most likely diagnosis from multiple diseases). Existing benchmarks mostly focus on single-turn question answering or knowledge recall, making it difficult to evaluate complex reasoning chains. TreeDDx was created to fill this gap, introducing an evaluation paradigm based on clinical decision trees.

TreeDDx formalizes differential diagnosis as a decision tree generation and matching problem: 1. Structured decision tree representation (nodes are diagnostic hypotheses, edges are supporting/excluding evidence); 2. Comparison of model outputs with expert-annotated ground truth decision trees; 3. Multi-dimensional evaluation (correctness of final diagnosis, rationality of reasoning path, coverage of key nodes, integrity of logical chain). It can capture reasoning flaws missed by traditional metrics (e.g., guessing the diagnosis correctly but with reasoning jumps).

TreeDDx data comes from real challenging cases in the JAMA Network Clinical Challenge, with preprocessing scripts provided to convert them into decision tree samples. Key components include: gt_decisiontree_generation.py (generates standard decision trees), llm_decisiontree_generation.py (calls LLMs to generate decision trees), and evaluation.py (similarity calculation and multi-dimensional evaluation). Note: Original JAMA cases require users to obtain authorization on their own.

Decision tree similarity evaluation is the core, using a combination of graph edit distance (structural level) and node semantic similarity (calculated by medical pre-trained models): topological similarity (node hierarchy, branches) is computed at the structural level; node text similarity at the semantic level; a weighted combination gives the matching score. This more finely characterizes differences in reasoning ability than traditional methods.

The value of TreeDDx: 1. Emphasizes the interpretability of medical AI (decision trees are naturally interpretable); 2. Diagnoses model flaws at a fine-grained level (locates weaknesses in specific diseases or reasoning links); 3. Versatility (transferable to complex reasoning fields such as law and engineering).

Limitations of TreeDDx: It relies on the in-context learning ability of LLMs; incomplete decision trees may be generated for challenging cases; ground truth annotation requires a lot of expert time, limiting dataset size. Future directions: Introduce stronger LLMs to generate decision trees, develop semi-automated annotation tools, and combine RLHF to optimize reasoning ability.

TreeDDx is an important advancement in medical AI evaluation methods, formalizing the complex clinical cognitive process of differential diagnosis into a computable and comparable structured task. For medical LLM researchers and developers, it is not only a benchmark tool but also a paradigm reference for evaluating and optimizing models' clinical reasoning ability.