LLM-CAT: Efficient Medical Benchmark Evaluation of Large Language Models Using Computerized Adaptive Testing

The LLM-CAT project innovatively applies Computerized Adaptive Testing (CAT) technology to the field of medical benchmark evaluation for large language models. Its core goal is to maintain accurate assessment of the model's medical knowledge level while significantly reducing the number of evaluation questions, addressing the bottleneck of high computing and time costs in traditional fixed testing modes.

Evaluation Cost: An Invisible Bottleneck for Large Language Model Development

As the capabilities of large language models (LLMs) improve, traditional benchmark evaluations require models to answer a large number of pre-set questions, leading to huge computing and time costs. This is particularly prominent in the medical field: medical benchmark tests contain thousands of professional questions (covering diagnosis, treatment, pathology, and other dimensions), and a complete evaluation consumes a lot of API call fees or computing resources, limiting the frequency of experiments by researchers and hindering the participation of resource-constrained teams in evaluations.

Principles of Computerized Adaptive Testing (CAT)

CAT originates from educational psychology. Its core is to dynamically adjust the difficulty and content of questions based on the test-taker's performance to obtain an accurate assessment with the fewest questions. The steps include initial estimation, question selection, ability update, and termination judgment.

LLM-CAT Technical Architecture and Process

• Technical Architecture: Estimates LLM ability parameters based on Item Response Theory (IRT) models; selects optimal questions via an adaptive question selection algorithm (using Fisher information to measure information gain); supports an online learning mechanism to optimize IRT parameters as data accumulates.
• Evaluation Process: Question bank preparation (collecting and annotating medical questions and estimating IRT parameters) → Model initialization → Adaptive testing (question selection-answering-update cycle) → Result report (outputting ability estimates and confidence intervals).

Cost-Benefit Analysis Results

LLM-CAT can reduce the number of test questions by 50% to 70% while maintaining assessment accuracy, bringing three major advantages:

1. Reduced API Costs: Corresponding reduction in commercial API call fees;
2. Shorter Evaluation Time: Fewer questions mean faster cycles;
3. Environmental Friendliness: Reduced computing resource consumption and lower carbon footprint. In medical scenarios, cost savings are more important (medical questions require expert review, and the cost of question bank construction and maintenance is high).

Limitations and Challenges of LLM-CAT

1. Question Characteristic Differences: The answering behaviors of human test-takers and AI models are inherently different (humans are prone to carelessness/nervousness, while model errors are related to training data/architecture), affecting the applicability of IRT models;
2. Question Bank Coverage: When the question bank is sparse in certain ability intervals, it is difficult to accurately evaluate models in those intervals;
3. Cold Start Problem: New models/domains lack prior data, making it difficult to establish accurate IRT parameters;
4. Multi-dimensional Capabilities: Medical knowledge is multi-dimensional (diagnosis, treatment, etc.), and single-dimensional IRT models cannot fully capture complex ability structures.

Future Outlook for LLM-CAT

1. Multi-dimensional CAT: Extend IRT models to support multi-dimensional ability assessment and fully characterize model performance;
2. Cross-domain Transfer: Explore the possibility of transferring CAT models between different medical specialties;
3. Integration with Active Learning: Dynamically expand and optimize the question bank;
4. Open Source Ecosystem: Establish an open medical evaluation CAT question bank and toolchain to promote community collaboration.

LLM-CAT demonstrates the innovative application potential of traditional psychometric methods in the field of AI evaluation. By introducing CAT technology, it provides an efficient and economical solution for the medical benchmark evaluation of large language models. As large model technology develops, such evaluation innovations will become an important force driving the progress of the field.