Rveda: A Rigorous Benchmark Environment for Evaluating AI Medical Coding Agents

Rveda is a benchmark environment for evaluating AI medical coding agents. Its core goal is to test whether large language model agents can accurately complete ICD-10 coding through retrieval and verification processes in human-machine collaboration scenarios, instead of directly generating potentially hallucinatory labels. It focuses on evidence-based clinical reasoning capabilities rather than mere label recall, aiming to address the hallucination or over-aggressiveness issues of AI models in medical coding caused by the pursuit of surface accuracy.

Medical coding is a key process that converts clinical diagnoses and procedures into standardized codes, affecting hospital revenue cycle management, insurance claims, and medical data analysis. The fundamental problem faced by AI automatic coding is: benchmarks that simply reward final label accuracy may train wrong behaviors—models may maximize surface specificity through hallucination or over-aggressiveness, lacking factual basis.

The cost of incorrect coding is high: An analysis by UC San Diego and Health Affairs predicts that aggressive diagnostic coding intensity may lead to over $200 billion in excess Medicare payments within a decade; a Zinnov report predicts that U.S. medical revenue cycle management spending will reach $200-210 billion by 2029. Inaccurate coding decisions can evolve into real financial and operational losses.

The core research question of Rveda (Rigorous Evaluation Environment for Agentic Medical Coding) is: Can AI agents behave like cautious medical coders rather than one-time label generators? Its design follows four principles: testing clinical reasoning rather than just label recall, testing search efficiency, penalizing hallucination or over-aggressive behavior, and supporting human-machine collaboration audits.

Difference from audit platforms like FraudLens: Rveda is a pre-deployment benchmark that tests the reasoning trajectory of a single AI agent; the latter is post-hoc detection of aggregated billing anomalies across populations. The two are complementary—Rveda ensures the trustworthiness of agents before deployment, while the latter discovers problematic claims after the fact.

Benchmark task flow: Each episode starts with a patient's medical record. The agent completes coding through three actions: SEARCH (query ICD-10 candidates), DETAILS (obtain code details and exclusion notes), and SUBMIT (submit the code), simulating the retrieve-check-submit operational logic.

Three-tier architecture:

1. Local ICD-10 engine: A SQLite-based retrieval backend that provides search_codes and get_code_details functions;
2. Environment and reward logic: An OpenEnv-compatible wrapper that records GradingTrace (difficulty, search history, conflict flags, etc.) to support trajectory analysis;
3. Reference reasoning loop: A deterministic submission process compatible with the OpenAI client, outputting standardized scores.

Rveda's scoring mechanism goes beyond binary judgment and evaluates agents through trajectory analysis:

• Whether submission is made after sufficient search;
• Whether detailed information and exclusion notes of relevant codes are checked;
• Whether Excludes1 conflicts (mutually exclusive codes) are avoided;
• Whether the search strategy is efficient (number of searches vs result quality).

This evaluation can distinguish between agents that 'guess right' and those that truly reason based on evidence—the latter is what the medical coding scenario requires.

Currently, Rveda uses SQLite's ICD-10 mock data and a single-agent loop, and its architecture supports multi-agent experiments (such as retriever-encoder-auditor pipelines). Potential expansion directions:

1. Multi-agent collaboration: Introduce dedicated retrieval and audit agents;
2. Real ICD-10 data: Migrate to complete ICD-10-CM/PCS code sets;
3. Multilingual support: Expand to coding systems in other languages;
4. Human-machine collaboration interface: Develop an interface for doctors/coders to intervene and correct.

Rveda provides a rigorous and reproducible benchmark for evaluating AI medical coding agents. By enforcing the retrieve-check-submit process, it tests evidence-based clinical reasoning ability instead of label memorization. In today's era of widespread medical AI adoption, this evaluation method focusing on reasoning processes is of great significance for ensuring the reliability and safety of AI systems during deployment.