Open-source Large Language Model Fine-tuning for Medical QA: Practices to Enhance Accuracy and Reliability of Medical AI

This article introduces the Open-Source-llm-tuning-for-MED-QA project, which addresses issues such as insufficient professional knowledge and low reliability of general-purpose large language models in medical QA. By fine-tuning open-source LLMs, it enhances their accuracy and reliability in medical question answering, providing a feasible path for medical AI applications.

Medical QA is inherently different from general QA: 1. Extremely high requirements for knowledge accuracy—general models are prone to "hallucinations"; 2. Medical knowledge is highly time-sensitive, and model training data cannot be updated automatically; 3. Complex responsibility attribution requires high interpretability and traceability. Directly applying general models carries risks, so targeted fine-tuning is a necessary approach.

The core goal of the project is to enhance the medical QA capabilities of open-source LLMs through fine-tuning. The technical route includes: data preparation (cleaning and validation of high-quality medical QA datasets), model selection (evaluation of open-source models such as Llama series, Mist, etc.), fine-tuning strategies (full-parameter or parameter-efficient fine-tuning like LoRA), and multi-dimensional evaluation. Advantages of choosing open-source models: low cost, data privacy protection, flexible customization, and high transparency.

Fine-tuning techniques: Full-parameter fine-tuning has excellent performance but high resource requirements; parameter-efficient fine-tuning (LoRA) only trains a small number of adaptation parameters, making it more suitable for scenarios with scarce data in the medical field. Training strategies need to avoid catastrophic forgetting (e.g., EWC technology) and use regularization and early stopping to prevent overfitting.

The evaluation system covers: 1. Accuracy (metrics like exact match, F1 score + expert manual evaluation); 2. Safety (red team testing to identify dangerous requests); 3. Consistency (unified answers to similar questions); 4. Interpretability (prompt engineering or post-processing to require models to cite sources or reasoning processes).

The project's open-source contributions include providing resources such as fine-tuned model code and dataset scripts, lowering the entry barrier for medical AI and promoting community collaboration. Limitations: Cannot fully replace doctors, knowledge update issues, weak handling of rare and complex cases. Future directions: Integrate RAG to access the latest literature, support multi-modality, implement continuous learning mechanisms, and optimize human-computer interaction.