LightFED_MVQA: A Medical Visual Question Answering Framework Combining Federated Learning and Multimodal RAG

LightFED_MVQA is a medical visual question answering system that integrates federated learning and multimodal retrieval-augmented generation (RAG) technology. It uses the Qwen2-VL small language model with 2B parameters to enable collaborative diagnosis across medical institutions while protecting patient privacy, providing a feasible solution for privacy-preserving collaboration in medical AI.

Medical Visual Question Answering (Med-VQA) is a key application area of medical AI, which can assist doctors in image interpretation and lesion localization. However, high-quality annotated medical data is scattered across various institutions, and privacy regulations (such as HIPAA and GDPR) prohibit data from leaving local premises. Traditional centralized training solutions are not feasible, and data silos limit the performance ceiling of medical AI models.

Federated learning solves privacy issues through the principle of 'data stays, model moves'. LightFED_MVQA innovatively combines federated learning, small language models (SLM), and RAG:

1. The core model uses Qwen2-VL 2B, which can run with 8GB of VRAM, lowering hardware barriers;
2. The Shared-Engine architecture solves the memory OOM problem for multiple clients through single-engine initialization + LoRA weight switching;
3. Integrates the FAISS vector database to build a local medical case library. During inference, it retrieves similar cases to enhance diagnosis, reduce hallucinations, and improve interpretability.

LightFED_MVQA sets up four comparative experimental configurations: Centralized+RAG (performance upper limit), Fed+RAG (this paper's solution), Fed-SLM (without RAG), and Fed-LLaVA-Med (13B baseline). Evaluation metrics include Accuracy/F1-Score for closed-ended questions and BLEU/ROUGE-L for open-ended questions. Experiments are executed via modular scripts, and results are saved to specified JSON files for easy analysis.

Core Values: Privacy compliance (data does not leave local premises), cost control (small models lower hardware barriers), knowledge sharing (cross-institution aggregation improves generalization), diagnostic assistance (RAG provides case references); Limitations: The 2B model has limited performance on complex cases, data heterogeneity across institutions needs optimization, FedAvg does not defend against malicious clients, and retrieval inference speed needs improvement.

LightFED_MVQA promotes the evolution of medical AI towards privacy-preserving collaboration and provides engineering references for inclusive deployment. Future explorations can include: optimization of federated training for 7B-13B models, stronger federated personalized algorithms to handle data heterogeneity, introduction of secure aggregation protocols to defend against poisoning attacks, and optimization of retrieval inference real-time performance to meet clinical needs.