Distilling the Reasoning Capabilities of Cutting-Edge Large Models into Local Biomedical Models: Technical Analysis of Biomed-IE-Distill-XAI

Project Core

This article introduces the Biomed-IE-Distill-XAI end-to-end biomedical information extraction pipeline project, which transfers the reasoning capabilities of cutting-edge large language models to a lightweight PubMedBERT model via knowledge distillation, enabling secure and localized biomedical text processing while integrating post-hoc explainable AI technologies.

Project Source

• Original author/maintainer: Francesco-Alb
• Source platform: GitHub
• Original title: biomed-ie-distill-xai
• Original link: https://github.com/Francesco-Alb/biomed-ie-distill-xai
• Release/update time: 2026-06-07T08:42:20Z

In biomedical research and clinical practice, extracting structured information from massive literature and medical records is a key but challenging task. Traditional methods need to balance accuracy and efficiency, while modern large language models have strong capabilities but face issues like data privacy, deployment costs, and inference latency. The sensitivity of medical data requires institutions to avoid cloud processing of patient information, leading to a core demand: running high-performance biomedical NLP models locally while maintaining reasoning quality close to cutting-edge large models.

The project's core architecture includes three layers:

1. Data Layer: Construct high-quality annotated datasets using large-scale biomedical literature and clinical texts, covering relationship types such as disease diagnosis, drug interactions, and gene-disease associations.
2. Model Layer: Based on PubMedBERT (pre-trained for biomedical literature with deeper understanding of medical terminology), compress the reasoning patterns of large teacher models into PubMedBERT via knowledge distillation, maintaining lightweightness while possessing strong information extraction capabilities.
3. Explanation Layer: Integrate post-hoc explainable AI technologies to provide decision-making basis, meeting ethical and regulatory requirements in the medical field.

Knowledge distillation is the core technology:

• Let the student model (PubMedBERT) learn the "soft labels" of the teacher model (probability distribution instead of hard classification results) to transfer the understanding of inter-class similarity.
• Adopt a strategy combining response distillation (focusing on final output consistency) and feature distillation (requiring similarity of intermediate layer representations), with dual constraints to ensure the internal mechanism of the student model aligns with the teacher model.

Explainability of medical AI is crucial:

• Integrate post-hoc explainable technologies (attention visualization, SHAP value calculation, LIME local explanation, etc.) to analyze the contribution of input features to decisions.
• Example: When extracting the "drug treats disease" relationship, highlight the drug name, disease name, and connecting verbs/prepositional phrases to verify model rationality and assist error analysis.

• Privacy Compliance: Sensitive medical data does not need to leave the institution's internal network, complying with regulations like HIPAA and GDPR.
• Low Latency: Supports real-time/near-real-time information extraction, suitable for scenarios like clinical decision support and document structuring.
• High Availability: Eliminates network dependency, allowing full AI functionality to be used in offline environments.

• Literature Review: Automatically extract research objects, intervention measures, outcome indicators, etc., to accelerate systematic review writing.
• Clinical Practice: Process unstructured medical records, extract information like diagnoses and medications, supporting precision medicine and adverse drug reaction monitoring.
• Drug Development: Mine compound-target-disease relationship networks to assist new drug discovery and repurposing.

Limitations

• The distillation process requires large computational resources, which is a threshold for teams with limited resources;
• PubMedBERT's knowledge is limited to the time point of training data, leading to blind spots for the latest medical discoveries;
• The output of explainable technologies needs to be transformed into a form easily understandable by clinicians.

Future Directions

• Explore continuous model update and incremental learning mechanisms;
• Optimize the human-computer interaction design of explainable technologies;
• Reduce the computational resource threshold for distillation.