RAG-BioCompare: A RAG-Enhanced Benchmark Evaluation of Large Language Models in Bioinformatics

The RAG-BioCompare project aims to compare the performance of large language models (LLMs) with and without Retrieval-Augmented Generation (RAG) technology through systematic benchmark testing, in order to find the optimal AI solutions for bioinformatics and omics data analysis. The project focuses on the unique needs of the bioinformatics field, addresses the challenges of applying general-purpose LLMs to professional domains, and provides data support for model selection and technical optimization.

General-purpose LLM evaluation benchmarks (such as MMLU) struggle to accurately measure model performance in the bioinformatics field, which is characterized by highly specialized terminology, strict scientific accuracy requirements, and complex multimodal data. The project was initiated to address the following questions: What is the baseline performance of general-purpose LLMs on bioinformatics tasks? Can RAG compensate for the lack of domain knowledge? What are the performance differences between different model architectures and scales? How to select the optimal model configuration?

Evaluation Design: The core adopts comparative experiments to assess the performance difference between original LLMs and RAG-enhanced LLMs on the same test set, isolating the RAG gain. Tasks cover typical scenarios such as gene function annotation, protein structure prediction Q&A, and disease-gene association analysis. Metrics include answer accuracy, citation accuracy, answer completeness, and hallucination rate.

RAG Architecture: It includes knowledge base construction (integrating authoritative data sources like NCBI Gene and UniProt and converting them into vectors), retrieval module (optimizing professional terminology processing), and generation module (generating factual answers by combining retrieval context).

The project tests multiple representative LLMs, covering open-source/commercial models, different scales, and architectures (such as dense Transformer and mixture-of-experts architectures). The diverse selection aims to answer: Do bioinformatics applications require larger models? What is the gap between open-source and commercial models? Can RAG narrow the model performance gap?

Value: RAG significantly improves factual accuracy and reduces hallucination risks; it enhances the performance of small models and narrows the gap with large models; domain-adapted knowledge bases perform better than general knowledge bases.

Limitations: Retrieval quality determines generation quality (recalling irrelevant documents introduces noise); context length limitations (requiring trade-offs in document compression); high knowledge base update costs (bioinformatics knowledge updates rapidly).

Practical Guidance: Provides model selection references for research institutions, points out component optimization priorities for platform developers, demonstrates the potential and boundaries of LLMs in assisting scientific research for researchers, and offers best practices for knowledge base construction, retrieval tuning, and prompt engineering.

Future Directions: Multimodal RAG (incorporating multimodal data such as sequences/structures), specialized domain models, dynamic knowledge update mechanisms, and human-machine collaboration interfaces.

The project also explores the application of responsible AI in the scientific field to ensure the system is reliable, interpretable, and auditable.