PhageBench: A Benchmark for Evaluating Large Language Models' Ability to Understand Phage Genomes

PhageBench is the first benchmark specifically designed to evaluate Large Language Models (LLMs) on their ability to understand phage genomes. It contains 5600 high-quality samples, covers five core tasks, and reveals the potential and limitations of current models in biological sequence reasoning. This benchmark simulates the actual workflow of bioinformatics experts, providing an important platform for evaluating and improving LLMs' biological sequence understanding capabilities.

Phages are known as the 'dark matter' of the biosphere, with an estimated number exceeding 10^31. They play a key role in regulating microbial ecosystems and serving as alternatives to antibiotics. Accurately interpreting their genomes has significant scientific value and practical implications—especially today, with severe antibiotic resistance, phage therapy has become a hot topic in medical research. Traditional analysis relies on professional tools and domain knowledge, with cumbersome and time-consuming processes. After LLMs made breakthroughs in natural language processing, whether they can directly understand nucleotide sequences and perform complex biological reasoning remains an unsolved question.

General-purpose LLMs perform well in biological text understanding, but research on their ability to directly interpret raw nucleotide sequences is insufficient. Existing bioinformatics benchmarks either focus on specific subtasks or fail to comprehensively evaluate models' performance in real-world workflows. To fill this gap, the research team launched PhageBench—the first comprehensive benchmark specifically for evaluating phage genome understanding. Its unique feature is simulating experts' actual workflows, covering the complete analysis chain from raw data screening to functional annotation.

PhageBench contains 5600 high-quality samples, covering three stages and five core tasks: Stage 1 Screening (preliminary identification and classification of phage sequences); Stage 2 Quality Control (sequence integrity check and contamination detection); Stage 3 Phenotypic Annotation (advanced tasks such as host prediction and functional gene identification). This phased design reflects real bioinformatics analysis workflows, ensuring a comprehensive and realistic evaluation of model capabilities.

The research team conducted a systematic evaluation of eight different LLMs. The results show that in tasks like phage sequence identification and host prediction, general reasoning models significantly outperform random baselines, demonstrating their potential in genome sequence understanding. However, when facing complex tasks involving long-range dependencies and fine-grained functional localization (e.g., identifying gene regulatory relationships or the functional impact of distal sequence elements), model performance is clearly limited (with accuracy dropping sharply), revealing the shortcomings of current LLMs in biological sequence reasoning and directions for improvement.

Models face fundamental challenges in handling long-range dependencies: some functional elements in phage genomes are separated by thousands of base pairs but have close regulatory relationships. Human experts can infer this based on domain knowledge, but LLMs struggle to capture such long-distance associations. Additionally, fine-grained functional localization is another difficulty: phage genes are closely arranged or even overlapping, and accurately identifying gene start/end positions and functional contexts requires high precision and biological prior knowledge, which current LLMs are unable to achieve.

The results from PhageBench emphasize the need to develop next-generation models with enhanced biological sequence reasoning capabilities. Key directions include: 1. Improving architectures to capture long-range dependencies (e.g., specialized attention mechanisms or hierarchical modeling); 2. Integrating more biological prior knowledge (codon preferences, gene structure features, etc.); 3. Developing pre-training strategies for biological sequences to learn general biological rules from massive genome data. These directions will promote the development of computational biology and help AI play a greater role in genome annotation, function prediction, and phage therapy design.

PhageBench provides an important benchmark platform for evaluating and improving LLMs' biological sequence understanding capabilities. It not only demonstrates the potential of current technologies but also reveals their limitations. With further research, future AI systems are expected to understand the 'code of life' like human experts, opening up new possibilities for biological science and medical research.