Multimodal Protein Language Model: An AI Prediction System Fusing Sequence and Structural Information

A multimodal protein model based on the Transformer encoder-decoder architecture, which combines Mixture-of-Experts and image encoding to enable sequence-to-structure/function prediction

Proteins are the core executors of life activities, and their functions are mediated by the three-dimensional structure determined by the amino acid sequence. Traditional protein structure prediction methods such as X-ray crystallography and cryo-electron microscopy (cryo-EM) are highly accurate but expensive and time-consuming. The rise of computational methods, especially the breakthrough of AlphaFold, has opened a new path for high-throughput protein structure prediction.

However, the challenges in protein research go far beyond structure prediction. Understanding protein functions, predicting interactions, and designing new proteins all require more comprehensive information integration. Multimodal protein language models have emerged in this context—they can not only process sequence information but also fuse multimodal data such as structural images to achieve more accurate predictions.

This project implements a complete multimodal protein prediction system using an encoder-decoder architecture, with core components including:

The encoder is built based on Transformer layers but introduces the Mixture-of-Experts (MoE) mechanism to enhance expressive power. The specific structure includes:

• Embedding layer: Converts amino acid sequences into dense vector representations
• Positional encoding: Uses sine/cosine positional encoding to capture sequence order information
• Multi-layer encoder layers: Each layer contains a multi-head self-attention mechanism and an MoE feedforward network

The introduction of the MoE layer is a key innovation of the architecture. Unlike the single feedforward network in standard Transformers, MoE uses multiple expert networks and selects the most suitable combination of experts for each input token through a gating mechanism. This both increases model capacity and maintains computational efficiency.

The decoder also adopts the Transformer architecture and is responsible for generating structure label sequences from the encoder output. Its features include:

• Masked self-attention: Ensures causality in autoregressive generation
• Encoder-decoder cross-attention: Introduces sequence information from the encoder into the decoding process
• MoE feedforward network: Consistent expert mixing mechanism with the encoder

The decoder outputs structure labels (e.g., secondary structure types: alpha helix, beta sheet, random coil) instead of directly predicting 3D coordinates. This level of abstraction is more suitable for understanding the functional properties of proteins.

This is the highlight of the model. The system optionally accepts structural image inputs (such as 2D structure diagrams) and extracts visual features via a dedicated image encoder:

• Image encoder: 3 layers of Conv2D + MaxPool structure, compressing images into fixed-dimensional feature vectors
• Feature fusion: Concatenates sequence features and image features, then maps back to the model dimension via a projection layer

This design allows the model to use visual cues to assist prediction beyond sequence information. For example, some structural patterns are obvious in images but difficult to directly identify in sequences.

The project implements multiple key components in layers.py:

MultiheadAttention: Standard Transformer attention mechanism, supporting dropout and layer normalization.

ExpertLayer: A simple feedforward network, serving as the building block of MoE.

MixtureOfExperts: Implements the gate routing mechanism, selecting Top-k experts for weighted combination for each token.

positional_encoding: Generates a sine/cosine positional encoding matrix.

The project implements the learning rate strategy from the Transformer paper:

• Warmup phase: Linearly increases the learning rate for the first warmup_steps
• Decay phase: Then decays according to the inverse square root of the number of steps

This strategy stabilizes the optimization process in the early stage and fine-tunes parameters in the later stage.