Multimodal-RoPEs: Revisiting Multimodal Positional Encoding in Vision-Language Models

Introducing the official implementation of an ICLR 2026 paper, this thread revisits the multimodal positional encoding mechanism in vision-language models and explores more efficient cross-modal positional encoding schemes.

Vision-Language Models (VLMs) are among the most active research directions in the current field of artificial intelligence. These models need to process both text and image data simultaneously, and how to effectively perform positional encoding for these two modalities has always been a core problem plaguing researchers. Traditional large language models use Positional Encoding (PE) to inject sequence order information, among which Rotary Position Embedding (RoPE) is the most successful. However, when applying RoPE to vision-language models, researchers have found some unique problems: Images are usually represented as 2D patch grids, while text is a 1D token sequence; How to align and interact the positional spaces of the two modalities? Is simple concatenation the optimal solution? The ICLR 2026 paper "Revisiting Multimodal Positional Encoding in Vision-Language Models" conducts in-depth research on these issues.

Before diving into the paper, let's first understand the role of positional encoding. The Transformer architecture itself is permutation invariant to inputs, meaning it does not know the order of tokens. The role of positional encoding is to tell the model the position of each token in the sequence. RoPE injects positional information into attention computation via rotation matrices, and its advantages include: handling sequences of arbitrary length; having good extrapolation capabilities; and performing excellently in relative positional encoding.

When RoPE meets multimodal scenarios, the problem becomes complex: 1D vs 2D: Text: 1D sequence, position can be represented by a single integer; Image: 2D grid, position requires two coordinates (x, y); Modality Fusion: How do image patches and text tokens share the positional space? Do we need to design different positional encodings for different modalities? Cross-modal Attention: How to calculate the relative position between image patches and text tokens? What impact does this have on the model's cross-modal understanding ability?

The paper first systematically analyzes the positional encoding schemes used in current mainstream VLMs and finds some overlooked issues: Problems with simple concatenation: Most VLMs use a simple 1D concatenation approach: [Image Patch 1, Image Patch 2,..., Text Token 1, Text Token 2,...]. The problems with this scheme are: The 2D spatial information of images is compressed into 1D; The positional spaces of images and text are not clearly distinguished; The calculation of cross-modal relative positions is not precise enough. Problems with independent encoding: Some other works try to use independent positional encodings for images and text, but this brings difficulties in modality alignment.

Based on in-depth analysis, the paper proposes a series of principles for designing multimodal positional encoding: Principle 1: Preserve modality characteristics. Different modalities have their inherent structural characteristics, and positional encoding should respect these: Text maintains 1D continuity; Images maintain 2D spatial relationships. Principle 2: Unified positional space. Despite different modality characteristics, all tokens should share a unified positional space to enable effective cross-modal attention computation. Principle 3: Explicit cross-modal positions. The model should be able to explicitly perceive the relative positional relationship between image patches and text tokens.

Based on the above principles, the paper proposes an improved multimodal RoPE scheme: 2D RoPE extension: For image patches, use 2D RoPE: Pseudo-code illustration: def apply_2d_rope(patch_embed, pos_x, pos_y): # Apply rotation to x and y directions respectively rotated_x = apply_rope(patch_embed, pos_x) rotated_y = apply_rope(patch_embed, pos_y) return combine(rotated_x, rotated_y). Modality-aware unified space: Through clever design, map two-dimensional image positions and one-dimensional text positions to a unified high-dimensional space: Text position: (t) → mapped to a specific subspace; Image position: (x, y) → mapped to a complementary subspace. Explicit modality identification: Introduce modality type embedding to allow the model to distinguish whether it is processing an image or text.

The paper conducts evaluations on multiple standard benchmarks: Image understanding: VQAv2, GQA, TextVQA; Image-text alignment: Flickr30K, COCO Retrieval; Multimodal reasoning: MMMU, MathVista; Pure text capability: Maintains performance comparable to the original LLM.