ILVR: Interleaved Latent Visual Reasoning Framework Enables Efficient Multimodal Reasoning

ILVR is an Oral paper accepted by ACL 2026. It proposes an interleaved latent visual reasoning framework, which addresses the efficiency-accuracy dilemma in multimodal large language model reasoning through interleaved latent visual representation and selective perception modeling. It significantly improves computational efficiency while maintaining fine-grained visual reasoning capabilities.

Multimodal large language models (MLLMs) have made significant progress in reasoning capabilities, but there are efficiency bottlenecks: Traditional interleaved reasoning requires re-encoding pixel-dense images, leading to extremely high computational costs. Although latent visual reasoning reduces overhead, existing methods either use a single-step non-interleaved structure that cannot capture intermediate states, or over-compress features at the expense of perceptual modeling capabilities, forming an "efficiency-accuracy" dilemma.

ILVR is proposed by multiple teams including China University of Geosciences and Shanghai Institute of Innovation. Its core innovation is the unification of dynamic state evolution and precise perception modeling. Key insight: Visual representations in reasoning can serve as prompt signals in a compact form in the latent space. By interleaving text generation and latent visual representation, it balances computational efficiency and fine-grained multi-step reasoning.

Interleaved Latent Representation

Each reasoning step generates a latent visual representation as a prompt. This representation is an internal signal autonomously generated by the model, with low dimensionality and low computational overhead. It carries the accumulation of previous reasoning and guides subsequent steps.

Selective Perception Modeling

Adopts a self-supervised selective distillation strategy: The momentum teacher model selectively extracts relevant features from real intermediate images, identifies key visual cues through contrastive learning, and guides the student model to generate focused latent representations.

Training and Implementation

Based on the Qwen2.5-VL-7B-Instruct model, trained on the CoMT dataset for 15 epochs with a gradient accumulation step of 8 and a latent representation dimension of 8. Modified the Transformers library and used HuggingFace Accelerate for distributed training; the code has been open-sourced.

ILVR significantly outperforms existing latent reasoning methods in multimodal reasoning benchmark tests. Its accuracy is comparable to pixel-level methods, and its computational efficiency has achieved an order-of-magnitude improvement. The framework is general-purpose and applicable to multiple scenarios such as visual question answering, image caption generation, and visual navigation.

ILVR opens up a new path for efficient reasoning of multimodal large models, solving the deployment challenges of edge devices or real-time applications. It inspires future model design ideas and promotes the development of lightweight and efficient architectures. Selective perception modeling simulates the characteristics of human visual cognition, which has cognitive heuristic significance.

The ILVR project is open-sourced on GitHub under the MIT license (URL: https://github.com/XD111ds/ILVR). It provides complete code, pre-trained model weights, the CoMT dataset, and detailed documentation to support research and development in the field.