LanteRn: A New Framework for Structured Visual Reasoning in Latent Space

LanteRn enables multimodal models to perform direct visual reasoning in the latent space. By generating continuous visual thought embeddings, it demonstrates more refined visual understanding capabilities on benchmarks such as VisCoT, V*, and Blink.

Current large multimodal models (LMMs) face challenges in visual reasoning; most can only convert visual content into text descriptions, which is a significant limitation for tasks requiring fine-grained spatial understanding.

LanteRn framework's core breakthroughs:

1. Latent Space Reasoning: Unlike reasoning directly in pixel space or using external tools, LanteRn allows models to reason within compact latent visual representations
2. Visual Thought Embeddings: The model can generate and attend to continuous visual thought embeddings
3. Two-Stage Training: First, anchor visual features to latent states via supervised fine-tuning, then align latent reasoning with task utility through reinforcement learning

It performs excellently on three perception-centric benchmarks:

• VisCoT: Visual Chain-of-Thought Reasoning
• V*: Visual Localization and Understanding
• Blink: Fine-Grained Visual Reasoning

Experiments show that internal latent representations provide a more efficient direction for multimodal reasoning, avoiding pixel-level computational overhead.

This work opens a new path for vision-language models: allowing models to "think" about images in the latent space just as they process language, instead of simply verbalizing visual information into text.