InteractVLM: A New Intelligent Paradigm for 3D Interaction Reasoning from 2D Vision Models

InteractVLM is a research project accepted by CVPR 2025. Its core is to use existing 2D foundational vision-language models (VLMs) to achieve 3D interaction reasoning without relying on expensive 3D sensors or complex multi-view reconstruction. This method opens up new possibilities for fields such as robotic manipulation and augmented reality. Its innovation lies in unlocking the 3D prior knowledge in 2D models through clever design, reducing data and deployment costs.

2D vision models (such as CLIP and SAM) can already extract rich semantic/geometric information, but lack explicit understanding of depth, spatial relationships, and physical interactions. 3D interaction reasoning needs to answer complex questions such as object operability, interaction methods, and hand placement. Existing solutions face limitations such as data scarcity (difficulty in 3D annotation), high computational cost, poor generalization, and complex deployment. The core idea of InteractVLM is to reuse 2D foundational models and achieve 3D reasoning through task adaptation.

The InteractVLM architecture consists of three parts: 1) 2D vision encoder (reusing pre-trained VLMs such as CLIP/LLaVA); 2) Interaction query generator (converting 3D interactions into 2D queryable forms); 3) Reasoning fusion module (integrating multi-view information). Key innovations: Proposing "interaction templates" to decompose complex 3D interactions into atomic 2D queries; solving single-view ambiguity through virtual view synthesis, geometric consistency constraints, and confidence-weighted fusion; training in two stages: interaction concept pre-training (2D image-text pairs) + 3D interaction fine-tuning (limited 3D data, freezing the 2D encoder).

Outstanding advantages of InteractVLM: 1) No need for 3D supervision: Learning 3D reasoning from 2D annotations reduces the data threshold; 2) Interpretability: The reasoning process is transparent, allowing viewing of attention areas and geometric constraints; 3) Zero-shot generalization: Relying on the generalization ability of VLMs to handle unseen objects/interactions; 4) Efficient reasoning: Mainly based on 2D domain computation, which is much faster than traditional 3D methods and suitable for real-time applications.

Application scenarios include: 1) Robotic manipulation planning: Increasing the success rate of grasping unknown objects by 25%; 2) Augmented reality: Achieving interaction understanding with <100ms latency on Hololens/Quest; 3) Human-computer interaction design: Analyzing product usability issues. Quantitative results: Affordance localization accuracy increased by 12% on the AGD20K dataset; functional region prediction IoU reached 0.78 on the CHAIRS dataset; interaction detection F1 score increased by 15% on the EPIC-KITCHENS dataset; cross-dataset generalization performance is better than specialized 3D models.

Current limitations: Depth ambiguity (when texture/geometric cues are insufficient), complex interactions (multi-object/fine-grained actions), dynamic scenes (mainly static images), physical authenticity (occasionally generating infeasible interactions). Future directions: Extending to video input, integrating physical simulators, combining robot active perception, and fusing multi-modal information such as touch/audio.

InteractVLM verifies three major trends: 1) Transfer value of foundational models: 2D models contain 3D priors, eliminating the need to retrain 3D models; 2) New paradigm of representation learning: Using 2D features to carry 3D semantics, blurring the boundary between 2D and 3D vision; 3) Practical AI implementation path: Prioritize 2D solutions (easy to deploy, low cost, mature ecosystem), and introduce 3D sensors only when necessary.

InteractVLM is an important milestone in the field of computer vision, proving that 2D foundational models can understand the possibility of 3D interactions and have transformative potential for applications such as robotics and AR/VR. Its methodology emphasizes the efficiency of reusing existing models and is worth paying attention to. In the future, with the development of multi-modal large models, the boundary between 2D and 3D vision will be further blurred, and AI is expected to read rich 3D interaction information from ordinary photos like humans.