Cambrian-P: A Camera Pose-Guided Video Multimodal Large Model

This article introduces Cambrian-P, a method that enhances the spatial reasoning ability of video multimodal large models (MLLMs) by incorporating camera pose signals. The method adds a learnable camera token and a pose regression head to each video frame, achieving a significant improvement of 4.5-6.5% on spatial reasoning benchmarks such as VSI-Bench, and reaching the state-of-the-art level in ScanNet streaming pose estimation.

Existing video multimodal large models (MLLMs) usually treat frame sequences as independent 2D snapshots, losing geometric correlations between frames and limiting spatial reasoning capabilities. However, when humans watch videos, they automatically integrate spatial information from perspective changes. Camera pose (position and orientation) is a key signal connecting observations across different frames, but it is overlooked by existing models. Cambrian-P proposes a solution to this problem.

Cambrian-P consists of two key components:

1. Learnable Camera Token: An independently learned embedding vector for each frame, encoding the spatial position and orientation of the camera for that frame. It is input into the Transformer along with visual tokens to explicitly utilize pose information.
2. Pose Regression Head: A lightweight prediction module that outputs camera pose estimates. It serves dual functions: supervising the model to learn pose-aware representations and acting as a pose estimator. Additionally, a diverse camera motion sampling scheme is adopted to avoid overfitting.

• Achieved a 4.5-6.5% performance improvement on VSI-Bench (Video Spatial Intelligence Benchmark), verifying the importance of pose signals for spatial reasoning.
• Reached the state-of-the-art level in streaming pose estimation on the ScanNet dataset.
• Also had a positive impact on 8 general video question-answering benchmarks such as MSVD-QA, indicating that spatial perception and semantic understanding complement each other.

Training the model using pseudo-labeled poses automatically extracted from in-the-wild videos not only improved performance on spatial reasoning tasks but also enhanced general video question-answering results. This shows that pose information provides a beneficial inductive bias and does not require expensive precise annotations, lowering the application threshold.

• Sampling Strategy: Ensure exposure to diverse camera motion patterns (translation, rotation, scaling, etc.) during training to prevent overfitting.
• Training Objectives: A multi-task learning framework that jointly optimizes video question-answering loss and pose regression loss to maintain the model's sensitivity to spatial information.

Limitations:

1. Focuses mainly on camera extrinsic parameters (position/orientation), with insufficient modeling of intrinsic parameters (focal length, distortion, etc.).
2. Relies on known or estimable poses, limiting applicability to scenarios where poses are missing. Future Directions:
3. Develop methods for self-supervised learning of pose representations.
4. Jointly model camera pose and dynamic scene content.

Cambrian-P significantly improves spatial reasoning ability through a simple and effective design (camera token + pose regression head), reminding us not to ignore the geometric nature of video understanding. As a bridge connecting 2D observations and the 3D world, camera pose provides an extensible paradigm for integrating geometric priors into video MLLMs, inspiring future research on injecting spatial intelligence into multimodal models.