CT-1: A Spatial Intelligence Model for Video Generation That Truly Understands Camera Motion

CT-1 is a joint vision-language-camera model that enables camera-controllable video generation aligned with user intent by transferring spatial reasoning knowledge to video generation tasks, and has released the CT-200K dataset containing 47 million frames. Its core is the two-stage paradigm of "Camera First, Generation Second", which solves the problems of ambiguous camera control and lack of spatial reasoning in existing video generation.

In recent years, the quality of video generation using diffusion models has improved, but the core problem lies in precise camera motion control. Existing methods rely on vague text prompts or predefined parameters, making it difficult to align with user intent; moreover, camera motion involves 3D spatial reasoning, and models lacking this capability tend to produce physically unreasonable motions.

CT-1 adopts the two-stage paradigm of "Camera First, Generation Second": 1. Camera trajectory prediction (inferring intent-aligned trajectories by understanding scene semantics and spatial layout based on reference images and text); 2. Video generation (generating aligned content using the trajectory as a conditional input to the diffusion model. Core components include: a vision-language module (establishing deep associations between images and text), a wavelet-regularized diffusion Transformer (learning in the frequency domain to capture complex trajectory distributions), and a spatially-aware video generation model (ensuring geometric consistency).

The team built the CT-200K dataset (2000+ video sequences, 47 million frames) with features such as carefully selected (clear camera motions), precisely annotated (intrinsic and extrinsic parameters), and diverse scenes (indoor/outdoor/driving, etc.). Experimental validation shows: good generation results for forward/rotational motions in complex scenes; trajectories compatible with existing models like CameraCtrl; driving scene tests demonstrate cross-domain generalization capabilities.

Existing methods are divided into two categories: explicit parameter-based (e.g., CameraCtrl, precise but difficult to handle natural language) and implicit representation-based (e.g., MotionCtrl, flexible but with poor interpretability). CT-1's advantages: explicit trajectory prediction (interpretable and compatible with downstream models), joint vision-language understanding (handling complex intents), and frequency domain learning (wavelet regularization first introduced into trajectory learning).

CT-1 is not open-source yet (planned to be released after the paper is accepted). Future directions: improving real-time performance (supporting interactive applications), long video generation (meeting film production needs), enhancing user interaction (hand-drawn trajectory/keyframe control), and introducing physical simulation (making motions more physically consistent).

CT-1 promotes video generation from "good-looking" to "controllable", which is crucial for film production (shot language), virtual reality (view switching), and autonomous driving simulation (physical camera motion). It demonstrates the value of spatial reasoning, suggesting that explicit spatial understanding is the key to breaking through data-driven bottlenecks.

CT-1 solves the camera control problem in video generation and has made significant progress through the two-stage paradigm, vision-language modeling, and frequency domain learning. We look forward to further development of the community after open-sourcing, driving the technology toward "understanding user intent" and providing new directions for video generation, computer vision, and other fields.