LMM-Track4D: Unleashing 4D Dynamic Reasoning Capabilities of Multimodal Models via Trajectory-Anchored Dialogue

This article introduces the LMM-Track4D model, which addresses the capability gap of multimodal models in 4D (3D space + time) continuous spatiotemporal dynamic reasoning through a trajectory-anchored dialogue paradigm. The model integrates three core technologies: RTGE Ray-Time Geometric Encoding, TRK Long-Range Dynamic State Tokens, and OSK-RA Object Slot Kinematic Residual Anchoring Decoder, and releases the Track4D-Bench benchmark dataset, providing a systematic framework for evaluating 4D reasoning capabilities.

In recent years, large multimodal models (LMMs) have made significant progress in image understanding and video analysis, but they perform poorly in complex scenarios that require continuous tracking of objects' 3D spatial changes over time. 4D dynamic reasoning capability is a core requirement for practical applications such as autonomous driving and robot navigation. Existing models struggle to maintain accurate tracking and reasoning of objects' long-term motion trajectories, limiting their application in continuous spatiotemporal understanding tasks.

The research team proposes a trajectory-anchored multi-turn spatiotemporal dialogue task paradigm, requiring the model to answer spatiotemporal queries and return structured 3D target trajectories. Based on this, the Track4D-Bench benchmark is constructed, which includes 526 segment-level dialogue samples, 23,500 frames of video data, and 7,500 object annotations, covering real-world challenges such as occlusion and perspective changes, ensuring that the evaluation reflects real application performance.

LMM-Track4D integrates three key technologies: 1. RTGE Ray-Time Geometric Encoding: Treats pixels as camera rays, tracks intersections with objects in the time dimension, and unifies spatiotemporal representation; 2. TRK State Tokens: Streaming state tokens propagate object dynamic information across frames, retain long-term memory through a gating mechanism, and handle issues like occlusion; 3. OSK-RA Decoder: Object slots decompose the scene, kinematic modeling ensures the physical rationality of trajectories, and the residual anchoring mechanism improves robustness under occlusion and perspective changes.

Experiments on Track4D-Bench show that LMM-Track4D consistently outperforms strong baseline models. Key findings include: Explicit dynamic state modeling effectively unleashes 4D reasoning capabilities; The synergy of RTGE, TRK, and OSK-RA components is greater than the sum of their individual effects; The model shows significant robustness in occlusion and perspective change scenarios, with the residual anchoring mechanism of OSK-RA playing an important role.

LMM-Track4D significantly improves the 4D dynamic reasoning capabilities of multimodal models through the trajectory-anchored dialogue paradigm and three technical innovations. This work not only provides a strong baseline model but also establishes a systematic benchmark framework for evaluating 4D reasoning capabilities, laying the foundation for subsequent research. As multimodal models are deployed in physical world applications, 4D reasoning will become an important research direction.

Application prospects include autonomous driving (improving perception accuracy), robot navigation (supporting dynamic environment interaction), motion analysis (extracting athlete trajectories), and AR/VR (enhancing immersive experiences). Limitations include: focusing on rigid object tracking, high computational cost, and the need to expand benchmark coverage. Future directions include extending to complex object types, combining language instruction for tracking, exploring self-supervised learning, and optimizing real-time reasoning strategies.