LMM-Track4D: Multimodal Large Model Empowers 4D Object Tracking and Trajectory Reasoning

The NeurIPS 2026 open-source project LMM-Track4D integrates large language models with multi-view vision to achieve end-to-end 4D object tracking and trajectory reasoning, opening up a new direction for multimodal spatiotemporal understanding. This project breaks through the limitations of traditional 3D detection and tracking, and endows the system with trajectory reasoning capabilities through a vision-language-geometry multimodal fusion architecture, which has broad application prospects in fields such as autonomous driving and robot navigation.

4D object tracking needs to address three major challenges: 1. Multi-view fusion: Single camera has limited perspective, so cross-view consistent association needs to be established; 2. Temporal continuity modeling: Maintain tracking coherence when objects are occluded or motion-blurred; 3. Trajectory reasoning: Traditional methods only output discrete sequences, but real applications require high-level understanding of object intent, future trajectories, and interaction relationships—this is where large language models excel.

The LMM-Track4D architecture consists of three modules: 1. Multi-view visual encoder: Improved ViT + view-aware cross-attention to alleviate ID switching issues; 2. 4D spatiotemporal feature aggregation: Hybrid structure of sparse convolution + temporal Transformer, which updates object representations through a trajectory query mechanism; 3. Large language model reasoning head: Convert 4D features into structured text input to LLM, output tracking results and natural language trajectory analysis (e.g., collision prediction, pedestrian behavior reasoning).

Core technical highlights: 1. Trajectory-aware contrastive learning: Cross-view and cross-time features as positive samples to learn robust identity representations; 2. Temporal self-supervised pre-training: Reconstruct scenes by randomly occluding inputs, and obtain spatiotemporal priors from unlabeled videos; 3. End-to-end differentiable architecture: Joint optimization of gradients across all modules, with visual and language modules evolving collaboratively.

LMM-Track4D performs excellently on datasets such as nuScenes and Waymo: 1. Multi-object tracking (MOT) achieves SOTA, with ID switching rate reduced by about 35%; 2. Trajectory reasoning tasks (future trajectory prediction, anomaly detection, scene description) are significantly better than traditional methods, and human evaluation shows high accuracy and fluency of descriptions.

Application Scenarios: Autonomous driving (understanding the intent of traffic participants), robot navigation (predicting human behavior), sports analysis (device-free motion capture), intelligent monitoring, etc. Limitations: High computational complexity, prediction bias in extreme scenarios. Future Directions: Lightweight architecture for real-time applications, unsupervised/semi-supervised learning to reduce annotation dependency, expansion to complex scenarios such as group behavior analysis.