ThinkJEPA: A Dual-Path Embodied Prediction Framework Combining Visual-Language Reasoning Capabilities with Latent World Models

ThinkJEPA proposes an innovative dual-path architecture that combines the Qwen3-VL-Thinking visual-language model (high-level semantic reasoner) with the JEPA branch (low-level dynamic controller) to address the disconnect between high-level semantic reasoning and low-level physical execution in the field of embodied intelligence, opening up new directions for the development of embodied intelligence.

In the field of embodied intelligence, traditional methods often separate high-level semantic reasoning from low-level physical execution: Large Visual-Language Models (VLMs) excel at scene understanding and planning but are weak in handling continuous dynamics and physical consistency; world models like JEPA can capture video dynamics but lack high-level semantic understanding capabilities. This gap is a long-standing challenge.

ThinkJEPA's design is inspired by the division of labor in the human nervous system and includes two core branches:

VLM-Thinker Branch (High-Level Semantic Reasoning)

Based on the Qwen3-VL-Thinking model, it is responsible for high-level semantic understanding of complex scenes, long-range intent planning and reasoning, and providing pyramid-shaped high-level guidance signals.

JEPA Branch (Low-Level Dynamic Control)

Based on the V-JEPA2 architecture, it focuses on modeling continuous dynamics between video frames, maintaining physical consistency and kinematic constraints, and providing fast local correction capabilities.

The two branches collaborate through a conditional mechanism: the JEPA branch receives guidance signals from the VLM branch when predicting future trajectories, enabling seamless integration of high-level intent and low-level execution.

The training process of ThinkJEPA is elaborate, leveraging the complementary characteristics of the two branches:

1. Cache Preprocessing: Use the Qwen3-VL model to extract high-level semantic features from videos and store them as precomputed caches;
2. Dual-Branch Training: The JEPA predictor receives video features and VLM guidance signals to learn to predict future trajectories;
3. End-to-End Optimization: Optimize the entire framework through standard supervised learning.

The project provides open-source implementations, including cache generation scripts, the EgoDex dataset evaluation suite, the Hugging Face cache dataset, and the V-JEPA2 dependency subtree.

The project team provides detailed reproducibility guidelines, supporting two environment configurations:

Training/Evaluation Environment (Python3.11 recommended): PyTorch2.10+CUDA12.8, decord, opencv-python, timm, etc.; Cache Extraction Environment (Python3.10 recommended): transformers5.2.0+qwen-vl-utils, torchcodec for efficient video decoding.

The decoupled design allows users to quickly reproduce results using precomputed caches directly or build the feature extraction process from scratch.

The significance of ThinkJEPA for the field of embodied intelligence:

1. It proves that the reasoning capabilities of visual-language models can be effectively injected into world models, breaking through the limitation of traditional world models' lack of semantic understanding;
2. The dual-path architecture provides a feasible solution for the collaboration between long-range planning and real-time control, suitable for scenarios such as robot manipulation and autonomous driving;
3. Open-source release and detailed documentation lower the barrier to reproducibility, promoting further research in the field.

ThinkJEPA represents an important step forward for embodied intelligence towards a "brain + cerebellum" collaborative architecture. With the improvement of VLM capabilities and advances in world model training technology, this dual-path framework that integrates high-level reasoning and low-level control is expected to become the standard paradigm for next-generation embodied intelligence systems.