ActionJEPA: A Vision-Language-Action Robot Learning System Based on the JEPA World Model

ActionJEPA is a master's thesis project in Artificial Intelligence and Robotics at the University of Rome. It combines the JEPA (Joint Embedding Predictive Architecture) world model proposed by Meta with the Vision-Language-Action (VLA) framework, aiming to improve the learning efficiency and generalization ability of robots in manipulation tasks, and solve the problems of traditional imitation learning such as high data demand and difficulty in handling out-of-distribution task changes.

The field of robot learning has long faced core challenges: how to enable robots to efficiently learn complex manipulation skills from limited demonstration data and generalize to new scenarios. Traditional imitation learning requires large amounts of data and struggles to handle out-of-distribution task changes. In recent years, the combination of world models and VLA frameworks has provided new ideas to address this problem. ActionJEPA is a representative work in this direction, developed by a master's student in the AI and Robotics program at the University of Rome as the core result of their master's thesis.

JEPA is a new world model architecture proposed by Yann LeCun's team. It uses a joint embedding predictive architecture and makes predictions in the representation space rather than the pixel space, offering advantages such as high computational efficiency, strong generalization ability, and good robustness. The VLA framework unifies visual perception, language understanding, and action execution into an end-to-end model, enabling physical operations based on language instructions. The innovation of ActionJEPA lies in the deep integration of the two: using JEPA to learn the environmental dynamics model, and realizing language-conditioned action generation through the VLA framework, which can predict future states and plan action sequences.

ActionJEPA is implemented based on the LIBERO benchmark suite (a robotic manipulation learning evaluation framework). The core technology stack includes Meta's open-source JEPA world model, the LIBERO benchmark, and Hugging Face Transformers. The project solved the weight loading compatibility issue with PyTorch version 2.6+: modified the torch.load call to add the weights_only=False parameter, and provided a fix script to ensure stable system operation.

ActionJEPA is trained using the LIBERO dataset, which includes multiple subsets (total ~100GB): libero_10 (13.7GB, 10 basic tasks), libero_90 (66.7GB, 90 diverse tasks), libero_goal (6.37GB), libero_object (7.44GB), and libero_spatial (6.24GB). Two download methods are supported: script download and Hugging Face Hub download. Training process: load pre-trained visual and language encoders, fine-tune on LIBERO data while training JEPA to learn environmental dynamics; during inference, receive language instructions and visual observations, VLA generates actions, and JEPA predicts future states and plans.

Academic value: explores a new paradigm of combining world models with VLA frameworks, which is expected to reduce the data demand of traditional VLA methods and improve generalization ability to new tasks and environments. Engineering value: provides a complete reproducible codebase (including installation guides, dataset scripts, and training configurations), open-sourced under the MIT license, to promote subsequent research and applications.

Potential development directions for ActionJEPA include: expanding to real robot platforms; introducing stronger vision-language pre-trained models; implementing online learning and adaptation after deployment; integrating multi-modal perception fusion such as tactile and force sensing.