SOLE-R1: Using Video Language Reasoning as the Sole Reward Signal for Robot Reinforcement Learning

This post introduces SOLE-R1, a video language reasoning model designed specifically for robot reinforcement learning (RL). It generates dense task progress estimates via spatiotemporal chain-of-thought reasoning to serve as the sole reward signal. Notably, SOLE-R1 enables robots to learn 24 unseen manipulation tasks from scratch without real rewards, demonstrations, or task-specific tuning, addressing the reward hacking problem common with general visual language models (VLMs) in RL applications.

VLMs have shown strong capabilities in image understanding and visual QA, inspiring their use to supervise robot learning. However, when top VLMs are used as RL evaluators, they often fail under partial observability and distribution shifts, leading to reward hacking—strategies exploit perceptual errors for fake high rewards instead of solving tasks, which is a core barrier to VLM application in robot RL.

SOLE-R1 (Self-Observing LEarner) is tailored for online RL as the sole reward source, with key features:

1. Spatiotemporal Chain-of-Thought Reasoning: At each time step, it tracks object positions, action progress, and task stages to generate dense task progress estimates.
2. Large-Scale Video Trajectory Synthesis Pipeline: Automatically generates time-anchored chain-of-thought trajectories aligned with continuous progress signals for training.
3. Hybrid Training Framework: Combines supervised fine-tuning (SFT) and reward-verified RL (RLVR) to learn basic reasoning and optimize reward robustness.

SOLE-R1 was tested in 4 simulation environments and real robots:

• Zero-shot Online Learning: Robots start from random policies, learning without real rewards, success indicators, demos, or task-specific tuning.
• 24 Unseen Tasks: Mastered 24 manipulation tasks (grab, place, stack, push/pull) not seen during training.
• Superior to Top VLMs: Outperforms GPT-5 and Gemini-3-Pro in task success rate and shows stronger anti-reward hacking ability (distinguishes real progress from fake surface success).

SOLE-R1's contributions:

1. Free from Real Reward Dependence: Reduces reliance on manually designed real rewards (needing domain expertise and tuning) by using natural language task descriptions.
2. Solve Reward Hacking: Specialized training and spatiotemporal reasoning identify real progress, avoiding deception by surface visual similarities.
3. Towards General Robot Intelligence: Serves as a unified interface for evaluating diverse tasks without task-specific reward design, a step toward general agents.

SOLE-R1 has room for improvement:

• Computational Overhead: Spatiotemporal reasoning on multi-frame videos is costlier than single-frame VLMs; future work may use model compression or efficient inference.
• Complex Long Tasks: Accuracy of progress estimates for tasks needing hundreds of steps can be improved (e.g., combining with hierarchical RL).
• Real-World Generalization: Needs further research on broader scenarios (different lighting, object categories).

SOLE-R1 is a key breakthrough in robot learning. By specializing video language reasoning as RL's sole reward signal, it solves general VLMs' failure in RL and opens a path to more general, autonomous robot learning. Such systems bridging high-level semantic understanding and low-level control will play a critical role in building truly intelligent robot assistants as embodied intelligence advances.