Alibaba Open-Sources ROLL: A New Choice for Reinforcement Learning Training Frameworks of Large Language Models

Alibaba has open-sourced ROLL (Reinforcement Learning Optimization for Large-scale Learning), an efficient, easy-to-use, and scalable framework designed specifically for reinforcement learning (RL) training of large language models (LLMs) on large-scale GPU clusters. It addresses key pain points in LLM RL training, including complex resource scheduling, scalability bottlenecks, and high development barriers. It supports multiple training paradigms, integrates advanced acceleration technologies, and is compatible with multiple hardware platforms, providing a powerful tool for tech pioneers, algorithm developers, and researchers.

As demand for LLMs grows in scenarios like reasoning, human preference alignment, and multi-turn agent interactions, RL-based post-training has become a critical component. However, there are three major challenges:

1. Complex resource scheduling: Need to coordinate heterogeneous tasks such as generation, training, and reward calculation;
2. Scalability bottlenecks: Distributed expansion from single-machine multi-GPU to hundreds or thousands of GPUs requires fine-grained parallel strategies;
3. High development barriers: Existing frameworks require in-depth understanding of underlying distributed principles, making rapid experimental iteration difficult.

ROLL adopts a single-controller architecture, abstracting the distributed training process into unified control logic so that developers do not need to focus on underlying details. The framework divides into multiple roles: Actor (generates rollout data), Trainer (updates parameters), Reward Model (calculates rewards), and Environment Worker (interacts with Agentic RL environments), with flexible resource allocation implemented based on Ray. In addition, it deeply integrates acceleration technologies: Megatron-Core (large-scale training), vLLM/SGLang (efficient inference), FSDP2 (data parallelism), and GPU partial overlapping computation (reduces idle time); it introduces a Rollout Scheduler to manage sample lifecycles and solve the long-tail rollout problem.

ROLL supports multiple training paradigms:

• RLVR: A mainstream post-training paradigm that optimizes models via verifiable rewards, supporting Qwen2.5, Qwen3, Qwen3-MoE, and Qwen3.5 series models;
• Agentic RL: For multi-turn interactions, supporting synchronous/asynchronous training, step-by-step learning (e.g., GiGPO), and tool usage (compatible with GEM environments);
• Other modes: SFT (Supervised Fine-Tuning), DPO (Direct Preference Optimization), distillation (VLM distillation), and online policy distillation.

ROLL is compatible with multiple hardware:

• NVIDIA GPU: Full support, with optimized configurations for 80GB VRAM;
• AMD GPU: Out-of-the-box Docker images and dedicated configurations;
• Ascend NPU: Support for domestic chips, reducing hardware dependencies. For deployment: It provides single-machine quick start, multi-node distributed deployment, and Alibaba Cloud Function Compute DevPod development environment.

The academic achievements of the ROLL team include:

• APPO: Asymmetric Proximal Policy Optimization, with a mini-critic mechanism to improve reasoning ability;
• Preplan-and-Anchor attention mechanism research;
• RollPacker: Mitigates the long-tail rollout problem;
• ROCK: Supporting open-source ecosystem tools;
• ROME: Open-source Agentic model, introducing the IPA algorithm. These achievements are quickly implemented into the framework, forming a research-engineering closed loop.

ROLL focuses on developer experience:

• Configuration system: YAML-based configuration for declarative definition of complex processes;
• Debugging guide: Detailed troubleshooting documentation;
• Metric tracking: Built-in Tracker and Metrics systems for real-time monitoring of training status;
• Checkpoint management: Supports resuming training from breakpoints and Hugging Face format conversion;
• LoRA support: Parameter-efficient fine-tuning to reduce VRAM requirements.

ROLL is an important contribution of Alibaba in the field of LLM infrastructure, connecting academic research and industrial practice:

• For tech pioneers: A large-scale training solution with controllable costs and strong fault tolerance;
• For algorithm developers: Flexible workflow control capabilities;
• For researchers: An agile environment for experimental iteration. In the future, ROLL will continue to support the Qwen3.5 series, improve VLM training, and adapt to domestic hardware, becoming an important infrastructure for RL training in the Chinese LLM community, and is worth the attention and trial of developers.