Trinity-RFT: A Unified Framework for Reinforcement Fine-Tuning of Large Language Models

Trinity-RFT is a general-purpose reinforcement fine-tuning framework open-sourced by the AgentScope team. It unifies support for multiple RFT modes (synchronous/asynchronous, online/offline, on-policy/off-policy) through a decoupled three-component architecture, providing a one-stop solution for agent developers, RL researchers, and data engineers. It has been implemented in real business scenarios and is continuously iterated.

With the rapid improvement of LLM capabilities, reinforcement fine-tuning has become an important research direction in the AI field. However, existing tools face issues such as difficulty switching training modes, tight coupling between agent interaction and training, and lack of systematic design for data pipelines. Trinity-RFT was open-sourced in April 2025 to provide a general and flexible RFT framework. It has released version v0.5.2 and been implemented in businesses like Taobao Flash Sale.

Trinity-RFT decouples the process into three independent components:

• Explorer: Interacts with the environment to generate experience data, supporting synchronous/asynchronous, single/multi-turn dialogues, and complex agent workflows;
• Trainer: Updates model weights based on experience, supporting algorithms like GRPO, PPO, DPO, and compatible with distributed training backends and Tinker backend;
• Buffer: Acts as a data hub, responsible for data cleaning, enhancement, filtering, and formatting, supporting advanced features such as human-machine collaborative annotation.

1. Unified RFT Mode Support: Abstracts multiple training modes through the RFT-Core module, which can be switched via configuration;
2. Efficient Agent-Environment Interaction: Decouples reasoning and interaction, supporting multiple environment types and optimization for multimodal scenarios;
3. Extensible Algorithm Platform: Plugin-based design with built-in implementations of cutting-edge algorithms like CHORD and BOTS, facilitating R&D sharing.

1. Taobao Flash Sale's Healthcare Business: Empowered this business in December 2025; AI agents can understand vague symptoms and recommend products accurately;
2. CoPaw-Flash Model: Released in March 2026, a localized small agent model that has been open-sourced on ModelScope and HuggingFace.

Optimized for three types of users:

• Agent Developers: Low-code setup of end-to-end training pipelines;
• RL Researchers: Modular algorithm interfaces to focus on logic implementation;
• Data Engineers: The Buffer component provides rich data operation operators.

Trinity-RFT provides detailed documentation and tutorials, supports PyPI installation and Docker images; sample code covers multiple scenarios; the technical report has been published on arXiv.

Trinity-RFT addresses the pain points of existing tools through its decoupled architecture, supporting multiple training modes and targeted functions. As LLMs develop, reinforcement fine-tuning will become an important optimization method, and the framework is expected to accelerate the progress of related research and applications.