Trinity-RFT: A Universal Reinforcement Fine-Tuning Framework for Large Language Models

Trinity-RFT is a universal Reinforcement Fine-Tuning (RFT) framework designed specifically for large language models, offering flexible and scalable solutions to help developers optimize model performance more efficiently.

With the rapid development of Large Language Models (LLMs), how to make these models better adapt to specific tasks and scenarios has become a key issue. Traditional Supervised Fine-Tuning (SFT) can help models learn output in specific formats, but often fails to fully leverage human feedback to optimize model behavior. Reinforcement Fine-Tuning (RFT) emerged in this context, using reward signals to guide models to learn better strategies.

However, existing RFT tools often have the following pain points: complex configuration, poor scalability, and difficulty integrating with existing training workflows. Trinity-RFT was created to address these issues, providing developers with a truly universal, flexible, and scalable reinforcement fine-tuning framework.

Trinity-RFT is an open-source reinforcement fine-tuning framework developed by the AgentScope team, designed to be the "Swiss Army knife" for reinforcement learning of large language models. The framework adopts a modular architecture, decomposing the RFT process into multiple independently configurable and replaceable components, allowing developers to flexibly combine them according to actual needs.

The core design principles include:

• Universality: Supports multiple reinforcement learning algorithms, not limited to PPO (Proximal Policy Optimization), but also includes cutting-edge methods like DPO (Direct Preference Optimization) and KTO (Kahneman-Tversky Optimization).
• Flexibility: Through a configuration file-driven approach, developers can switch between different reward models, training strategies, and optimizers without modifying code.
• Scalability: Uses a plug-in design, allowing new algorithms and components to be easily integrated into the framework without affecting existing functions.

The architecture of Trinity-RFT can be divided into three layers:

Data Layer: Responsible for data loading, preprocessing, and batch management. Supports multiple data formats, including conversational data, preference pair data, and trajectory data with reward signals. The framework has built-in data validation and cleaning mechanisms to ensure the quality of input data.

Training Layer: This is the core of the framework, implementing multiple reinforcement learning algorithms. In addition to standard PPO, it also supports:

• DPO (Direct Preference Optimization): Directly optimizes using preference pair data without explicitly training a reward model.
• KTO: A human decision model based on prospect theory, better simulating humans' asymmetric perception of gains and losses.
• Online/Offline Hybrid Training: Supports flexible switching between pre-collected data and newly generated data.

Inference Layer: Responsible for model inference and sampling. Supports integration with high-performance inference engines like vLLM and Text Generation Inference, significantly improving training efficiency.

The effectiveness of reinforcement fine-tuning largely depends on the quality of the reward model. Trinity-RFT provides multiple reward modeling solutions:

• Rule-Based Reward: Suitable for tasks with clear evaluation criteria, such as code correctness checks and mathematical problem verification.
• Model-Based Reward: Uses trained reward models or the LLM-as-Judge mode, suitable for open-ended generation tasks.
• Hybrid Reward: Allows combining multiple reward signals to achieve more fine-grained control through weighting or conditional logic.

To meet the training needs of large-scale models, Trinity-RFT natively supports multiple distributed training strategies:

• Data Parallelism: Processes different batches of data in parallel across multiple GPUs.
• Model Parallelism: Splits large models across multiple devices, supporting training of models with tens of billions of parameters.
• Pipeline Parallelism: Assigns different layers of the model to different devices, enabling overlapping of computation and communication.

The framework is compatible with mainstream distributed training libraries like DeepSpeed and FSDP, allowing developers to choose the most suitable solution based on their hardware conditions.

In code generation tasks, traditional SFT can only help models learn the syntax and format of code, but cannot guarantee the correctness of generated code. Using Trinity-RFT, you can:

1. Define a reward function based on unit test pass rates.
2. Allow the model to continuously attempt code generation during training.
3. Adjust the model strategy based on test pass rates.
4. Finally obtain a model that generates higher-quality code.

For dialogue robots, a fine balance between safety and usefulness is often required. Trinity-RFT allows:

• Training reward models using manually annotated preference data.
• Optimizing the model via the PPO algorithm to keep it helpful while avoiding harm.
• Supporting full trajectory optimization for multi-turn dialogues.