DT2IT-MRM: Debiased Preference Construction and Iterative Training Method in Multimodal Reward Modeling

DT2IT-MRM is an open-source project focused on multimodal reward modeling, maintained by zhang123434. The source code is available at GitHub, and it was released on 2026-05-25T14:39:38Z. This project addresses the reward signal bias problem in multimodal large model training through debiased preference construction and iterative training strategies, providing a new path to improve the alignment quality of multimodal AI systems. Core keywords include multimodal reward modeling, debiased learning, iterative training, AI alignment, etc.

Multimodal Large Language Models (Multimodal LLMs) are reshaping the boundaries of AI, but they face a core challenge in training: how to construct high-quality reward signals to align with human preferences. Traditional methods have two major issues: 1) Inconsistent cross-modal preference annotations (significant differences in judgment criteria between images and text); 2) Systematic biases are easily introduced during dataset collection/annotation (e.g., overrepresentation of certain visual styles or language patterns), leading to cumulative biases in the reward model and outputs that deviate from real human preferences.

The core innovations of DT2IT-MRM are divided into two parts: 1. Debiased Preference Construction

• Bias Identification and Quantification: Establish a multi-dimensional framework (inter-modal bias, group bias, annotator bias) to quantify the degree of bias through statistical analysis and machine learning.
• Dynamic Resampling Strategy: Adaptively adjust data distribution during training (oversample underrepresented patterns, undersample/downweight overrepresented patterns) to maintain diversity and balanced coverage.

2. Iterative Training Strategy

• Iterative Preference Refinement: Initial model training → prediction validation → hard sample mining → data augmentation retraining, with multiple iterations to improve generalization ability.
• Model Ensemble and Consistency Constraints: Maintain a multi-model ensemble, use inter-model consistency to filter noise and improve the reliability of reward signals.

Architecture Design: Modular and scalable, including data preprocessing modules (bias detection, resampling, augmentation), core reward model (support for multiple architectures), iterative training engine (process management), and evaluation visualization tools (performance analysis). Multimodal Fusion Strategies: Explore various methods such as early fusion (feature-stage fusion), late fusion (decision-layer fusion), cross-modal attention (dynamic weight adjustment), and contrastive learning (enhance alignment quality).

Debiasing Effect: Compared to baseline models, the preference prediction accuracy increased by 8-12%, the cross-group fairness gap narrowed by about 40%, and the consistency with human expert annotations improved by 15%. Iterative Training Benefits: The first iteration significantly improved the performance on hard samples; the second iteration enhanced overall generalization ability; the third iteration showed saturated performance with good convergence.

1. Multimodal Content Generation: Help image/video generation models avoid data biases and produce diverse, high-quality content.
2. Multimodal Dialogue Systems: Improve the ability of AI assistants to handle mixed inputs of text, images, and voice.
3. Content Moderation and Recommendation: Build fair and accurate systems to reduce the impact of algorithmic biases on users.

1. Cross-modal Bias Transfer: Study the interaction mechanism of biases between different modalities.
2. Active Learning Integration: Combine active learning to reduce manual annotation costs.
3. Real-time Online Learning: Explore debiasing strategies in online scenarios.
4. Multilingual and Multicultural Expansion: Apply the method to multilingual and multicultural scenarios.

DT2IT-MRM provides a new technical idea for multimodal reward modeling through two core innovations: debiased preference construction and iterative training. It not only improves the prediction accuracy of the reward model but also enhances fairness and robustness, laying the foundation for building reliable multimodal AI systems. With the development of multimodal large models, this method will play an important role in practical applications.