MGDA-Decoupled: Geometry-Aware Multi-Objective Optimization for Fair LLM Alignment

MGDA-Decoupled is a geometry-aware multi-objective optimization algorithm designed specifically for the DPO framework. It aims to balance multiple objectives in LLM alignment (e.g., usefulness, truthfulness, harmlessness) and avoid procedural unfairness caused by fixed weights. This method achieves the highest win rate on the UltraFeedback dataset, providing a new path for building fair and balanced AI systems.

LLM alignment needs to satisfy multiple objectives such as usefulness, truthfulness, and harmlessness simultaneously, but these objectives often have tensions (e.g., over-emphasizing harmlessness may impair usefulness). Traditional fixed scalarization methods convert multi-objective into single-objective optimization, which has three major flaws: procedural unfairness (systematically underestimating hard-to-optimize objectives), lack of adaptability (unable to dynamically adjust objective weights), and insufficient exploration of the Pareto frontier (only finding one point on the frontier).

The core idea of MGDA-Decoupled is to decouple the convergence dynamics of different objectives, allowing each objective to adjust its optimization pace according to its own characteristics. Its geometry-aware optimization steps include: 1. Calculate gradients for each objective; 2. Analyze the angle, magnitude, and convergence state of the gradients; 3. Dynamically assign weights based on geometric analysis; 4. Compute a shared descent direction to update parameters. This method runs entirely within the DPO framework, without the need for complex RL training or additional reward models, and its computational overhead is comparable to standard DPO.

Evaluations on the UltraFeedback dataset show that MGDA-Decoupled performs excellently: 1. It achieves the highest overall win rate, surpassing all comparison methods; 2. It balances all objectives well, with leading or near-leading win rates in individual dimensions; 3. Fairness validation: It performs prominently on hard-to-optimize objectives, avoiding procedural unfairness. The evaluation metric is the win rate against golden responses (both overall and per objective dimension).

Method	Framework	Multi-Objective Handling	Key Advantages	Key Limitations
DPO	DPO	Single-objective	Simple and efficient	Cannot handle multi-objective
MODPO	DPO	Scalarization	No RL required	Unfair fixed weights
GAPO	RL	Geometry-aware	Considers convergence dynamics	Complex RL training
MGDA-Decoupled	DPO	Geometry-aware	Lightweight + fair	Requires parameter tuning
MGDA-Decoupled's unique positioning is that it introduces geometry-aware multi-objective optimization capabilities while maintaining the lightweight nature of DPO.

The technical contributions of MGDA-Decoupled include: 1. Convergence dynamics modeling: Explicitly considers the convergence state of each objective to achieve balanced optimization; 2. Conflict detection and resolution: Intelligently handles gradient direction conflicts and finds Pareto improvement directions; 3. Adaptive learning rate: Assigns a larger effective learning rate to slow-converging objectives to avoid being ignored.

The value of MGDA-Decoupled for LLM alignment practice: 1. Value balance: Avoids dominance by a single value and achieves neutral system behavior; 2. Safety-capability trade-off: Ensures safety objectives are not eroded by capability optimization while not sacrificing usefulness; 3. Multilingual/multicultural adaptation: Adapts to changes in objective importance in different cultural contexts without the need to redesign weights.

Limitations: Hyperparameter sensitivity (requires tuning experience), insufficient theoretical analysis (convergence and optimality guarantees need improvement), multi-objective expansion (effectiveness needs verification for 10+ objectives), dynamic objectives (cannot handle addition/removal of objectives). Conclusion: MGDA-Decoupled is an important advancement in multi-objective alignment for LLMs, proving that fair multi-objective optimization can be achieved within the DPO framework. As AI's social role grows, its follow-up work is expected to support the fairness and comprehensiveness of AI value alignment.