Visual Gradient Steering: Breaking the Optimization Bottleneck in Multimodal Model Knowledge Distillation

Core Insight: Researchers found that the gradients of language priors and visual localization are almost orthogonal in vision-language model distillation, and proposed the Visual Gradient Steering (VGS) method to dynamically adjust the optimization direction, significantly improving the visual reasoning ability of small models.

Basic Information:

• Original author team: Hee Suk Yoon, Eunseop Yoon, Jaehyun Jang, SooHwan Eom, Ji Woo Hong, Mark Hasegawa-Johnson, Qi Dai, Chong Luo, Chang D. Yoo
• Source: arXiv (accepted as ICML 2026 Spotlight)
• Original paper link: http://arxiv.org/abs/2606.00564v1
• Publication date: May 30, 2026
• Code open source: https://github.com/hee-suk-yoon/Decomposed_OPD

In recent years, reasoning models have improved their ability to handle complex tasks through "chain-of-thought", but training faces challenges:

• RLVR (Reinforcement Learning + Verifiable Reward) is used in math/programming fields, but small models have a "cold start" problem (lack of training signals due to sparse rewards).
• On-Policy Distillation uses teacher models to provide dense token-level supervision, overcoming reward sparsity.
• There is insufficient research on distillation optimization dynamics in the multimodal field (vision-language models), which is the problem solved in this paper.

Traditional vision-language distillation uses a monolithic objective function, assuming that language prior matching and visual localization matching are synergistic, but the study found:

1. The loss is decomposed into two independent components: ℒ_Lang (language prior loss) and ℒ_Vis (visual localization loss).
2. The gradient vectors of the two are almost orthogonal (close to 90 degrees), meaning the two objectives are independent in the optimization space. Standard optimizers can only make passive compromises and cannot be optimal for both at the same time.

Based on the finding of gradient orthogonality, the Visual Gradient Steering (VGS) method is proposed:

• Core idea: Dynamically redirect the gradient update direction, prioritizing visual localization (assuming it is the main bottleneck).
• Steps: Decompose loss → Gradient normalization → Visual priority steering (via learnable coefficient γ).
• Mathematical expression: L_VGS = (1-γ) · g_Lang/||g_Lang|| + γ · g_Vis/||g_Vis|| (γ is usually 0.7-0.9).
• Reasons for effectiveness: Break symmetry, alleviate gradient interference, and minimal computational overhead.

Experiments verify the effect of VGS:

• Distillation setup: Teacher model (8B parameters), student models (2B/4B parameters), test tasks cover visual question answering, image understanding, etc.
• Main results: VGS consistently outperforms standard distillation, and the visual reasoning accuracy of 2B/4B models is significantly improved, especially in key tasks of visual localization.
• Training efficiency: No additional teacher inference, no increase in parameters, and negligible computational overhead.

Verify the hypothesis that "visual localization is the bottleneck":

1. Language prior is relatively mature: Small models perform well in pure text reasoning.
2. Visual localization is a shortcoming: Small models lag behind in image details and spatial relation understanding.
3. Improvements from VGS are concentrated on visual tasks, confirming the correctness of the hypothesis. Implication: Under resource constraints, priority should be given to improving visual understanding ability.

Limitations:

• Decreased training throughput: Decomposing the loss and computing two gradient components brings certain overhead.
• Dependence on teacher calibration: The teacher's visual localization bias will be transferred to the student. Future directions:
• Adaptive steering: Automatically adjust the γ coefficient.
• Token-level steering: Apply different steering strengths to different token positions.
• Extend to more modalities: Audio, video, etc.

Practical significance:

1. Importance of geometric perspective: Understand the optimization process from the geometry of gradients and discover problems ignored by traditional methods.
2. Bottleneck identification: Identifying bottleneck tasks in multi-task learning can significantly improve performance.
3. Power of simple methods: VGS does not require complex architecture, and only adjusts the optimization objective to achieve significant results.

Conclusion: VGS deepens the understanding of the essence of multimodal learning, provides an elegant solution for lightweight model training, and is an important progress in multimodal knowledge distillation.