TRACER: A New Method for Robust Multimodal Finetuning via Persistent Regularization

TRACER is an innovative method for multimodal model finetuning, aiming to solve the problems of catastrophic forgetting and out-of-distribution (OOD) robustness degradation in traditional finetuning. Its core innovation is the proposal of a weighted moving average (WMA) teacher model to replace the traditional exponential moving average (EMA) teacher, enabling persistent regularization, which improves target task performance while retaining general knowledge from the pre-training phase.

Multimodal pre-trained models (e.g., CLIP) have strong zero-shot capabilities, but when finetuned to downstream tasks, they tend to suffer from OOD robustness degradation and catastrophic forgetting. Traditional regularization methods (such as weight decay, knowledge distillation) lack unified theoretical guidance and have limited effectiveness.

This study was proposed by the Hesam Asad team, published on arXiv (May 28, 2026), with the original title "TRACER: Persistent Regularization for Robust Multimodal Finetuning", and the code repository address: https://github.com/HesamAsad/TRACER.

TRACER establishes a theoretical framework for multimodal contrastive finetuning from a geometric perspective, proving that self-distillation is more effective in knowledge retention. The study reveals that traditional EMA teachers have a collapse problem (regularization effect decays over time), and thus proposes a WMA teacher model with the following characteristics:

1. Persistent regularization force: effectively constrains the student model throughout the process;
2. Unbiased convergence: unbiased optimization in the task subspace, retaining orthogonal general knowledge;
3. Multi-view distillation: combining contrastive learning to constrain model consistency from multiple granularities.

The TRACER architecture consists of four main components:

• Contrastive Encoder: Based on the contrastive learning framework, it learns image-text aligned representations;
• WMA Teacher Network: Uses weighted moving average to aggregate historical information and update slowly;
• Multi-view Distillation Loss: Constrains the consistency between the student and teacher models at the instance level, category level, and global level;
• Task-Adaptive Regularization: Dynamically adjusts the regularization strength, relying on pre-trained knowledge in the early stage and focusing on task optimization in the later stage.

Experiments were conducted on three backbones of CLIP: ViT-B/16, ViT-L/14, and ResNet-50. The results show:

1. Significant improvement in OOD accuracy: The generalization ability of the finetuned model is enhanced;
2. Improved calibration performance: The model's prediction confidence is more accurate;
3. Strong hyperparameter robustness: Good results can be obtained without tedious parameter tuning.

Insights:

• Theoretical analysis can guide the design of practical algorithms;
• The limitations of EMA teachers need to be re-examined;
• Knowledge retention should be refined (selectively retaining key general knowledge).

Application Value:

• Plug-and-play: Easy to integrate into existing CLIP finetuning workflows;
• Computationally efficient: WMA has low additional overhead;
• Widely applicable: Supports tasks such as image classification, retrieval, and visual question answering.

TRACER is an important advancement in the field of multimodal finetuning, providing both a practical algorithm and a theoretical framework to understand the finetuning process. As multimodal models are widely applied, such methods that balance task performance and generalization ability will become increasingly important. The project code has been open-sourced and can be obtained via https://github.com/HesamAsad/TRACER.