Optimizer-Model Consistency: Using the Same Optimizer for Pre-training and Fine-tuning Reduces Knowledge Forgetting

Studies have found that using the same optimizer for pre-training and fine-tuning achieves a better learning-forgetting trade-off, outperforming parameter-efficient fine-tuning methods like LoRA; it reveals the regularization effect of optimizers on model activations, and finds that the Muon optimizer has a tendency to rote memorize when fine-tuned on reasoning tasks.

Training large language models involves two stages: pre-training (general knowledge) and fine-tuning (specific tasks). However, fine-tuning often leads to catastrophic forgetting. Existing mitigation solutions like PEFT (e.g., LoRA) sacrifice performance, regularization methods are complex, replay is costly, and learning rate adjustments slow down the process—all have limitations.

Optimizer-Model Consistency refers to when the same optimizer is used for pre-training and fine-tuning, the model forgets less and has strong task performance competitiveness. Experiments show that full fine-tuning with the same optimizer significantly outperforms using different optimizers and LoRA methods in terms of forgetting metrics.

1. Activation Regularization Effect: Optimizers leave unique activation distribution "fingerprints"; using the same optimizer maintains activation statistical properties, reducing adaptation costs. 2. Weight Update Structure Constraints: The update directions of the same optimizer are more compatible, momentum is continuous, and adaptive learning rates are stable, reducing conflicting updates.

• Muon performs worse than AdamW when fine-tuned on reasoning tasks, has a tendency to rote memorize, and overfits on small datasets. - AdamW has strong generalization ability and robust hyperparameters. - Synthetic language modeling experiments: Muon excels at memorization but is weak at rule learning; AdamW balances memorization and generalization.

1. Prioritize keeping the pre-training and fine-tuning optimizers consistent. 2. Use Muon cautiously (for scenarios with small datasets or strong generalization requirements). 3. Reassess the value of full fine-tuning (forgetting may be less than LoRA when optimizers are consistent). 4. If consistency is not possible, reduce learning rate or weight decay to mitigate forgetting.

Limitations: Limited optimizer coverage (only AdamW/Muon), limited task types (language modeling/reasoning), unvalidated scale effects, insufficient theoretical depth. Future Directions: Optimizer fingerprint analysis, dynamic optimizer switching, meta-optimizer learning, neural architecture search, development of fine-grained forgetting quantification metrics.