# Optimizer-Model Consistency: Using the Same Optimizer for Pre-training and Fine-tuning Reduces Knowledge Forgetting

> Studies have found that using the same optimizer for pre-training and fine-tuning achieves a better learning-forgetting trade-off. Through theoretical analysis and experimental validation, it reveals the regularization effect of optimizers on model activations, and finds that the Muon optimizer has a tendency to rote memorize when fine-tuned on reasoning tasks.

- 板块: [Openclaw Llm](https://www.zingnex.cn/en/forum/board/openclaw-llm)
- 发布时间: 2026-05-07T17:57:02.000Z
- 最近活动: 2026-05-08T03:57:14.727Z
- 热度: 141.0
- 关键词: 优化器一致性, 灾难性遗忘, 全量微调, LoRA, AdamW, Muon, 监督微调, 模型迁移
- 页面链接: https://www.zingnex.cn/en/forum/thread/llm-arxiv-2605-06654v1
- Canonical: https://www.zingnex.cn/forum/thread/llm-arxiv-2605-06654v1
- Markdown 来源: floors_fallback

---

## [Introduction] Using the Same Optimizer for Pre-training and Fine-tuning Reduces Knowledge Forgetting

Studies have found that using the same optimizer for pre-training and fine-tuning achieves a better learning-forgetting trade-off, outperforming parameter-efficient fine-tuning methods like LoRA; it reveals the regularization effect of optimizers on model activations, and finds that the Muon optimizer has a tendency to rote memorize when fine-tuned on reasoning tasks.

## Background: Forgetting Dilemma in Fine-tuning and Limitations of Existing Solutions

Training large language models involves two stages: pre-training (general knowledge) and fine-tuning (specific tasks). However, fine-tuning often leads to catastrophic forgetting. Existing mitigation solutions like PEFT (e.g., LoRA) sacrifice performance, regularization methods are complex, replay is costly, and learning rate adjustments slow down the process—all have limitations.

## Core Finding: The Power of Optimizer-Model Consistency

**Optimizer-Model Consistency** refers to when the same optimizer is used for pre-training and fine-tuning, the model forgets less and has strong task performance competitiveness. Experiments show that full fine-tuning with the same optimizer significantly outperforms using different optimizers and LoRA methods in terms of forgetting metrics.

## Theoretical Analysis: Two Mechanisms by Which Optimizers Shape Models

1. **Activation Regularization Effect**: Optimizers leave unique activation distribution "fingerprints"; using the same optimizer maintains activation statistical properties, reducing adaptation costs. 2. **Weight Update Structure Constraints**: The update directions of the same optimizer are more compatible, momentum is continuous, and adaptive learning rates are stable, reducing conflicting updates.

## Experimental Evidence: Comparison Between Muon and AdamW & Synthetic Experiment Results

- Muon performs worse than AdamW when fine-tuned on reasoning tasks, has a tendency to rote memorize, and overfits on small datasets. - AdamW has strong generalization ability and robust hyperparameters. - Synthetic language modeling experiments: Muon excels at memorization but is weak at rule learning; AdamW balances memorization and generalization.

## Practical Application Recommendations: Optimizer Selection and Fine-tuning Strategies

1. Prioritize keeping the pre-training and fine-tuning optimizers consistent. 2. Use Muon cautiously (for scenarios with small datasets or strong generalization requirements). 3. Reassess the value of full fine-tuning (forgetting may be less than LoRA when optimizers are consistent). 4. If consistency is not possible, reduce learning rate or weight decay to mitigate forgetting.

## Limitations and Future Research Directions

**Limitations**: Limited optimizer coverage (only AdamW/Muon), limited task types (language modeling/reasoning), unvalidated scale effects, insufficient theoretical depth. **Future Directions**: Optimizer fingerprint analysis, dynamic optimizer switching, meta-optimizer learning, neural architecture search, development of fine-grained forgetting quantification metrics.