Muon Optimizer: Feature Learning Advantages Surpassing Adam in Robustness and Transferability

Key takeaway: The Muon optimizer learns features during LLM pre-training that are significantly superior to those from Adam and SGD in terms of robustness and transferability. This research is from an arXiv paper (published on June 8, 2026, link: http://arxiv.org/abs/2606.09658v1), and the conclusion is supported by experimental validation and theoretical analysis.

For a long time, Adam and its variants (such as AdamW) have been the mainstream optimizers for LLM pre-training. The Muon optimizer, proposed by Keller Jordan, draws its design inspiration from spectral normalization and orthogonal parameterization. It maintains the geometric structure of parameter updates by performing Singular Value Decomposition (SVD) on the gradient matrix and re-orthogonalizing it. Previous studies focused on Muon's training efficiency (convergence speed, memory usage, etc.), while this study shifts to analyzing feature quality (robustness, transferability).

The research is carried out from two dimensions: robustness and transferability:

1. Robustness test: Evaluate model performance using damaged image and text data with noise, occlusion, etc.;
2. Transferability test: Two approaches—linear classifier transfer with frozen pre-trained parameters, and full model fine-tuning transfer;
3. Auxiliary analysis: Use hierarchical probe classifiers to measure logit margins of each layer, and effective rank to measure feature diversity;
4. Theoretical verification: Design a theoretical model for simplified classification problems to prove Muon's advantages.

Robustness Advantages: Muon features perform better than Adam/SGD on damaged data, and this advantage is universal across Transformer and CNN architectures. Through hierarchical probe classifiers, it is found that Muon's logit margins in each layer are larger, sample distributions are more separated, and it is less sensitive to perturbations. Transferability Advantages: Muon pre-trained models outperform others in both linear classifier transfer and full model fine-tuning transfer. The effective rank of hidden states in each layer of Muon is higher, indicating stronger feature diversity.

The study uses a theoretical model for simplified classification problems to rigorously prove that Muon can achieve larger logit margins and higher effective ranks than Adam/SGD. This advantage stems from Muon's optimization mechanism: spectral normalization and orthogonal parameterization constrain the parameter matrix to maintain a "well-conditioned" state, naturally leading to better feature representations.

Optimizers not only affect training speed but also profoundly determine the quality of features learned by the model. Traditional model evaluation needs to be extended to the dimensions of robustness (stability to input perturbations) and transferability (adaptability to downstream tasks). Muon's feature quality advantages provide a new optimization direction for LLM pre-training.

Muon's computational overhead (mainly from SVD decomposition) may be a consideration for large-scale models. In resource-constrained scenarios, one needs to balance computational cost and feature quality advantages; if pursuing the best model quality, Muon's advantages are worth the additional computational investment.