Practical Guide to LLM Training Acceleration: In-Depth Comparative Study of LoRA Combined with Three Optimizers

This article focuses on the core challenge of high training costs for large language models, deeply studies the LoRA low-rank adaptation technique, and systematically compares the performance of three optimization strategies—AdamW, Muon, and MeZO—in training acceleration. It provides data support and decision-making references for developers to choose the optimal training configuration.

Large language models now have billions or even hundreds of billions of parameters, leading to extremely high training costs (e.g., GPT-level models require thousands of GPUs running for weeks, costing millions of dollars). Traditional full-parameter fine-tuning needs to update all parameters, with resource consumption comparable to original training, which is unaffordable for most researchers and developers. Therefore, reducing training costs while maintaining performance has become an urgent issue in the AI field.

Core idea of LoRA: Freeze almost all parameters of the pre-trained model and only train a small number of additional low-rank matrices. Assuming weight updates have a low-rank structure, introduce the product of small matrices A and B to approximate weight updates, and only optimize A and B during training. Advantages include: significantly reduced memory usage (no need to store gradients of original weights), no latency when merging low-rank updates during inference, and performance close to full-parameter fine-tuning.

AdamW

A popular optimizer in deep learning, based on Adam with correct weight decay, adaptively adjusts learning rates, effective for sparse gradients and non-stationary objectives. It is a stable and reliable default choice in LoRA training.

Muon

A new optimizer designed for large-scale models. Through efficient second-order information approximation, it improves convergence characteristics while maintaining computational efficiency, potentially bringing advantages in convergence speed and final performance.

MeZO

Uses zero-order optimization technology, requiring only forward propagation without backpropagation, further reducing memory requirements. It is suitable for ultra-large-scale models or memory-constrained scenarios, where its memory advantage can compensate for the drawback of slower convergence.

This study systematically compares the performance of the three optimizers in LoRA training, focusing on key dimensions: convergence speed (number of steps needed to reach target performance), memory efficiency (differences in memory usage), final performance (accuracy on downstream tasks), and stability (training variance and repeatability). The results are of significant value to practitioners: choose MeZO for limited memory, Muon for fast convergence, and AdamW for stability and reliability, helping developers select the optimal configuration based on their scenarios.

Implementation requires controlling variables (consistent hyperparameters such as model architecture, initialization, learning rate scheduling, batch size) to ensure that optimizer differences are the main cause of result differences. For tools, Hugging Face Transformers and PEFT libraries are used to implement LoRA; MeZO may require custom or open-source code. The dataset selection covers multi-task types such as text classification, question answering, summarization, and translation to comprehensively evaluate the performance of optimizers in different scenarios.

1. Provide direct decision-making basis for LoRA users, allowing them to get started quickly without trying each option one by one;
2. Showcase the performance of new optimizers in parameter-efficient fine-tuning scenarios for optimizer researchers, revealing improvement directions;
3. Promote a culture of reproducible research, setting an example of rigorous experiments through open code and detailed experimental configurations.

LoRA technology has democratized large model fine-tuning, and optimizer selection determines training efficiency and effectiveness. This study provides data support for key decisions. Future outlook: new optimizers to accelerate convergence, LoRA variants (AdaLoRA, QLoRA) to expand options, and combination of quantization technology with parameter-efficient fine-tuning to enable ultra-large models to be fine-tuned on personal devices. It is recommended that developers start with this project to cultivate systematic experimental capabilities.