Optimus: Zeroth-Order LoRA Search Infrastructure for Large Language Models

Optimus is an automatic LoRA configuration search framework for large language models developed by plugyawn, based on zeroth-order optimization methods. It aims to solve the problem that manual tuning of LoRA hyperparameters is time-consuming and it's hard to find the optimal configuration. The framework uses finite differences to approximate gradients, significantly reducing the computational overhead of search. While maintaining search quality, it adapts to resource-constrained environments, providing an efficient hyperparameter optimization solution for parameter-efficient fine-tuning of LLMs. (Source: GitHub, Release Date: 2026-05-24)

In LLM fine-tuning, LoRA is a widely adopted parameter-efficient fine-tuning method, but its performance highly depends on hyperparameter choices such as rank, learning rate, dropout rate, and target modules. Traditional manual tuning is time-consuming and it's difficult to find the global optimum. Zeroth-order optimization can optimize the objective function without computing gradients, providing a new possibility for automatically searching optimal LoRA configurations. The Optimus project builds LoRA hyperparameter search infrastructure based on this.

Optimus is an automatic search framework focused on LLM LoRA configurations. Unlike Bayesian optimization or grid search, which require full computational resources for training, it leverages the characteristics of zeroth-order optimization and uses finite differences to approximate gradients, reducing computational overhead. The core design philosophy is to reduce computational resource requirements while maintaining search quality, enabling effective LoRA configuration search even in resource-constrained environments.

Principles of Zeroth-Order Optimization

Estimate gradients via finite differences of function values: ∇f(x) ≈ [f(x + εu) - f(x)]u / ε (u is a random direction vector on the unit sphere, ε is a small perturbation coefficient). Only two forward propagations are needed, no backpropagation required.

LoRA Configuration Space

Parameterized multi-dimensional space: rank (dimension of low-rank matrix), scaling factor (Alpha), target modules (e.g., q_proj/v_proj), dropout rate, learning rate.

Search Strategy

Adaptive search strategy based on zeroth-order optimization is adopted, dynamically adjusting directions, converging to high-quality configuration regions faster than random search.

1. Fine-tuning in Resource-Constrained Environments: Help teams without large-scale GPU clusters find high-quality LoRA configurations within a limited budget, saving resources for final fine-tuning;
2. Rapid Prototype Validation: Evaluate configuration feasibility in a short time, assisting early product decisions;
3. Cross-Model Transfer Learning: Evaluate the response characteristics of new models to LoRA hyperparameters, accelerating the adaptation of new models.

1. Validation Set Selection: Need to use validation data consistent with the target task distribution to ensure the quality of zeroth-order optimization;
2. Search Budget: Although efficient, a reasonable budget is needed to get good results;
3. Configuration Space Limitation: For overly large spaces, it is recommended to perform a coarse-grained search first and then refine high-potential regions;
4. Result Reproducibility: Due to random sampling, it is recommended to run multiple times and take the average result.

Optimus provides an automated tool for LLM LoRA configuration search through zeroth-order optimization, balancing computational efficiency and search quality, and is suitable for resource-constrained scenarios. Its significance lies in promoting the transition of LLM fine-tuning from manual tuning to automated and efficient search, proving the feasibility of zeroth-order optimization in this field, and opening up paths for subsequent directions such as combining with Bayesian optimization and exploring more efficient perturbation sampling. In the future, as the scale of LLMs grows, automated configuration optimization will become more important, and the direction of efficient search with approximate gradients represented by Optimus is expected to play a greater role.