MergeKit: A Training-Free Large Model Merging Tool to Combine the Strengths of Multiple Models

MergeKit is an open-source toolkit developed by Arcee AI. It supports merging multiple pre-trained large language models without additional training, enabling capability fusion and transfer through weight space operations. Its core advantage lies in its out-of-core computing architecture, which has low resource consumption (can run on CPU or only requires 8GB of VRAM), lowering the threshold for large model customization and allowing individual researchers and small teams to participate in model engineering practices.

With the rapid development of large language models (LLMs), different models excel in specific domains, but running multiple models is computationally expensive. Traditional model ensembles require loading and inferring multiple models simultaneously, consuming significant resources; model merging technology, however, generates a single model by merging at the weight level, preserving the comprehensive capabilities of multiple models while maintaining the inference cost of a single model.

Supported Model Architectures

MergeKit supports mainstream language model architectures such as the Llama series, Mistral, GPT-NeoX, and StableLM, and is expanding to more types.

Rich Merging Algorithms

It implements multiple methods including SLERP (Spherical Linear Interpolation), TIES (Trimming Redundant Parameters and Sign Voting), DARE (Random Dropping and Rescaling), Task Arithmetic, Frankenmerging (Layer Slice Assembly), and evolutionary merging.

Memory Optimization

It uses lazy tensor loading technology, loading parameters only when needed, significantly reducing memory usage and allowing consumer-grade hardware to handle models with billions of parameters.

LoRA Extraction

Can extract LoRA adapters from full models, facilitating model fine-tuning and efficient deployment.

MoE Merging

Supports merging multiple dense models into a Mixture of Experts (MoE) architecture, expanding model capacity while maintaining inference efficiency.

Tokenizer Transplantation

Provides the mergekit-tokensurgeon tool to handle the transplantation and merging of tokenizers from different models, avoiding tokenizer mismatch issues.

Multi-Stage Merging

Supports multi-stage pipelines (mergekit-multi), chaining multiple merging operations to build finely customized processes.

Application Scenarios

• Capability Integration: Merge code generation and dialogue models to get an all-round assistant;
• Domain Adaptation: Merge general-purpose models with domain-specific models to enhance professional performance;
• Behavior Tuning: Merge models with different behavioral characteristics to find a balance for requirements;
• Knowledge Transfer: Transfer specific capabilities to other architectures without original data.

Usage

Generate models by providing a YAML configuration file via the mergekit-yaml command, supporting CUDA acceleration and lazy loading.

Community Ecosystem

Has an active open-source community, with supporting tools like the FrankensteinAI browser hosting service, and the community maintains a merged model leaderboard.

MergeKit lowers the technical threshold for large model customization, promoting the democratization of model merging technology and allowing small teams to participate in cutting-edge practices. In the future, with the rise of multimodal models and Agent systems, model merging is expected to play a core role in scenarios such as vision-language integration and multi-Agent coordination.

1. Merging Uncertainty: Differences in model architectures and data may lead to unpredictable results;
2. Capability Conflicts: Some mutually exclusive capabilities may cause performance degradation after merging;
3. Evaluation Challenges: Need to cover multi-dimensional test benchmarks to evaluate merged models;
4. License Compliance: Pay attention to compatibility when merging models with different open-source licenses.