MergeKit: An Open-Source Large Model Merging Tool to Combine Multi-Model Capabilities Without Training

MergeKit is an open-source model merging toolset developed by the Arcee AI team, supporting multiple mainstream model merging techniques such as TIES, SLERP, and DARE-TIES. It allows developers to combine the advantages of multiple models without additional training, reducing the cost of improving model capabilities. The tool is designed to be flexible and easy to use; users can define merging strategies via YAML configuration files, making it suitable for researchers and developers to quickly iterate on merging solutions.

With the booming development of the open-source large language model ecosystem, platforms like Hugging Face have seen thousands of high-quality pre-trained models emerge, each with unique advantages (e.g., code generation, multilingual understanding, domain-specific performance). However, traditional improvement paths (training from scratch or fine-tuning) are costly, leading to the rise of model merging technology—combining the advantages of multiple models by merging parameters without additional training, and without increasing inference costs. The technology has evolved from simple weight averaging to complex task vector operations.

MergeKit is developed by the Arcee AI team and hosted on GitHub. It provides a complete infrastructure supporting multiple merging algorithms (TIES, SLERP, DARE-TIES, etc.). Its core design principles are modularity and composability; users define merging strategies (specify base models, algorithm parameters) via YAML configuration files. The declarative configuration lowers the barrier to experimentation, facilitating rapid iteration of solutions.

• SLERP: Spherical Linear Interpolation, which preserves the geometric structure of high-dimensional parameter spaces, suitable for models of similar architectures trained on different data;
• TIES: Trims redundant parameters, uses sign voting for conflicting parameters, and performs merging operations, effectively handling parameter conflicts;
• DARE: Randomly drops some parameters and rescales them. DARE-TIES combines sparsification and conflict resolution, achieving excellent performance in benchmark tests; Additionally, it supports techniques like task arithmetic and FrankenMerging, and is compatible with mainstream model architectures such as Mistral, Llama, and Qwen.

The value of model merging technology is reflected in multiple dimensions:

1. Resource-constrained teams: Improve model capabilities at low cost (e.g., merging a general base model with a fine-tuned medical domain model to gain both general and medical capabilities);
2. Enterprise customization: Merge public base models with internal domain models to obtain customized AI capabilities while protecting privacy;
3. Research perspective: By observing the effects of merging strategies, gain in-depth understanding of the organization and interaction of knowledge in model parameters, promoting the development of explainable AI and model editing.

MergeKit offers two usage methods: command-line tools and Python API. The documentation provides detailed explanations of configuration parameters and tuning suggestions. It is deeply integrated with the Hugging Face ecosystem, supporting direct loading of models from the Hub and pushing merged results, seamlessly integrating into existing workflows. Community feedback shows that MergeKit has produced multiple models with excellent performance on the Open LLM Leaderboard, demonstrating its potential.

MergeKit is an important contribution of the open-source community in the field of model merging, translating academic achievements into practical tools and lowering the threshold for application. Future directions include supporting more model architectures, introducing intelligent automatic selection of merging strategies, and synergistic integration with optimization techniques such as quantization and pruning. For developers and researchers exploring the potential of model merging, MergeKit is worth in-depth study.