A Survey of Large Language Model Fusion Technologies: A Unified Perspective from Parameters to Behavior

This article surveys the latest advances in large language model fusion technologies, proposes a three-layer classification system (parameter-level, representation-level, behavior-level), systematically organizes the principles, advantages, and limitations of various fusion methods, and discusses application prospects, challenges, and future directions. The GitHub repository of this survey is continuously updated and serves as a high-quality resource entry point for the field.

With the rapid development of large language models, model fusion technology has become an important direction. Unlike traditional training paradigms, model fusion integrates the capabilities of multiple source models into a target model without relying on the full operation of the source models. Practical background: The open-source community has a large number of task-specific fine-tuned models; deploying them individually is costly and difficult to coordinate, and model fusion provides the possibility of capability superposition with "1+1>2" effect.

The survey proposes a three-layer classification framework:

1. Parameter-level fusion: Directly operates on weights/parameter increments, such as Model Soup, Task Arithmetic, TIES-Merging, etc., requiring white-box access;
2. Representation-level fusion: Uses intermediate representations like hidden states/activation values, such as weight and representation matching, closed-form representation solving, etc., requiring hidden states or calibration data;
3. Behavior-level fusion: Relies on observable behaviors like output logits/generated results, such as distribution fusion, demonstration fusion, etc., requiring only output/API access.

Comparison of methods across different layers:

Fusion Layer	Main Signals	Access Requirements	Advantages	Limitations
Parameter-level	Weights/Task Vectors	White-box	Low cost to build deployable models	Sensitive to parameter alignment
Representation-level	Hidden States/Activations	Hidden states or calibration data	Fixes internal representation mismatches	Requires calibration data and layer matching
Behavior-level	Outputs/Demonstrations	Output/API	Suitable for heterogeneous/closed-source models	High training cost
Selection strategy: Parameter-level is suitable for fast combination of same-architecture models; representation-level for fine-grained control; behavior-level for closed-source models.

Application prospects: Multi-task learning, capability expansion, model updating, federated learning, open-source ecosystem integration. Challenges: Parameter conflicts, capability trade-offs, insufficient interpretability, lack of unified evaluation standards.

Future breakthrough directions: Automated fusion tools, deepening theoretical understanding, cross-modal fusion, dynamic fusion, integration with model editing.

Model fusion provides new possibilities for AI capability integration and reuse, and the three-layer classification system offers a clear framework for method selection. With the development of the open-source ecosystem, this technology will play an important role in reducing deployment costs and accelerating capability iteration. The survey repository is continuously updated and is a high-quality resource in the field.