Model Merging

Model merging is a technique for combining the parameters of two or more neural networks — usually models fine-tuned from the same base — into a single model, using arithmetic on their weights rather than any form of training.^[1] A merged model can inherit several specialised behaviours at once: a coding assistant merged with a reasoning-tuned variant and a multilingual variant, for example, may perform well across all three without a further fine-tuning run.

History

The approach became prominent with model soup, published in 2022, which averaged the weights of models saved at different points during training and of models trained with different hyperparameters, improving accuracy without extra inference cost.^[2] Later work extended averaging to models fine-tuned on different tasks, introducing the vocabulary of task arithmetic — treating the difference between a fine-tuned model and its base as a vector that can be added to another model — and algorithms designed to resolve the conflicts that arise when those vectors disagree.^[3]

Key Concepts and Technology

Merging methods form a ladder of complexity:

• Uniform weight averaging takes the mean of corresponding parameters across models. It requires that all models share the same base checkpoint and architecture.
• SLERP (spherical linear interpolation) blends weights along a spherical path rather than a straight line, preserving magnitude better when combining two models.
• Task arithmetic adds scaled "task vectors" — the delta between a fine-tuned model and its base — to create a multi-task model.^[3]
• TIES-Merging addresses sign interference, where two models push the same parameter in opposite directions, by keeping only the majority sign and resolving ties before merging.
• DARE (drop-and-rescale) randomly drops a large fraction of the delta parameters and rescales the remainder, shrinking interference and enabling many models to be merged at once.^[1][4]

Practically, merging is executed with tools such as mergekit, which stream models layer by layer so that models too large to fit in memory simultaneously can be combined on a single machine. Because no training occurs, merging requires no dataset, no gradients and only seconds or minutes of compute for models that would take thousands of GPU-hours to fine-tune.^[1]

Its limits matter: merged models can underperform a well-chosen single model when the specialisations conflict, gains tend to shrink as more models are added, and all participants must originate from a compatible base — merging models with incompatible licences also creates legal risk that must be checked before distribution.

Applications

Merging is used to build generalist models from specialist ones, to fold evaluation-driven improvements from several checkpoints into a production release, to combine safety-tuned and capability-tuned variants, and to produce compact community models that stack many LoRA fine-tunes. It is particularly popular in the open-weight ecosystem, where practitioners redistribute merged checkpoints on public model hubs.

>See Also

• Fine-Tuning
• Parameter-Efficient Fine-Tuning
• LoRA
• Model Compression
• Open-Weight Models
• ILMU