Unlocking the Potential of Model Merging for Low-Resource Languages

Model merging is a technique for combining multiple models possessing different capabilities into a single versatile model without additional training. For example, we can merge a model specialized for Japanese and a model specialized for math to obtain a model that excels at solving mathematical problems in Japanese Akiba et al. (2024). In this work, we investigate two commonly-used methods of model merging: weighted averaging Choshen et al. (2022); Wortsman et al. (2022) and TIES Yadav et al. (2023). Here we provide a brief overview of these methods.

Weighted averaging is simply averaging the parameters of two models with a weight tuned on the validation set.

TIES aims to handle the parameter conflicts across multiple models more meticulously. Suppose we have two models specialized for distinct tasks, denoted as $\theta_{1}$ and $\theta_{2}$, both trained from the same initial model $\theta_{\text{init}}$. Task vectors for these models are calculated as follows: $\tau_{1}=\theta_{1}-\theta_{\text{init}}$ and $\tau_{2}=\theta_{2}-\theta_{\text{init}}$. The objective is to merge these task vectors and reintegrate them into the initial model.

The merging process of TIES consists of three steps: (1) Trim: For $\tau_{1}$ and $\tau_{2}$, we trim the redundant parameters by keeping the top-$k_{1}$% and top-$k_{2}$% values, respectively, creating $\hat{\tau}_{1}$ and $\hat{\tau}_{2}$. (2) Elect Signs: For each parameter $p$ in $\hat{\tau}_{1}$ and $\hat{\tau}_{2}$, we select the sign (+1 or -1) with the higher magnitude, denoted as $\gamma^{p}=\mathrm{sgn}(\hat{\tau}_{1}^{p}+\hat{\tau}_{2}^{p})$. (3) Disjoint Merge: For each parameter $p$, we only keep the parameter values from $\hat{\tau}_{1}$ and $\hat{\tau}_{2}$ whose signs are the same as the aggregated elected sign and calculate their mean. Specifically, for each parameter $p$, its disjoint mean is calculated as $\tau_{m}^{p}=\mathrm{avg}(S^{p})$, where $S^{p}=\{\hat{\tau}_{i}^{p}|\mathrm{sgn}(\hat{\tau}_{i}^{p})=\gamma^{p},i=1,2\}$.

Given the final merged task vector $\tau_{m}$, we scale it and add it to the initial model $\theta_{\text{init}}$ to obtain the merged model $\theta_{m}$ as $\theta_{m}=\theta_{\text{init}}+\lambda\cdot\tau_{m}$, where $\lambda$ is a scaling hyperparameter.

For TIES, we tune three hyperparameters in total on the validation set: two sparsity rates $k_{1}$, $k_{2}$ and a scaling factor $\lambda$.

Please refer to the original paper of TIES Yadav et al. (2023) for more details.

Given a base model only pre-trained on an English-centric corpus, e.g., Llama-2-7B Touvron et al. (2023) in our study, we want to construct a model capable of solving tasks in a low-resource language. For the target language, there are very limited pre-training texts, ranging from 1B to 20B tokens, and almost no data for supervised finetuning (SFT). In this scenario, we investigate two representative paradigms of constructing such a model: CT-then-SFT and model merging. We illustrate the roadmap in Figure 1, which demonstrates the relations between the models.

For all the studied languages except Bengali, the models constructed by merging outperform those built by CT-then-SFT⁶⁶6The exceptional performance in Bengali may be attributed to its larger CT corpus, which we discuss in Section 5.2.. For example, the merged models based on TIES (TIES-X-flan) achieve an average score of 46.80% across languages, surpassing CT-then-SFT models (CTSFT-X-flan) by +4.69%. Although using a naïve merging algorithm, model merging with WAVG (WAVG-X-flan) still achieves better performance than CT-then-SFT (CTSFT-X-flan) in four languages.

In the conventional CT-then-SFT approach, LLMs often fail to acquire sufficient comprehension of the target language from a small-scale CT corpus, and this ability may be further diminished by supervised fine-tuning. In contrast, model merging can preserve the language modeling acquired during CT while incorporating task-solving capabilities by resolving parameter conflicts carefully.

In conclusion, model merging can be an effective pathway to obtain task-solving LLMs for low-resource languages.

Although model merging proves more effective than CT-then-SFT in most languages of our study, this is not the case for Bengali, which utilizes the largest corpus in the experiments: CTSFT-ben-flan outperforms TIES-ben-flan. We guess that the applicability of model merging may be related to the amount of corpus used in CT. Thus, we examine the performance changes of the merged models and the CT-then-SFT models under different amounts of CT tokens.

We collect the intermediate checkpoints of CT-ben and CT-tel, which are the two languages with the largest size of pre-training corpora. Then, we derive CTSFT-X-flan and TIES-X-flan models based on these checkpoints.

Figure 2 illustrates the performance of CT-only (CT-X), CT-then-SFT (CTSFT-X-flan), and model merging (TIES-X-flan) based on every checkpoint. For Bengali and Telugu, TIES-X-flan outperforms CT-X in each checkpoint, indicating model merging can robustly enhance LLM’s task-solving ability, while CT-then-SFT demonstrates greater variability. And in both languages, for all checkpoints where the amount of pre-trained tokens is less than 10.4B tokens, TIES-X-flan can achieve better results than CTSFT-X-flan.

Model merging can integrate language modeling and task-solving capabilities more effectively than CT-then-SFT, in scenarios with limited language resources ($<$10B tokens). Four out of the seven languages in our study utilize corpora with fewer than 10B tokens. This data scarcity is characteristic of low-resource languages. For instance, 81% (135 out of 166) of the languages in the multilingual corpus CulturaX Nguyen et al. (2024) have fewer than 10B tokens. Consequently, our findings on the effectiveness of model merging are broadly applicable and have the potential to benefit a wide range of human languages.

As the amount of CT tokens increases, the CTSFT-X-flan models show more rapid improvement in task-solving capabilities compared to TIE-X-flan. Specifically, when pre-trained with more than 14.4B tokens, CTSFT-ben-flan demonstrates superior performance over TIES-ben-flan. Similarly, in Telugu, the performance gap between the two models diminishes with additional CT tokens. However, due to the smaller size of the Telugu corpus compared to Bengali’s, we have not observed CTSFT-tel-flan overtaking TIES-tel-flan. We note that CT-then-SFT may be a better method to construct task-solving LLMs in languages with sufficient resources, for example, Bengali, Vietnamese, Indonesian, etc.

Revisiting the mechanism of TIES, we find that during the merging stage, parameters from one model may be discarded due to differences in the parameter signs between the two models. The discarding of model parameters usually leads to a decline in the corresponding capabilities. Considering that the CT model’s language modeling ability continuously improves with the increase of pre-training data, we suspect that it is more likely that the SFT model discards too much information during the merging process, resulting in the merged model’s inability to further enhance its task-solving ability in the target language.

To verify this hypothesis, we take Bengali, the language using the largest amount of CT corpus in our study, magnitude We explore the changes in the number of discarded parameters in the SFT model when merging it with CT models trained with different quantities of data.

As explained in Section 3.1, we first calculate the task vectors of SFT-flan and each checkpoint of CT-ben. In the trimming stage, we find the optimal hyperparameters are $k_{\text{sft}}=0.2$ and $k_{\text{ct}}=1.0$ for most scenarios⁷⁷7If not being optimal, this set of hyperparameters can also result in comparable performance to the optimal ones.. Thus, to provide a fair comparison, we freeze $(k_{\text{sft}},\,k_{\text{ct}})$ as $(0.2,\,1.0)$ for all checkpoints. Then, we examine the signs and magnitudes of each parameter of the trimmed task vectors of SFT-flan and CT-ben. A parameter that has contrary signs in two models can be regarded as a parameter with sign conflict. And if its magnitude in SFT-flan is smaller than that in CT-ben, it will be removed at the stage of sign election.

Figure 4 illustrates the proportion of parameters that are discarded from SFT-flan during the merging stage. We can find as the LLM is pre-trained with more tokens, 4% more parameters are removed in trimmed SFT-flan. We believe that when using more tokens for CT, TIES discards a larger proportion of parameters from the SFT model, which may continuously undermine the task-solving capabilities.

To mitigate the information loss in SFT-flan during the merging stage, we propose TIES-SV, enhancing TIES with a Slack Variable to reduce the number of discarded parameters in the model with higher information density, i.e., SFT-flan in this situation.

According to the process of TIES, for each parameter to be discarded from SFT-flan in the Disjoint Merge step, its magnitude is smaller than the magnitude of its counterpart parameter in CT-X. To retain the parameters of SFT-flan while minimizing the information loss of CT-X, we first rank these pairs of parameters between SFT-flan and CT-X according to their differences in the magnitude. Next, we select a subset of parameters with the smallest magnitude differences to reserve.

We evaluate the effectiveness of TIES-SV on the model merging process of SFT-flan and the last checkpoint of CT-ben. Based on the results in Figure 4, we reserve 4% parameters in SFT-flan that were to be discarded in the original algorithm of TIES. Table 4 shows the results of vanilla TIES and our TIES-SV on three Bengali tasks. In all three tasks, our TIES-SV outperforms vanilla TIES by 0.52% on average.

In conclusion, we propose a simple strategy to improve model merging. It suggests that during the process of model merging, the importance of different models’ parameters is not the same. When parameter conflicts occur, we cannot simply rely on the magnitude of the vectors to decide which model’s parameters to reserve. Instead, we should use prior information obtained through pilot studies or other means to reserve the parameters of the more important model. We hope our TIES-SV can shed light on the study of new model-merging algorithms in the future.

Previous work Akiba et al. (2024) shows that model merging algorithms can be used to combine multiple LLMs with different capabilities, which may enhance the model to solve complex problems. We wonder whether multiple LLMs adapted to different low-resource languages can be merged with the same SFT model, to construct a a task-solving LLM supporting these languages simultaneously.

As a pilot study, we attempt to merge Mongolian and Uyghur LLMs with the English task-solving model SFT-flan. Due to the limited budget for computational resources, we cannot conduct a grid search across the hyperparameter space of the three models. Therefore, we employ the optimal hyperparameters derived from merging each of the two CT models with SFT-flan.

Table 5 illustrates the average scores of tasks in the two languages. The merged model serving two low-resource languages (TIES-mvf&uig-flan) performs comparably to the two single-language merged models (TIES-mvf-flan and TIES-uig-flan) on tasks in the respective languages.

This indicates that model merging has great potential for constructing multilingual task-solving LLMs. We hope this approach can assist multilingual speakers, particularly those using underrepresented languages, by combining multiple existing LLMs in distinct languages without the need for expensive pre-training.

Collecting synthetic SFT data through MT is an intuitive method for constructing task-solving LLM in non-English languages Muennighoff et al. (2023); Li et al. (2023a). However, the experiment results in Table 3 show that MT-translated SFT data may not work when it comes to low-resource languages. The models trained with MT-translated data (CTSFT-X-mt) models exhibit inferior performance across all languages compared to those trained on English SFT data (CTSFT-X-flan), with a gap of -3.38%$\sim$-24.61%.

The decline in model performance can be attributed to the low-quality MT results. Current open-source MT systems have limited abilities of low-resource languages. For example, we ask Uyghur and Tibetan native speakers to evaluate sampled translation results by NLLB. They find that there are often irrelevant contents and undesired code-switching in the translation results, as shown in Appendix C.2. This kind of noise not only hinders the model’s ability to learn task-solving strategies but also interferes with language modeling, which itself is inadequately learned.

In contrast, model merging eliminates the need to collect SFT data in the target languages. This approach also eliminates the reliance on unreliable machine translation systems.