CharED: Character-wise Ensemble Decoding for Large Language Models

As large language models (LLMs) have become increasingly ubiquitous and powerful models have been open-sourced, there has been extensive research on methods to achieve improved task-specific performance from these models. The long-standing method for doing this is through fine-tuning, in which domain-specific datasets are used to update weights of large foundation models to improve performance on certain tasks. However, direct fine-tuning is both time-consuming and computationally intensive (Strubell et al., 2019). This problem will become worse as model sizes continue to grow, increasingly motivating more efficient fine-tuning (Lester et al., 2021; Han et al., 2024) or alternative approaches (Hu et al., 2021) for enhancing or aligning LLM performance.

Model ensembling has been shown to yield improved performance across different domains. An established method for doing this is through shallow fusion, which was originally used to integrate an LLM into a neural machine translation (NMT) model (Gulcehre et al., 2015). Such ensembling methods, which aggregate models during beam search, have shown promise for improving translation quality in NMT settings (Sutskever et al., 2014; Firat et al., 2016; Stahlberg et al., 2018), but require the same vocabulary and tokenization. Twist decoding (Kasai et al., 2022) modifies beam search to bypass the shared vocabulary restriction, but its reliance on beam search reduces the inference speed. Other more recent approaches related to combining language models include proxy tuning (Liu et al., 2024) and Composition to Augment Language Models (CALM) (Bansal et al., 2024). Proxy-tuning adjusts next-token predictions of a larger LLM using a pair of tuned and untuned smaller LMs, but is essentially limited to models from the same family, as it requires shared vocabulary. CALM can combine any LLMs via cross-attention but requires additional training.

Historically, major advances in LMs have come out of subword-level tokenization schemes, which gained traction for their flexibility (Yang, 2024), including byte-pair encoding (BPE), SentencePiece, and WordPiece (Sennrich et al., 2016; Kudo & Richardson, 2018; Devlin et al., 2019; Zhang et al., 2019). These tokenization methods have generally outperformed character-based language modeling, like LSTMs and other RNNs. Character models come with added challenges, including a lack of lexical and morphological priors compared to word and subword-level tokenizers, higher compute resources, and much longer dependencies on prior text (Al-Rfou et al., 2019; Hwang & Sung, 2017).

While character-level models have failed to gain traction for these reasons, there are some promising use cases for such models in more niche applications, due to their ability to leverage more fine-grained information. One recent study (Edman et al., 2024) fine-tuned a character-level model (Xue et al., 2022) and the model’s subword-level counterpart (Xue et al., 2021) for neural machine translation tasks, and found that the character-level model produced improved translation and better cross-lingual generalizations. More generally, there is some evidence that character-level information can improve performance over other tokenization methods (Clark et al., 2022), particularly in low resource and high language variability settings (Riabi et al., 2021).

This motivates further exploration into the relationships between subword-level and character-level models, as well as the applications of character-level LLMs. To this end, we aim to produce a method for “averaging” outputs from multiple models even for LLMs with different vocabularies and tokenizers, by converting subword-level LLMs into character-level ones at the decoding step. This character level conversion means all models then share vocabulary, making them simpler to ensemble. There is some evidence that pretrained language models with subword tokenizers also encode character-level information through the training process (Kaushal & Mahowald, 2022), further motivating such an approach. Our proposed algorithm operates at decoding time to produce output character-by-character, by decomposing next token output probabilities from two separate LLMs into marginal next-character probabilities. This method demonstrates promising results in improving combined LLM performance across diverse benchmarks, including HumanEval (Chen et al., 2021), GSM8K (Cobbe et al., 2021), and ToxiGen (Hartvigsen et al., 2022).

We propose CharED, an algorithm to convert LLMs into character-level models and combine them.

Let $\mathcal{M}_{1}$, $\mathcal{M}_{2}$ be the LLMs to combine. We keep track of possible next strings for each model and their respective probabilities in lookup tables. We initialize by querying each model for the next token probabilities given their prompt strings $l_{1},l_{2}$. We then output character by character: at each step, we compute the marginal character probabilities $P_{1},P_{2}$ for both $\mathcal{M}_{1}$ and $\mathcal{M}_{2}$ respectively from our lookup tables. Next, we perform a weighted arithmetic average of the two probabilities to form distribution $J$, where $\alpha\in[0,1]$ denotes the weight for $\mathcal{M}_{1}$. Then, we choose the next character either greedily or by sampling from $J$. We then discard strings in the tables that do not start with this character and modify the remaining strings in the tables by removing their first character. Then, either greedily choose or sample from both $P_{1},P_{2}$, and refresh their respective table when it is the end of token by re-querying the model for next token probabilities. Then remove the end of token from each table and renormalize. Note that the end of token can be signified by the empty string. Repeat the above steps to generate the output sequence.

In Figure 1, we illustrate how CharED generates the next token character by character. Next, we provide an example to illustrate the “repopulation” step in lines 20-23 of Algorithm 1. Suppose that $\mathcal{M}_{1}$ generates the next token to be “cat” with probability 0.9 and $\mathcal{M}_{2}$ generates the next token to be “cats” with probability 0.85, where $\alpha=0.5$ and we use CharED with sampling. Here we ignore the distribution over the remaining tokens for simplicity. Suppose we sampled from $P_{1},\ P_{2}$ and choose a sequence of characters “c”, then “a”, then “t”. At this point, we find that $\mathcal{M}_{1}$ ends the token with probability 0.9, and $\mathcal{M}_{2}$ continues to the letter “s” with probability 0.85. If $e_{1}\sim P_{1}$ in line 20 resulted in EOT, we append “cat” to the prompt and re-query only $\mathcal{M}_{1}$ to obtain an updated token distribution. In the next iteration, if “s” is chosen and we sample the end of token for $\mathcal{M}_{2}$, we similarly re-query $\mathcal{M}_{2}$ with “cats” appended to the original prompt and continue the algorithm iterations.

We analyze coding, math, and toxicity avoidance using three standard benchmarks: HumanEval (Chen et al., 2021), GSM8K (Cobbe et al., 2021), and ToxiGen (Hartvigsen et al., 2022).

We run three experiments, one for each pairwise combination of domains, using CharED to combine the domain-specific models $\mathcal{M}_{1}$ and $\mathcal{M}_{2}$ for their respective fine-tuned domains. For each configuration, we vary $\alpha$ from 0 to 1 in 0.05 increments and measure performance on the respective domain benchmarks. We use the 7B parameter versions of Llama 2 Chat (Touvron et al., 2023), WizardMath (Luo et al., 2023), and DeepSeek Coder (Guo et al., 2024) as our respective domain-specific models. Each model can use its own prompt and template. We present prompting details in Appendix C.

Thus, we run CharED using greedy selection on the following pairs of models $\mathcal{M}_{1},\mathcal{M}_{2}$ and settings: (1) DeepSeek Coder and WizardMath, tested on HumanEval and GSM8K. (2) DeepSeek Coder and Llama 2 Chat, tested on HumanEval and ToxiGen. (3) WizardMath and Llama 2 Chat, tested on GSM8K and ToxiGen. Further evaluation details are provided in Appendix D.

Using CharED, we test pairwise model combinations on GSM8K, ToxiGen, and HumanEval. Results are shown in Figure 2. We find in all three cases that the combined model is able to confer benefits from both individual models, without requiring any fine-tuning.

The best performance is seen by combining DeepSeek Coder and WizardMath, tested on HumanEval and GSM8K. Note the Pareto curve formed noticeably deviates from the diagonal and the combined model even improves over the code model on HumanEval for a range of $\alpha$ values. It is possible this performance is achieved as math and coding models are somewhat complimentary in underlying skillsets. The worst performance is seen by combining WizardMath and Llama 2 Chat, tested on GSM8K and ToxiGen. Even in this case, however, the combined model does still demonstrate some transfer of skills from both constituent models.

Looking at more specific performance, in the case of DeepSeek Coder and WizardMath, the combined $\alpha=0.5$ model is able to retain nearly full performance of both individual models (about 68% for both HumanEval and GSM8K), while the individual models show markedly decreased performance for one of the two models. For DeepSeek Coder and Llama 2 Chat, the combined model at $\alpha=0.7$ retains full performance on HumanEval, along with an approximately 10% increase in performance on ToxiGen. With an $\alpha=0.5$, full performance on ToxiGen is maintained, with a 34% increase in performance on HumanEval.

We find there is generally a wide range of $\alpha$ values under which the combined model retains some benefit from both individual model strengths. See Appendix E for further results on optimal $\alpha$ values for each benchmark combination.

Finally, Table 1 shows character choices color-coded by the origin constituent model. Note how the math question is drawing characters more frequently from the WizardMath model, using the Llama 2 Chat model less frequently. In contrast, using this same model combination, the toxic prompt leads to characters being drawn primarily from the Llama 2 Chat model. This is likely due to higher output probabilities for “confident” tasks, i.e., tasks that the model excels at. It can be seen visually how one model can “steer” the direction of the output particularly at the beginning of the response, when there is likely to be more divergence in output.

Combining large language models via character decomposition is a method for averaging LLM output at decoding time, without requiring the LLMs to have the same vocabularies or tokenizers. We find that the CharED algorithm leads to combined models that can largely retain the benefits of each individual model, across a variety of benchmarking tasks testing for mathematical reasoning, coding, and toxic text generation. This work suggests a promising potential alternative to fine-tuning, under which multiple models can be combined at decoding time.

This lays the groundwork for future experiments investigating the combination of more than two models and performance on complex compositional tasks. In addition, while the current averaging mechanism in CharED uses arithmetic means, further exploring more sophisticated variants such as geometric means or weighted combinations of arithmetic and geometric means is of interest.

This collaboration was made possible by Harvard’s Responsible Computing Collective (ReCompute). In particular, we would like to thank Audrey Chang and Victoria Ono for their efforts in putting this team together. DAM acknowledges support from the Dean’s Fund for Promising Research.