What to do if language models disagree? Black-box model ensembling for
textual and visual question answering

Data Preparation. First, we randomly sample $N$ content-question pairs $\{(C_{i},Q_{i})\}^{N}_{i=1}$ and the corresponding ground truth answers $\{\widetilde{A}^{l}_{i}\}^{N}_{i=1}$ from various benchmark datasets (refer to Section 4.1). Next, we build prompts $P_{i}$ following specific prompt rules $P_{i}=R(C_{i},Q_{i})$ (refer to Section 4.1). Using these prompts instead of plain $(C_{i},Q_{i})$ text improves the LLMs’ answer quality (Bach et al., 2022).

InfoSel-TT. We use a textual BERT-base (Devlin et al., 2019) transformer $f^{t}_{\theta}$, ($\theta$ denote trainable model parameters) as the backbone of InfoSel-TT. To achieve faster convergence, we load the pre-trained weights of bert-base-uncased model.⁴⁴4https://huggingface.co/google-bert/bert-base-uncased The input vector $X^{l}_{i}$ is passed to $f^{t}_{\theta}$ to generate $K$ sentence representations for each value in $X^{l}_{i}$ respectively. Thus, the sentence representation $R^{t}_{ij}$ of $[P_{i},A^{l}_{ij}]$ from $f^{t}_{\theta}$ is:

Finally, the trained InfoSel-TT model ($M^{it}$) selects the winner model $M^{l}_{i,win}$ from $\{M_{j}^{l}\}^{K}_{j=1}$ for the input $P_{i}$ with the highest probability score based on the selection logits produced by $f^{d}_{\theta}$. $A^{l}_{i,win}$ denotes the answer provided by $M^{l}_{i,win}$.

FT-TT. A potential limitation of using InfoSel-TT model only is the lack of exposure of the base models to new (unseen) labels appearing in the task-specific datasets. To address this, we fine-tune a separate lightweight TT model directly on the TQA datasets to learn these new labels. Specifically, the training objective is to locate the start and end token position of the answer from the context $C_{i}$. We provide the token positions of $\widetilde{A}^{l}_{i}$ as the target label, such that the model is optimized to classify each token in two classes (start/end token). This fine-tuned textual transformer model is referred to as FT-TT ($M^{ft}$).⁵⁵5The training scheme is adapted from https://huggingface.co/learn/nlp-course/chapter7/7?fw=pt with the additional option to allow the model to return empty answers for unanswerable questions. We denote the answer predicted by $M^{ft}$ on $P_{i}$ as $A^{ft}_{i}$.

InfoSel^∗-TT. This model performs a further ensemble training of FT-TT and InfoSel-TT models with the same training scheme and labeled training data as InfoSel-TT. We anticipate that the thus trained InfoSel^∗-TT model ($M^{it+}$) on the output of InfoSel-TT and the label finetuned FT-TT, will improve the ability to handle labels unseen by base models. As a result, we expect an improvement in the overall task-specific performance. The winner model selected by $M^{it+}$ belong to $\{M^{it},M^{ft}\}$.

Data Preparation. Given $N$ image-question pairs $\{(I_{i},Q_{i})\}^{N}_{i=1}$ from dev data of VQA benchmark datasets, we use $K$ ($K$=3) pre-trained VQA models to predict answers $A^{v}_{ij}$ as follows: $\{M^{v}_{j}((I_{i},Q_{i}))\rightarrow A^{v}_{ij}\}^{K}_{j=1}$. We denote the ground truth answer for image-question pair $(I_{i},Q_{i})$ as $\widetilde{A}^{v}_{i}$. Target labels $Y^{v}_{i}$ for ensemble training are given by the accuracy scores of the $K$ candidate answers evaluated on $\widetilde{A}^{v}_{i}$:

The concatenation of question ($Q_{i}$) with each of the candidate answers ($A^{v}_{ij}$) obtained from the base models and the corresponding image ($I_{i}$) serves the input to our ensemble model InfoSel-MT:

InfoSel-MT. A Multimodal Transformer (MT, $f^{m}_{\theta}$) (Li et al., 2021b) is employed as the backbone for InfoSel-MT. Specifically, we first generate visual features $V_{i}$ of $I_{i}$ using a pre-trained R-CNN model ($M_{r}$) (Anderson et al., 2018). $V_{i}$ is composed of a vector of the image region features $v_{i}$ and the detected tags ( i.e., object labels of the image) (Li et al., 2021b). The concatenated question-answer pair $[Q_{i},A^{v}_{ij}]$ and $V_{i}$ is then passed together with to MT ($f_{\theta}^{m}$) to generate a fused contextual representation $R^{m}_{ij}$:

Finally, we use an additional dense layer ($f_{\theta}^{d}$) to map $R^{m}_{ij}$ to the target label $Y^{v}_{i}$. The training is optimized using binary cross-entropy loss:

We denote $M^{im}$ to the trained InfoSel-MT model, $M^{im}$ selects the winner model $M^{v}_{i,win}$ from $\{M_{j}^{v}\}^{K}_{j=1}$ to predict answer $A^{v}_{i,win}$ for the image-question pair $(I_{i},Q_{i})$ based on the selection logits produced by $f^{d}_{\theta}$.

FT-MT. Similar to FT-TT, FT-MT composed of a trainable MT and a Multilayer Perceptron (MLP) is fine-tuned with the same training data as InfoSel-MT. Differently, FT-MT solves a multi-label classification task by classifying the fused contextual representation of $Q_{i}$ (instead of $[Q_{i},A^{v}_{ij}]$ like InfoSel-MT) and $V_{i}$ to a predefined answer list (labels). This list contains frequent answers from the training data. As a result, a trained FT-MT model ($M^{fm}$) can learn to predict the unseen (new) labels (answers) contained in the task-specific datasets, but not in the pre-training data of the base models. $A_{i}^{fm}$ denotes the answer predicted by $M^{fm}$ over $(I_{i},Q_{i})$. The training scheme is adapted from  (Li et al., 2021b).

InfoSel^∗-MT. Similar to InfoSel^∗-TT, InfoSel^∗-MT model $M^{im+}$ ensembles the FT-MT and InfoSel-MT models using the same training scheme as in InfoSel-MT. The winner model selected by $M^{im+}$ belong to $\{M^{im},M^{fm}\}$.

TQA datasets. We generated two Mini datasets, Mini-SDv2 and Mini-NQ, consisting of 1,000 randomly sampled instances from SQuAD-V2 (Rajpurkar et al., 2018) and NQ-Open (Kwiatkowski et al., 2019) train splits, respectively. For Mini-NQ, we followed (Fisch et al., 2019) to use long answers as the context, and short answers as the ground truth answers. The 1,000 samples are divided into train and validation data using an 8:2 ratio, while the trained models are tested on the dev data of the original datasets due to the unavailability of original test data. We use the setup proposed in (Laskar et al., 2023) to generate the answers from LLMs. Concretely, this setting relies on prompts generated by PromptSource (Bach et al., 2022), which we apply to our Mini-SDv2 dataset (unavailable for Mini-NQ). The prompts are in two forms depending on the context: (1) “What is the answer? [Context]; [Question]; If you can’t find the answer, please respond ‘unanswerable’. Answer:”; (2) “Answer the question depending on the context. [Context]; [Question]; If you can’t find the answer, please respond ‘unanswerable’. Answer:”. Differently, Mini-NQ does not contain unanswerable questions and thus we use the prompt “Answer the question depending on the context without explanation. [Context]; [Question]; Answer:”. Note that the answers of LLMs can be greatly influenced by some factors such as the use of different prompts or temperatures. However, our study does not focus on prompt engineering but rather on selecting the optimal base model to generate an answer. We will publicly release our prompts as well as the answers from LLMs for reproducibility.

VQA datasets. Our results (Figure 2) reveal that VQA tasks demand a greater quantity of training samples compared to TQA tasks. Therefore, we constructed Mini-GQA and Mini-Viz datasets using a larger fraction (the dev data) of GQA (Hudson & Manning, 2019) and VizWiz (Gurari et al., 2018) datasets compared to TQA datasets. The resulting Mini-GQA and Mini-Viz were divided into train and validation subsets using 8:2 ratio, while the test subset remained the same as in the original datasets.

Weighted Voting (WV). We adopt a strategy similar to Schick et al. (2020), where the model accuracy of the train data before training is used as the weight for average weighting. In our case, we use the corresponding accuracy of the base models as the weight for voting.

PageRank (Brin & Page, 2012). We adapt PageRank as a baseline to determine the most suitable answer in a graph where all the answers to one question are connected by their BLEURT (Sellam et al., 2020) similarities.

Overall Local Accuracy (OLA) (Woods et al., 1997). Following (Cruz et al., 2018), we use the k-nearest neighbors algorithm to divide the input space (representations of prompts for TQA, representations of images and questions for VQA) of training data into 7 regions. The overall local accuracy of each base model in different regions is computed as its region competence. The model presenting the highest competence level is selected to predict the label of inputs that fall in the same region.

PairRanker and LLM-Blender (Jiang et al., 2023). Both methods were originally designed for text generation tasks (e.g., machine translation and speech recognition). PairRanker model (DeBERTa (He et al., 2023), 400M parameters) is trained to rank a pair of candidate predictions from two LLMs using multiple optimizing objectives (i.e., log probabilities, BART score, BERTScore, etc). A bubble sort is applied to get the top k (we use k=2) predictions among multiple pairwise comparison results to feed to a GENFUSER model (Flan-T5-XL (Chung et al., 2022), 3B parameters) to generate the final fused prediction. LLM-Blender is a composite of the PairRanker and the GENFUSER model. We use a pre-trained (0-shot) PairRanker and LLM-Blender models which have been trained over massive data (105k) including TQA data as baselines to test on our data.

Setup. We fixed the batch size to the upper limit of the server capacity, while the learning rates and epochs are selected after a grid search on a set of values (learning rates: {e3, 5e4, e4, 5e5, e5, 5e6, e6}, epochs: {3, 5, 10, 15, 20}). Models for TQA are trained for 5 epochs using a learning rate of $5\times 10^{-5}$ and batch size of 4. Models for VQA use the same learning rate but a batch size of 16 for 20 epochs. We spent $\sim$74 and $\sim$290 seconds training 1 epoch on 1,000 samples for TQA and 4,319 samples for VQA respectively. The training was performed on 1 GPU with 16GB memory of a DGX1 server ((Pascal) Tesla P100).

Task-specific Performance. Table 3 demonstrates the task-specific performance of InfoSel, base models and baselines on textual and visual QA datasets. For TQA, we observe that LLaMA underperforms other base models. Upon closer examination, we found that LLaMA generates longer explanation text which, although often accurate, decreases the EM and F1-score values. Conversely, a more consistent performance of base models is observed for VQA. All the models demonstrate superior performance on Mini-NQ compared to Mini-SDv2. This is because Mini-SDv2 contains 50% of unanswerable questions, written adversarially and specifically designed to be challenging for the QA task (Rajpurkar et al., 2018). Similarly, Mini-Viz contains 28% of unanswerable questions, and the label “unanswerable” has never been seen by base models. Consequently, this lack of exposure leads to significantly lower performance scores. The Oracle scores of the base model indicate the performance of an ideal ensemble method.

Data Efficiency. The experimental results shown in Figure 2 demonstrate the data efficiency of our method by evaluating the model’s performance across varying training data sizes. We observe that InfoSel-TT achieves a higher F1-score compared to the base models when trained on as little as 10 samples from Mini-SDv2. This result has been further verified with the mean F1-score of 10 test results using different seed variations for sampling the training data. Conversely, the number of training of samples needed to surpass the performance of base models is higher for VQA datasets: 5% (6,603 samples) for Mini-GQA and 20% (864 samples) for Mini-Viz. We hypothesize that this is due to the inherent complexity of the VQA task.

Lightweight Model. Table 4 reports the parameter size of the base models and their ensemble models (InfoSel-TT and InfoSel-MT). InfoSel  provides an efficient method for ensembling large LLMs such as ChatGPT (175B parameters) using only 110M trainable parameters. Even though only 37% ((182M-115M)/182M) trainable parameters are saved for the VQA task, we still demonstrate that InfoSel can effectively enhance the task-specific performance of small-size black-box VQA models, offering a lightweight solution.

Multimodal Data. InfoSel is able to utilize multimodal data (image and text) for VQA tasks, and thus outperform the latest text-exclusive LLM ensemble methods (PairRanker and LLM-Blender) as evidenced in Table 3. In contrast, LLM-Blender cannot process image features, thereby lacking crucial information in the multimodal setting, leading to an accuracy of 0.0 on VQA datasets.⁷⁷7The most frequent answer of LLM-Blender on VQA datasets is “I’m sorry, I don’t have enough context to answer that question.” Further insights into the significance of modality information are elaborated in Section 4.6 and Table 5.

Which modality information helps the most for ensembling? In the Table 5, we compare the effect of providing different modality information to InfoSel-MT during ensemble training. Notice that even with just the question and answer (Q+A) information, our model surpasses the performance of the 0-shot PairRanker and LLM-Blender. The setting that yields the lowest accuracy solely utilizes the image (V) as the signal. This can be explained by the fact that a single image often corresponds to multiple questions in VQA datasets, making it challenging for the model to acquire discriminative features. Furthermore, we conclude that a superior performance of our model when utilizing image, question, and answer (V+Q+A) data demonstrates the effectiveness of our model in multimodal setting.