Boosting Zero-Shot Crosslingual Performance
using LLM-Based Augmentations with Effective Data Selection

A teacher model is trained on the source data ($D_{\text{en}}$) using cross-entropy loss. The teacher is used to pseudolabel the generations in $G_{\text{en}}$. A subset of $G_{\text{en}}$ is chosen via various selection techniques described in Section 2.3. We will refer to this subset as $D^{\prime}_{\text{en}}$. A student model is trained on both $D_{\text{en}}$ and $D^{\prime}_{\text{en}}$ combined, using cross-entropy loss with the gold labels in $D_{\text{en}}$ and a KL-divergence loss with the soft pseudolabels derived from the teacher model. Equation (1) refers to the overall loss computed, where $y_{c}(x)$ is the one-hot label corresponding to each $x\in D_{\text{en}}$, $q_{c}$ is the student model probability for class $c$ (with temperature 1), $p_{c}(x)$ is the teacher model probability for each $x\in D^{\prime}_{\text{en}}$ for class $c\in C$ and $q_{c}^{*}$ is the student model probability for class $c$ (scaled by temperature value 1.5). This is the standard teacher-student paradigm, and we will refer to the trained student model as $\mathcal{T}$ in our experiments.

Rather than using English source data and English generations, we can also adopt the translate-train setting Artetxe et al. (2020) where $D_{\text{en}}$ and $G_{\text{en}}$ are translated to the target language using an off-the-shelf neural machine translation system to yield $D_{\text{tg}}$ and $G_{\text{tg}}$, respectively. The rest of the above-mentioned teacher-student training pipeline stays the same, except with using translated data everywhere.

Instead of using a teacher model to pseudolabel the generations in $G_{\text{en}}$/$G_{\text{tg}}$, we use the teacher’s label probabilities for data selection (detailed in Section 2.3) after which we label the data using the labels in the LLM prompts that we use for class-conditional generation. A single model is trained using cross-entropy loss on both source data in $D_{\text{en}}$/$D_{\text{tg}}$ and prompt-labeled data sampled from $G_{\text{en}}$/$G_{\text{tg}}$. The main difference from teacher-student training is the use of hard prompt labels for the sampled generations with a cross-entropy loss instead of soft pseudolabels from a teacher model with a KL-divergence loss. Here, we first utilize the teacher for data selection and subsequently use the LLM prompt labels for the generations. This model will henceforth be referred to as $\mathcal{T}_{\text{pl}}$. Similar to $\mathcal{T}$, even with $\mathcal{T}_{\text{pl}}$, we can adopt the translate-train setting and use translated source data and LLM generations.

We use SST5 (Socher et al., 2013) and SNLI (Bowman et al., 2015) datasets for sentiment analysis (SA) and natural language inference (NLI), respectively. SST5 is a sentiment classification dataset featuring five distinct labels: negative, very negative, positive, very positive, and neutral, that we collapse into three labels: positive, negative and neutral to match the target tasks. Similar to (Li et al., 2021), we consider a random subset of the SNLI train set (15K training sentences, 5K per class) to simulate a low-resource setting and for quicker experimental turnaround.

Our target SA tasks include Marathi Sentiment (Pingle et al., 2023), GLUECoS Hindi-English code-switched Sentiment (Khanuja et al., 2020), and Hindi Product Reviews (Akhtar et al., 2016). For NLI, we evaluate on Hindi, Urdu, and Swahili from the XNLI (Conneau et al., 2018) corpus; these are some of the the least-represented XNLI languages. Appendix A provides more details about the source and target tasks. Appendix C shows how we generate code-mixed data for the translate-train setting of the GLUECos task.

Here, the model is trained on the train set of the source tasks (refer Section 3.1). No synthetic data is used to train the model. 8,544 and 15K instances are used for SA and NLI tasks, respectively.

Here, the model is trained on a mixture of source and synthetic datasets. The synthetic dataset (7.5K) is sampled randomly from among the generations and is not selected via any data selection technique or with the help of a teacher resulting in 16K and 22.5K instances for SA and NLI tasks, respectively.

Here, we train the model only on the synthetically generated data. The labels come from the prompts that we used for class conditional generation. For a fair comparison with Src+Gen, we maintain the same total size of  16K and 22.5K instances for SA and NLI tasks, respectively

Table 3 shows XNLI results of student models/prompt-based models trained on data selected using amb-k and easy-k selection techniques. Augmenting the source data with prompt-labeled ambiguous instances benefits the model the most. Ambiguous instances are ones that the model is most uncertain about and are likely to help the model generalize well. This is consistent with observations about ambiguous instances in prior work (Swayamdipta et al., 2020; Liu et al., 2022).

Table 4 shows the translate-train accuracies of a student model ($\mathcal{T}$) trained using teacher hard pseudo labels and soft pseudo labels. CE implies training using teacher hard labels and cross-entropy loss, KLD implies training using teacher soft labels and KL-divergence loss. Large delta values highlight the significance of using soft teacher labels instead of hard labels. If the teacher labels are noisy, soft labels help the student model generalise better to unseen data.

To study the effect of augmented data size on cross-lingual transfer, we experiment with div-k selection ($\mathcal{T}$ model) and Src+Gen model for the Marathi SA task. Models are trained in the translate-train setting over varying amounts of augmented data. Table 5 shows that increasing $k$ leads to a decrease in accuracy. The consistent downward trend in the div-k selection technique underscores the importance of data selection; the best accuracies were obtained using div-k with $k=7500$. Also, augmenting synthetic data also results in increased training time. Determining the optimal augmentation size for each target task is left as future work.

To explore whether LLM generations boost performance even in the presence of source data that matches in domain to the target task (henceforth referred to as target training data), we train teacher models on a subset of 15K sentences from the XNLI train set. (Note that the numbers in Table 1 were obtained using SNLI as the source data.) As expected, we see significant improvements in teacher accuracies in Table 6 when using target train data. Student models trained on generations pseudolabeled with this superior teacher further boost accuracies; the best results are obtained using a combination of translate-train and div-k selection. Also, the accuracies obtained using Src+Gen models are inferior to the best scores obtained using $\mathcal{T}$ models, again highlighting the importance of data selection.

By default, we create class-balanced augmentations by sampling 2500 instances from each class based on teacher labels. To analyse the effect of class imbalance on downstream evaluation, we augment the sentiment source data with class-imbalanced augmented data sets for the Marathi SA task. The total size of the augmented data remains constant, while the class distribution is altered by eliminating neutral sentences from 2500 to 0, thereby transitioning towards sentiment-rich augmentations. We see in Table 7 that students trained on data that is uniformly distributed across classes along with the top-k selection strategy exhibits superior performance, compared to those trained on a subset of the generated data with imbalanced class proportions. This suggests that employing a class-balanced augmentation is an important consideration.

To evaluate whether the LLM prompt labels are truly reflected in the filtered generated text or not, we ran a human evaluation on a set of 100 generations each for sentiment analysis and NLI. These instances were randomly selected generations from among a set of pseudolabels predicted with high probability by the teacher model for the respective tasks. The average accuracy of label alignment between annotator-provided labels and prompt-derived labels was found to be 87.88% and 71.72%, with Cohen’s kappa coefficients of 0.752 and 0.749, respectively for the two tasks. This suggests that the prompt labels in the subset obtained after teacher-based filtering are of fairly high quality. Please refer to Appendix G for more details.

We introduce a simple metric that we call a “diversity score" to capture the dissimilarity in text embeddings across sentences in a dataset. This is computed by encoding each instance using LABSE (Feng et al., 2022), taking the average of the cosine distance of the LABSE embedding with every other instance in the data sample and finally taking an average of these mean distances across all data samples. To check if the data selected using the div-k selection technique is indeed diverse, we compute the diversity score for each task and data selection strategy. Figure 2 shows the trend of diversity scores. It is clear that the diversity of the 7500 sentences selected using div-k technique is greater than the diversity of the sentences selected via top-k and rand-k across all tasks.

Recall that the prompts to the LLM also contained domain information of the target task. To evaluate the impact of domain-specificity of the generations on zero-shot performance, we create two cross-domain datasets in the medical and law domain (unrelated to the target task domains).

Table 8 shows zero-shot results of the student models trained on the in-domain and out-of-domain (law, medical) augmented datasets for the Marathi sentiment and XNLI Hi tasks. We observe that the student model trained on out-of-domain datasets perform comparably to the student models trained on in-domain data. This shows that models can effectively learn task information from augmented data even if it comes from domains that differ from the target task.

To further disentangle the roles of task and domain, we made an attempt to capture sentiment richness present in the in-domain vs out-of-domain data for the Marathi SA task. We use a sentiment lexicon⁶⁶6We used the SentiWordNet 3.0 sentiment lexicon available at: https://github.com/aesuli/SentiWordNet?tab=readme-ov-file). consisting of a positive and negative sentiment score for each word along with its POS tag, word sense, etc. We simply add the corresponding scores for each word in the corpus found in the lexicon (along with matching the POS tag) and normalize the sums by the word count of the corpus. Table 9 shows that the computed sentiment scores for the generated out-of-domain datasets are much better than the in-domain dataset for Marathi SA. Here, overall score is calculated by summing both the positive and negative scores for each word. Hence, it is plausible that the sentiment richness in the generated out-of-domain datasets compensates for the domain mismatch, thus yielding comparable or slightly better results in Table 8. (Tables 28, 29 in Appendix E show examples of generations from different domains.)