Argument Mining in Data Scarce Settings: Cross-lingual Transfer and Few-shot Techniques

The are a number of different theoretical approaches to describe the argument structures that can be inferred from text analysis. For instance, Toulmin (1958) identified different functional roles in arguments (evidence, warrant, backing, qualifier, rebuttal, and claim) based on how the conclusion is drawn from evidence in the text. Furthermore, Freeman (2011) investigated how to transfer arguments via diagramming techniques of the informal logic tradition. Others Dung (1995) tried to create a graph-based representation of argumentation by applying non-monotonic reasoning in Artificial Intelligence (AI) and logic programming. Finally, Peldszus and Stede (2013) introduced a diagram structure with models of the textual representation of arguments and globally optimized argumentative relations. They argued that support and attack relations are sufficient to describe the overall relationships between argument components. Moreover, they identified five different types of argument graphs based on the connections that exist between them, namely, one claim having relations with multiple premises, a claim followed by another claim, etc.

In Natural Language Processing Argument Mining (AM) is focused on automatically identifying the argument components and classifying the relations that may exist between them. Following the theoretical models proposed, a number of empirical approaches have been developed in the last few years. Thus, Stab and Gurevych (2017) tackled AM in two different steps. First, they try to locate the span argumentative text and classify the type of component at token level. Second, they classify the relations linking the identified argument spans. In addition to the two step system to address AM, they also generate Persuasive Essays, perhaps the most popular NLP dataset manually annotated with argument structures Stab and Gurevych (2017). Later on, Eger et al. (2017) introduced an end-to-end AM system based on a bi-directional sequence-to-sequence model.

Other work includes Toledo-Ronen et al. (2020), which provides an in detail analysis at argument level of various multilingual datasets, while Rocha et al. (2018) experimented with cross-lingual argumentative relation identification from English to Portuguese.

Finally, Mayer et al. (2020) introduced the first dataset of English medical abstracts annotated for argument component detection and argument relation classification. Subsequently, Mayer et al. (2021) introduced a Transformer-based solution with Gated Recurrent Units (GRU) and Conditional Random Field (CRF) classification layers.

The availability of pre-trained language models allows to apply supervised methods with less amount of annotated data which is why some research in different NLP tasks has focused on few-shot training (Hofer et al., 2018; Fritzler et al., 2019; Li et al., 2022), namely, learning supervised models with very few manually annotated samples. The rise of prompt-based models (Radford et al., 2019; Brown et al., 2020) further increased the interest in learning the task describing the classification objective. This usually involves transforming traditional classification tasks into cloze tasks using textual templates and a predefined set of label words, highlighting the importance of template design in prompt-based learning.

In this line of work, Schick and Schütze (2021) presented a semi-supervised training approach that reformulates input instances into cloze-style phrases. Cui et al. (2021) proposed a template-based method for Named Entity Recognition (NER) by generating templates for each entity from a given example. However, template-based approaches are better suited to sentence-level tasks where the complexity of the templates remain manageable. As an alternative, EntLM Ma et al. (2022) proposed a template-free few-shot learning approach for sequence labelling tasks. Their method is based on computing a set of label words from the input text and replacing the entity-specific tokens with these label words in the training sample. EntLM obtains state-of-the-art results which is why we use it in this paper as the representative of few-shot learning for argument component detection. Huang et al. (2022) and Das et al. (2022) propose few-shot learning for NER involving contrastive learning via prompt-based meta-learning. However, their methods require large amounts of data to first train the model before adapting it with a handful of examples for various label sets.

Previous work on cross-lingual sequence tagging mainly focuses on tasks such as part-of-speech (POS) tagging, named-entity-recognition (NER) (Gaddy et al., 2016; Yang et al., 2017; Agerri et al., 2018; Chen et al., 2018; Liu et al., 2020), and Opinion Target Extraction (OTE) (Agerri and Rigau, 2019). García-Ferrero et al. (2022a) compared model-transfer and data-transfer approaches on a variety of sequence labelling tasks, datasets, and languages. They conclude that model-transfer using pre-trained multilingual language models such as XLM-RoBERTa-large Conneau et al. (2019) outperform data-transfer methods.

Closer to our work, Eger et al. (2018) generated parallel German and Chinese versions from English by applying manual and automatic translation and label projection to experiment with data-transfer approaches based on cross-lingual embeddings. They concluded that, while machine translated data degraded results when used for training a supervised model for the target language, results were promising enough to continue working on that research direction. Thus, Sousa et al. (2021) translated Persuasive Essays into Portuguese for further cross-lingual experimentation. However, it should be noted that current model-transfer, few-shot and supervised techniques based on multilingual pre-trained language models are clearly superior to the methods used at the time, which makes the purpose of our work rather relevant.

The main objective of Few-Shot Learning (FSL) is to generalize while learning from a small portion of data. In order to perform FSL, the data is sampled into smaller subsets and provided to the model. While state-of-the-art methods on few-shot for sequence labelling have been focused on the training method, they have not usually paid any attention to the data sampling technique Ma et al. (2022). In this paper, we demonstrate the importance of data sampling for a sequence labelling task such as Argument Mining.

We sample the data in two ways, using a method called k-shot (based on Ma et al. (2022)) and another one named k-percent, where $\textit{k}\in\{5,10,20,50\}$. In the k-shot method, each of the subsets contains exactly k argument component sequences of Claim and Premise. With the k-percent sampling method we calculate the k proportion for each argument component from the full data to reflect the distribution. The distribution of the sequences sampled with k-percent method and k-shot are shown in Table 3. The sequences in every sample are selected randomly in a greedy manner.

AbstRCT contains texts annotated with labels Claim, Premise, and O (Outside). One sentence could belong to one or more argument component classes from the beginning until the end. In many sequence labelling tasks, the span of the components to predict consists of several words that make up only a part of the sentence, whereas in argument mining argument components can constitute a whole sentence. Hence, for the few-shot training, it is crucial to include examples without any argument components separately, namely, examples in which every token in the sequence is labeled with the O class. If such examples are not included, the few-shot model fails to learn to classify sequences as non-arguments.

The data has a sentence-by-sentence split, where each token in the sentence is annotated with the labels following the IOB2 schema, meaning that the beginning of the argument is tagged as B- followed by the argument component class name (Claim or Premise), the rest of the argumentative tokens are labelled with I-, and non-argumentative sequences are labelled as O. Since one sentence holds one or more argument types, and they tend to be lengthy, a considerable imbalance between B- and I- tokens is created. In Table 3, we provide the distribution of the data at token level to show the imbalance in the number of tokens that are marked as B-, I- or O.

Along with sampling the training data for each language, we additionally merge all the training sets from every k-percent sampling into one to perform multilingual experiments. Therefore, the multilingual k-percent sample is a combination of k examples from each language from the k-percent sample.

Data-transfer involves generating training data in the target language by translating and projecting the annotations from the original English language to Spanish, French and Italian. This process was described in Section 3. The translated and projected training data is then used to fine-tune pre-trained encoder language models.

Initially, we separately fine-tune multilingual BERT Devlin et al. (2019), on the training sets of English, Spanish, French, and Italian AbstRCT corpora and evaluate the resulting models for each of the languages in a monolingual setting²²2Preliminary experimentation showed that mBERT outperformed other multilingual encoder-only models such as XLM-RoBERTa or mDeBERTa-v3-base. See mDeBERTa results in Appendix D..

We also tested data-transfer in a multilingual setting by fine-tuning multilingual BERT on the training sets for the 4 languages. Finally, both monolingual and multilingual settings were evaluated using both post-processed and manually corrected versions of the data (French, Italian and Spanish).

Model-transfer is facilitated by pre-trained multilingual language models such as mBERT by enabling them to label sequences in languages on which they have not been explicitly trained on, relying on their multilingual or crosslingual abilities. Thus, model-transfer allows to perform AM for languages for which no annotated data is available by training in English and generating predictions in the target language (French, Italian and Spanish). In our experiments, this amounts to fine-tuning mBERT using English data and evaluating its performance on test data from the other three languages.

Few-shot learning exploits limited annotated examples to train models, striking a balance between data scarcity and task complexity.

Ma et al. (2022) proposed a template-free method for few-shot prompting for Named Entity Recognition (NER) by tackling it as a Language Model (LM) task with an Entity-oriented LM (EntLM) objective to solve the NER task. This avoids generating a new template corpus for each example in the data. We use this method in our experiments as it represents current state-of-the-art, at the time of writing, for sequence labelling in few-shot settings. Their approach consists of first retrieving class-specific words called label words from a pre-trained model, and predict those label words at the position of each entity. They propose several ways of computing these label words, and in this work, we used the method based on the frequency, namely, we select the words that are the most frequent for the given class. We generate 10 such label words for each class.

Following EntLM’s methodology Ma et al. (2022), for every k sample three randomly sampled training sets are created. Training is then performed on each of these datasets over four iterations, and subsequently, sequence-level F1-scores and standard deviations are calculated.

In addition to the monolingual experiments, we also carry out multilingual experiments by combining all the French, Spanish, Italian, and English data. More specifically, we merge one sampled training file from each language of the k-percent sampling method. Evaluation is then conducted separately for each language.

Finally, we also compare EntLM with fine-tuning mBERT on few-shot settings.

Table 4 displays the F1-scores derived from the argument component detection experiments using full in-domain data across all the experiments. The rows corresponding to the monolingual data-transfer category present the results obtained from training and evaluating in the corresponding language. Similarly, multilingual data-transfer refers to the merged training set consisting of all 4 languages and evaluating each language separately. Cross-lingual refers to model-transfer, namely, training in English and evaluating in the other 3 languages. The last column corresponds to the average between all the results per language across all test sets. For a fair comparison, the average of the cross-lingual model transfer includes the F1-score of the monolingual English.

Results show that, contrary to previous work on crosslingual transfer García-Ferrero et al. (2022b), monolingual data-transfer clearly outperforms cross-lingual model-transfer for argument component detection. Another interesting point is that multilingual data-transfer obtains the overall best results outperforming also the original English gold results. This means that data-transfer may be employed as a cost-free data-augmentation technique.

If we look at the results obtained when fine-tuning with the post-processed data, results indicate that data-transfer may be used in a fully automatic way, restricting the manual correction of the projected labels to the generation of evaluation sets.

Figure 1 reports the results of few-shot using both sampling methods (k-shot and k-percent) for the data trained by means of both EntLM and fine-tuning techniques.

The first point to mention is that data-transfer also outperforms the few-shot prompting approach for sequence labelling proposed by EntLM. Furthermore, and quite surprisingly, fine-tuning remains competitive with respect to EntLM with the k-shot sampling while it is quite superior when tested on the percentage sampling. We hypothesized that k-percentage sampling produces better performance due to the higher proportion of outside tokens. In fact, when fine-tuned with 20% and 50% of the data performance is comparable to that of data-transfer and model-transfer results.

With respect to the multilingual experiments, one training sample from each k-percent sampling was merged into one training set, fine-tuned, and tested on each language (Table 5). As observed in Figure 1, fine-tuning with 50% of the data (dash lines) produces results almost as high as 100%. Furthermore, results demonstrate that merging 20% of the data performs slightly worse than the model trained on the full data.