The messages which constitute the corpus were obtained from public Telegram channels in which users tend to post messages which oppose the mainstream discourse related to the COVID-19 pandemic. Even though the social network X (formerly known as Twitter) is commonly used as a source for this kind of research (Langguth et al., 2023), we decided to opt for Telegram messages for several reasons: firstly, because they tend to be longer than X messages, which allows for a deeper analysis of oppositional narratives. Secondly, Telegram has much less restrictive moderation policies, so it is easier for conspiracy theorists to post their opinions more freely (Vergani et al., 2022).

The collection of the messages was carried out as depicted in Figure 1. Firstly, we created a list of 2,273 public Telegram channels, either in English or in Spanish, that contain oppositional non-mainstream views on the COVID-19 pandemic. Locating such channels is challenging since they are familiar only to members of closed groups aligned with channels’ attitudes. The first step in the creation of this list was to use a ‘snowballing’ procedure (Noy, 2008) : one of the researchers became a member of several social networks related to conspiracy theories. Specifically, he joined the extreme right-wing network GAP, several Facebook groups, and started to follow on X (former Twitter) a number of users who were openly critical of the government’s health measures, and/or public disseminators of conspiracy theories. After that, the sequence of characters “t.me/” was used to locate additional oppositional Telegram channels in the aforementioned social networks, since Telegram channels are often identified with the short URL “t.me/”, followed by the name of the channel. The final step was to repeat this last procedure within the Telegram channels themselves, until no new channels were found.

Then the messages were downloaded from the channels and filtered by removing duplicates, short texts (i.e. those containing fewer than 12 tokens), and texts with a large proportion of URLs and mentions. Finally, each message was ranked using a quality index. This quality index is composed of criteria capturing the activity of the channel and the message length. Concretely, the quality score of a message was the sum of the scores of the following sixcriteria: (1) the audience of the channel, (2) the number of authors in the channel, (3) the mean of messages by author, (4) the standard deviation of the mean of messages by author, (5) the number of messages in the channel, and (6) the number of words in the message. Once the messages were sorted using the quality score they were manually reviewed, starting from the top, to discard the messages not related to COVID-19. This way 5,000 comments related to COVID-19 and having highest quality scores were selected for each language. The details of the process of filtering and ranking of the messages can be found in the supplementary material. Table 1 contains basic statistics of the final English and Spanish corpora.

The aim of the first annotation task was to identify whether a message hints at the existence of a conspiracy theory, or whether it is criticizing mainstream views on COVID-19 but do not suggest the existence of a conspiracy. To that purpose, we developed an annotation scheme with two binary labels, Conspiracy and Critical. Each of these labels can have either the value 1 if the message contains the corresponding feature, or the value 0 if it does not contain it. These labels are mutually exclusive (i.e. no message can have the value 1 in both labels simultaneously). Messages annotated with the value 0 in both labels were not included in the final corpus (but are available if required). The total amount of these discarded messages are 767 in English and 360 in Spanish. This low amount of “neutral” messages is due to the fact that the corpus was extracted from Telegram channels which tend to contain oppositional messages rather than general domain ones.

For the first annotation task, we developed an annotation scheme to differentiate between texts hinting at the existence of a conspiracy, and those criticizing mainstream views on COVID-19 but without suggesting the existence of a conspiracy. A comment was labeled as Conspiracy if any of these five criteria were met:

1. 1.

   The message frames COVID-19 or a related public health strategy as the result of the agency of a small and malevolent agent (either known or unknown) operating in the dark, with the intention to carry out a hidden plan. This plan could be to control society, to physically or mentally manipulate citizens or to impose a false narrative, among others (Douglas and Sutton, 2023). For example: "Whether you believe it to be psychological, physical or both, vaccines have been weaponized as a tool for population control. This much is now obvious". It can also be considered a conspiracy if the vaccine is described as some sort of intentional ‘poison’, and the official vaccination campaign is portrayed as a deliberate murder or as an experiment carried out by an (explicit or implicit) agent, who is purposely trying to harm society. For example: “Pfizer knew the Covid jabs were killing 3% of those who received it. This video is an excellent and succinct summary of the dangers of the “poison death jab” and the musical chairs game big pharma media and govt agents have been playing on us. […]”. The hidden plan may be presented in the form of a ‘join the dots’ narrative, i.e. unrelated facts are presented as being connected with the intention of ‘proving’ the existence of a powerful agent who is deliberately responsible of large-scale events. For example: “Looks like #Covid and all the evil caused by the vaccines were just a test run for the larger plan of locking us all into areas codes… Denial is no longer an option”.

2. 2.

   The message implies that the pandemic is not real, and that "a false version of reality has been promoted or protected by the conspirators and their unwitting stooges" (Douglas and Sutton, 2023). For example: “They have already started their bull shit - so the vaccines didn’t work - the last lock downs didn’t work - but these globalist scum want it all to happen again as their immune compromised sheep have no immune system and are filling the hospital beds and dying […]”.

3. 3.

   The message suggests that those who publicly mock or deny the existence of a conspiracy theory and its negative consequences are actually involved in the secret plan and are collaborating with the agents. For example: “[…] Las vacunas experimentales COVID-19 están causando mucho daño. En este caso un infarto un día después del pinchazo. Y los sanitarios lo saben. […]”. (English translation: The experimental COVID-19 vaccines are causing a lot of damage. In this case, a heart attach just one day after the jab. And health staff know about it).

4. 4.

   A distinction is made between the ingroup (i.e. those who know the truth and reveal the conspiracy, ‘us’) and the outgroup (i.e. those who remain ignorant, society at large, ‘them’). The outgroup is homogenized and its members are perceived as being manipulated and/or ignorant. For example: "#NeverForget all of this fake staging they did - to be used as demagoguery for the pedantic sheep to use against one another, shaming their neighbours into mass compliance for a ‘global pandemic’ hoax". However, the simple presence of a narrative ‘them’ is not necessarily an indication of a conspiracy theory: when two opposing ideas are presented, it is quite usual to associate each idea to its own champion social entity, but there is a difference between establishing an opposition with another social group (in the sense of ‘their behavior is irrational’ or ‘they refuse to listen’, which is a feature of critical thinking) and presenting ‘them’ as an enemy (which is a feature of conspiracy theories).

On the other hand, messages with critical thinking also contain elements of an oppositional narrative, but focus on the way the pandemics has been managed by authorities, as well as the effects and/or efficacy of the vaccine. A message was labeled as Critical if any of these four criteria were met:

1. 1.

   The official vaccination strategy and/or the efficiency of vaccines is called into question (e.g. due to the fact that they were developed and commercialized in a very short period of time). This critical stance does not rely on esoteric explanations and the comment does not deny the validity of scientific protocols, but it rather provides what the author considers to be factual data or objective evidence (e.g. the author’s first-hand experience, second-hand accounts from acquaintances or other online users, etc.). For example: "YouTube has blocked renowned physician Dr. Drew Pinsky from going live on its platform after he showed a vaccine-related eye injury he incurred after receiving an experimental Covid jab. […]".

2. 2.

   The comment establishes a link between the vaccine and a number of side effects which are not included in the mainstream narrative. This causal relationship is sometimes referred to as a ‘crime’ due to a (perceived) unusually high mortality after the vaccination campaigns, but the comment does not imply any sort of deliberate intentionality. For example: "The Spike Protein’s Impact: “Vaccination may cause our overall immune system to fail to fight against such bad things” - Shigetoshi Sano, M.D., PhD., Professor and Chair, Kochi University School of Medicine".

3. 3.

   Asking for information about potential links between the vaccine and its effects on public health is also considered to be a feature of critical thinking, due to the context in which such requests are taking place (i.e. it is assumed that by making use of this kind of channels the author expects a different answer from what could be obtained at an official health center). For example: "Alguien del grupo tiene o puede tener contacto con médicos/sanitarios que vivan en Reino Unido (zona de Escocia) y estén ayudando/aconsejando a vacunados? Necesitaría algún teléfono, grupo de Telegram… lo que sea que sirva de ayuda. Gracias." (English translation: "Does anyone in the group have or may have contact with doctors/healthcare providers who live in the UK (Scotland area) and are helping/advising vaccinated? I would need any phone numbers, Telegram groups… anything that would be helpful. Thanks"). However, if the comment is simply asking for or giving information of any other kind (e.g. a phone number to contact someone, where a given product can be bought, etc.), it is not considered to include either a conspiracy theory or critical thinking, and is therefore annotated with the value 0 on both labels.

4. 4.

   The comment supports a deliberate rejection of the vaccine and/or expresses dissatisfaction with the negative consequences that this rejection entails (e.g. social discrimination, mobbing, administrative constraints, etc.). Oftentimes, the group of the vaccinated and the group of the unvaccinated are presented as antagonistic, but there is no allusion to an agent with a hidden agenda. For example: "Si vives en España,la vacunación es voluntaria. Si los superiores les dicen que se inoculen, que te lo dé por escrito, como no lo van hacer, grábalo y te servirá ante un juicio. Hay muchas personas como tú. RESISTE" (English translation: "If you live in Spain, the vaccine is optional. If your superiors tell you to get jabbed, ask them to give it to you in writing, since they are not going to, record it and it will help you in court. There are many people like you. RESIST").

A key aspect that serves to distinguish conspiratorial thinking from critical thinking is that the latter emphasises impersonal, systemic causes while the former focuses on the agency of a secret group. Recurrent themes in critical thinking are the questioning of the efficacy of vaccines, criticism of the pharmaceutical industry, criticism of the mandatory nature of vaccines, and the side effects of COVID-19 vaccines.

The annotation process of this first task was carried out by two teams of annotators (one for the Spanish corpus and one for the English corpus), all of them expert linguists. Each team was constituted of 3 different annotators, either native or highly proficient speakers in the corresponding language (therefore, each message was annotated by 3 different people). The annotators carried out the task independently, and on a weekly basis meetings were held to discuss disagreements among the annotators and also with the help of two senior annotators (the social psychologist who created the taxonomy, and an expert in corpus linguistics). This process lasted about 5 months. In order to assess the quality of the annotation, we performed a number of tests. For the English corpus, the IAA in terms of Krippendorf’s $\alpha$ is of 0.79 for Conspiracy messages and 0.60 for Critical messages, and the average observed percentage of agreement between the three annotators is of 91.4% for Conspiracy and 80.3% for Critical. For the Spanish corpus, Krippendorf’s $\alpha$ is of 0.80 for Conspiracy and 0.70 for Critical, and the average observed percentage of agreement is of 90.9% for Conspiracy and 84.9% for Critical. Finally, the Gold Standard was obtained by means of a majority vote (i.e. agreement between at least two annotators).

The aim of the second annotation task was to identify the main elements of intergroup conflict (IGC) which appear in oppositional narratives. In order to do that, we selected the span of text in which each element appears and assigned it the corresponding label. The tagset includes six labels, which can be applied both to messages containing a conspiracy theory and messages containing critical thinking. In this section, we describe the labels which compose the tagset, and also present the criteria for span delimitation.

We identified the following six categories of narrative elements (see Figure 2 foran example annotation of a Conspiracy message, and Figure 3 for an example annotation of a Critical message.):

1. 1.

   Agents (A): Those responsible for the actions and/or negative effects described in the comment. In Conspiracy, it could be the hidden power that pulls the strings (in Figure 2, "Private owned WHO", "investors like Bill Gates", "pharma companies" and "very evil beings"). In Critical, it could be the actors that design the mainstream public health policies (in Figure 3, "White House chief medical advisor Dr. Anthony Fauci" and "the lead of CDC director Rochelle Walensky, who questioned natural immunity").

2. 2.

   Facilitators (F): Those who collaborate with the agents and contribute to the execution of their goals. In Conspiracy, they could be governments or institutions which, either intentionally or unwittingly, collaborate with the conspirators and help the conspiracy move forward (in Figure 2, "the world governments ruled by their puppets", "their media", "the media" and "governments"). In Critical, they could be healthcare workers, mass media or authority figures who abide by governmental instructions (in Figure 3, "university hospitals" and "the vaccinated work - from - home hospital administrators who are firing her for not being vaccinated").

3. 3.

   Campaigners (C): Those who oppose the mainstream narrative. In Conspiracy, those who know the truth and expose it to society at large (in Figure 2, "those awake already"). In Critical, those who oppose the enforcement of laws and/or refuse to follow health-related instructions from the authorities (in Figure 3, "Dr Martin Kulldorff").

4. 4.

   Victims (V): Those who suffer the consequences of the actions and decisions of the agents and/or the facilitators. In Conspiracy, the people who are deceived by those in power, and suffer, become ill, lose their freedom, or die as a result of a hidden plan (in Figure 2, "people", "most people" and "regular people"). In Critical, the people who receive the negative consequences of the actions and the decisions made by those in power, and also suffer, lose their freedom, become ill, or die as a result of wrong decisions (in Figure 3, "all nurses, doctors and other health care providers").

5. 5.

   Objectives (O): The intentions and purposes that the agents are trying to achieve. In Conspiracy, the goals of the conspirators (in Figure 2, "agenda" and "destroying us"). In Critical, the goals of public authorities, pharmaceutical companies, organizations, etc (in Figure 3, "pushing vaccine mandates").

6. 6.

   Negative Effects (E): The negative consequences suffered by the victims as a result of the actions and decisions of those in power and/or their collaborators (in Figure 2, "the constant fear mongering" and "pay a hefty price, often with their health, lives, the loss of their loved ones"; in Figure 3, "will be fired if they do not get a Covid vaccine").

The decision of where each label should start and end is not as straightforward as it may seem; for this reason, we established the following criteria for span delimitation (in the examples given below, the span is enclosed within square brackets, and the initial of each label appears afterwards within round brackets):

1. 1.

   In general, punctuation marks are excluded from the span. For example, “[…] like you and everyone else who isn’t getting paid by these satanic billionaires and isn’t in on the whole thing? [The government](F)?". However, exclamation and question marks thought to be relevant for the interpretation of the label can be included. For example: [She leaves behind her year old son… How many more will we lose, how many children?](E). Also, if an opening punctuation mark has already been included in the span (e.g. quotes, parenthesis, etc.), its corresponding closing punctuation mark must also be included. For example: “Why SARS-Cov-2 ["vaccinated" people](V) can test HIV+".

2. 2.

   Paralinguistic elements, such hashtags, which are considered to be relevant for the interpretation of the comment can be labeled on their own or be included in an adjacent fragment if they both have the same communicative function. For example: "Another super fit athlete dies suddenly at the age of 35. I wonder what could be causing all these youg, fit athletes to die suddenly? #COVIDVaccine [#GenocideAgenda](O)".

3. 3.

   The label of categories relative to actors (i.e. Agents, Facilitators, Campaigners and Victims) must include the syntactic determiner of the noun phrase (articles, possessive determiners, etc.). For example: “[The billionaire Luciferian Elites](A) think of themselves as the apex predators of the world and therefore deserving of inheriting the Earth.".

4. 4.

   Each individual element in an enumeration must be annotated separately, with its own label. For example: “[…] while [Bill Gates](A), [the Rockefeller and Ford Foundations](A), and [the World Bank](A) control 10 per cent of the world’s germplasms […]".

5. 5.

   Only linguistic elements which have full semantic content can be labeled; thus, purely coreferential elements (personal pronouns, demonstrative pronouns, clitic pronouns, etc.) must not receive their own label unless they are part of a larger phrase which contains semantically full elements. For example: “[…] Bill Gates (GAVI, WHO, CDC, Microsoft, most vaccines etc etc) set on destroying and enslaving humanity, by poisoning and starving [us, common people](V)".

6. 6.

   In all of these categories referring to an actor of the oppositional narrative, the label must include those elements which help to identify and narrow down the referent (adjectives, defining relative clauses, etc.). For example: “[…] the soldiers of [the satanic billionaires who like to call themselves Elites](A).”.

The second annotation task was carried out by two teams (one per language) composed of two linguists each. Both teams were supervised by the same senior annotator, with whom they discussed doubts on a weekly basis. In turn, whenever it was necessary, this senior annotator held further meetings with another senior annotator and the psychologist who created the taxonomy to clarify doubts regarding the implementation of the annotation criteria. These criteria were updated in the annotation guidelines through an iterative process (i.e. we started with a basic set of criteria, which were adapted during the initial periods of the task to make sure they could accurately be implemented onto our dataset). In order to assess the quality of this annotation task, the Gamma ($\gamma$) measure (Mathet et al., 2015) was used to calculate the IAA. Following our preliminary annotation guidelines, we annotated an initial batch of 300 texts, which yielded an average $\gamma$ of 0.43. After revising and discussing the annotation of this first batch, we fine-tuned the guidelines and annotated a second batch of 300 texts with the updated criteria, which yielded an average gamma of 0.57. Finally, a third batch of 300 annotated texts yielded an average gamma of 0.61, which we deemed acceptable since it is close to or above the average agreement of other highly conceptual span-level annotation schemes (Da San Martino et al., 2019; Weimer et al., 2022a). After that, the rest of the corpus was annotated following the final version of the annotation guidelines. Following this final version, the IAA was calculated in batches of 200 texts. Whenever the average gamma result of a batch was below 0.6, the worst quality messages within this batch were discussed and reannotated in order to guarantee the quality of the annotation. After this reannotation, the final average gamma of the task was of 0.72. Given the difficulty of such a subjective and nuanced task, we deem this result as more than acceptable.

The Gold Standard (GS) of this second annotation task was automatically created applying the following criteria to resolve cases of disagreement:

1. 1.

   When the annotators agreed on the label but disagreed in the precise limits of a given span (i.e. there was partial overlapping), the GS consisted of the union of both spans. For example, if Annotator 1 labeled as Facilitators the span “administrators and superiors” and Annotator 2 used the same label but selected the span “superiors who reportedly never intended to approve them in the first place”, the GS was created by joining both spans, i.e. “administrators and superiors who reportedly never intended to approve them in the first place”.

2. 2.

   When only one of the annotators labeled a given span, or when there was a disagreement in the labels for the same span, the GS did not incorporate either annotation.

Table 2 shows the amount and the percentages of spans in the GS that have been annotated with each label for each category (Conspiracy or Critical). The low number of certain labels in this table might be due to the way the GS was created, especially in spans that can often be interpreted either as Agent or Facilitator in many messages, which results in label disagreement. The application of criterion (2) results in these instances not being present in the GS. Following the learning with disagreement paradigm (Uma et al., 2021) we provide not only the GS but also the individual annotations of each annotator.

The final step in the creation of the XAI-DisInfodemic corpus was the anonymization of sensitive personal data contained in the messages. Personal data can be proper nouns of anonymous people (i.e. not public figures), phone numbers, email addresses, bank account numbers, and mentions. The procedure we implemented is a combination of automatic and manual approaches:

1. 1.

   Phone numbers and bank account numbers were automatically anonymized.

2. 2.

   Mentions (@username) and emails ([email protected]) were automatically extracted, and a senior annotator who was familiar with the corpus decided which usernames needed to be anonymized (i.e. those belonging to private individuals, rather than public figures or Telegram channels). The anonymization process consisted in replacing the characters of an username with a randomly generated sequence of alphanumeric characters, with two random numbers added at the end.

3. 3.

   Proper nouns were selected automatically, using the spaCy tool²²2https://spacy.io/, and reviewed manually by the same senior annotator, taking into account the context in which they appeared. For each proper noun that needed to be anonymized, an alternative replacement was manually given, so that gender and format (e.g. first name, full name, short versions of a name, etc.) were maintained. This step is crucial to ensure that co-reference relationships and gender agreement are kept within each message, so that the cohesion of the discourse is not lost.

We first tackled the task of distinguishing between critical and conspiratorial messages, set as a binary classification problem (with Conspiracy as the positive class³³3We note that, in our experiment, the performance of the transformer-based classifiers is invariant to the definition of positive class: the results (rounded to two decimal places) remain the same when Critical is the positive class.) on Gold Standard data. We experimented with BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) models, using the original English models and their Spanish counterparts, BETO (Cañete et al., ) and BERTIN (Rosa et al., 2022). RoBERTa builds upon BERT by using an improved training procedure and more data. For English messages, we also experimented with DeBERTaV3 (He et al., 2023), an advanced model that builds upon the BERT/RoBERTa architecture by using improved pre-training and by disentangling the content and the position attention.

We built the classifiers by fine-tuning the transformer models from the huggingface⁴⁴4https://huggingface.co/models repository, and we used case-sensitive “base” versions. The number of tokens was set to 256. We tuned the models for 3 epochs using the AdamW optimizer, learning rate of $2e^{-5}$, slanted triangular LR scheduler with a 10% warm-up period, a batch size of 16, and a weight decay of 0.01. All the layers of the transformers were fine-tuned. The dropout rate for the classification head was 0.1. In order to obtain robust performance estimates, the results were computed using five-fold cross-validation (4K train and 1K test texts), with separate per-fold random seeds for initialization of model parameters.

In order to compare the performance of models with expected human agreement, we computed the agreement between annotators in terms of the binary F1 score. First, the annotations (Section 4.1) were used to extract binary Critical vs. Conspiracy labels for each annotator. Then these labels were viewed as class predictions and evaluated against two definitions of ground truth: labels of other annotators (human-vs-human), and the labels obtained by the majority vote (human-vs-gold). In each case, the final score was formed by averaging over all possible (prediction, ground truth) pairs. Note that the human-vs-human scores are pessimistic estimates of human performance, since labels of a single annotator represent a ground truth that is more variable than the average. On the other hand, the human-vs-gold scores are optimistic estimates since the individual predictions are merged into the average, and can therefore influence it.

Table 3 contains both the models’ and the inter-annotator binary F1 scores. These results show that the classifiers performed well both for Spanish and English, with the English models achieving higher F1 scores. Detection of Critical messages achieves very high scores for all the models and languages, while the Conspiracy scores are somewhat lower, especially for Spanish. For English, DeBERTaV3 performed best, while for Spanish BERT outperformed RoBERTa.

When compared to human scores, all the English models outperform the (pessimistic) human-vs-human scores and stay below the (optimistic) human-vs-gold scores. The best-performing DeBERTa model is close to the human-vs-gold scores. For Spanish, the models’ scores are below both the human-vs-human and human-vs-gold scores, with the performance on critical texts being much closer to the human thresholds than the performance of conspiracy texts.

Possible reasons for variation in the models’ performance across languages are the differences in quality between English and Spanish transformer models, and the differences in language structure, which could make the Spanish classification task more challenging for transformers. The slightly higher human F1 scores for Spanish suggest that the Spanish labeling task is not harder for humans, which is also demonstrated by the IAA scores in Section 3.2.

Confusion matrices of the best-performing models for English (DeBERTa) and Spanish (BERT), displayed in Table 4, offer more insight into performance differences between the Conspiracy and Critical classes. For both languages, the precision for the Conspiracy class is lower than the precision for the Critical class. For English, the precision of Conspiracy detection is 88% compared to 95% for Critical detection, while for Spanish the corresponding precision scores are 78% and 90%. In other words, models confuse conspiratorial messages as critical messages more often than they confuse critical messages as conspiratorial, and this asymmetry is more pronounced for Spanish. This is likely due to the fact that conspiratorial messages can be subtle and indirect in their support for conspiracy theories and are therefore more likely to be confused as critical messages. Interestingly, the IAA scores in Section 3.2 indicate that human agreement on messages being conspiratorial is higher than on messages being critical. This indicates that humans are able to reliably detect conspiracies based on subtle and indirect cues. Therefore, a reasonable first step in improving the Conspiracy vs. Critical classifiers would be an analysis of misclassified conspiratorial messages and the identification of the features that are reliable signals of conspiratorial discourse.

To summarize, the previous results demonstrate that the Critical vs. Conspiracy task is tractable, and that the expected performance of transformer classifiers is good for Spanish and very good for English, which is an expected result of combining high-quality annotations with strong transformer classifiers. High F1 scores for the detection of critical messages suggest that these classifiers could be reliably applied to make the automatic conspiracy-filtering tools more fair. In this use-case, the texts labeled as Critical could either be automatically unflagged as Conspiracy, or routed to humans for closer inspection.

We proceeded with the identification, on the fine-grained level of token spans, of the six elements of oppositional narratives defined in our annotation scheme (for details, see Section 3.2). Our task includes both long and overlapping spans (of different categories). Similar tasks have been carried out in the context of propaganda detection (Da San Martino et al., 2019, 2020), skill extraction (Zhang et al., 2022), and the analysis of literary texts (Weimer et al., 2022b).

For evaluation, we used the adapted F1 measure of Da San Martino et al. (2019) which accounts for partially correct predictions by looking at span overlap (span-F1). This approach offers a fairer evaluation in tasks with long spans, and with inherent subjectivity of the span boundaries. For tasks like traditional non-nested NER, where named entities are shorter and expectedly have well-defined boundaries, exact matching is a reasonable method of evaluation.

We approached the task by fine-tuning transformer models with token classification heads. To account for the possibility of overlapping spans with different categories, we used 6 separate per-category heads that performed BIO sequence tagging. We employed multi-task learning (Ruder, 2017) by connecting the per-category taggers to the same transformer backbone. Multi-task learning has several advantages such as improved regularization and implicit data augmentation (Ruder, 2017), and the described approach was successfully deployed for a similar task of span-level skill extraction (Zhang et al., 2022).

We experimented with the same transformers used for binary classification (Section 4.1) and we used the same configuration and hyperparameters. The exception was the number of epochs, which we increased to 10 in order to accommodate for the increased task complexity. Gold Standard data was used and it was adapted for multi-task learning by creating a separate training instance for each text and each span category, which resulted in a large proportion of texts without spans. In order to avoid overfitting to the unlabeled tokens, empty texts were downsampled (per category) to 10% of the original size. Five-fold cross-validation was used to obtain performance estimates, in the same way as in Section 4.1.

The results, presented in the top half of Table 5, show the average and per-category span-F1 scores for both languages. For English, DeBERTaV3 outperforms RoBERTa, which outperforms BERT, while for Spanish RoBERTa outperforms BERT. The per-category performance correlates well with the number of spans annotated with a category, i.e. with the size of the training data for the corresponding token classifier. For example, the (less frequent) Facilitators and Objectives categories are harder to detect. Additionally, the Spanish BERT and RoBERTa models are better than their English counterparts for the Victims and Negative Effects categories which are more frequent in Spanish, while for most of the other categories the English models work better. For English, the most sophisticated DeBERTaV3 model achieves noticeable performance gains for most of the categories.

Additionally, we experimented with the SpanBERT model (Joshi et al., 2020) as the transformer backbone. SpanBERT is a model based on a pre-training method “designed to better represent and predict spans of text” (Joshi et al., 2020). The SpanBERT model achieved an inferior performance of 0.31 average span-F1 on English texts (Spanish SpanBERT model does not exist). This is in line with the skill extraction experiment (Zhang et al., 2022), in which the SpanBERT model was not the best solution and it rarely outperformed BERT.

Finally, we experimented using CRF (Lafferty et al., 2001), a probabilistic model that learns patterns of transition between token classes⁵⁵5We used the pytorch-crf package: https://pytorch-crf.readthedocs.io, as the final token classification layer. We experimented with two CRF-based approaches: one in which the CRF layers were optimized together with the transformer (end-to-end), and the other in which the CRF layers were optimized after training and freezing the transformer (fine-tuned). Both of the CRF-based approaches led to an inferior performance. In terms of the span-F1 measure, the average (end-to-end CRF, fine-tuned CRF) results for English were: (0.23, 0.19) for BERT, (0.37, 0.36) for RoBERTa, and (0.47, 0.46) for DeBERTa. For Spanish, the results were (0.32, 0.38) for BERT, and (0.23, 0.25) for RoBERTa. The likely reason for the inferior performance of CRF is that its inductive bias (an assumption that there is valuable information in the surface patterns of transition between token tags) does not fit well with the nature of the data in our case. This is supported by the average number of spans of the same category per text (in which the category occurs), which is 2.29 for English and 2.39 for Spanish.

In terms of the results on a different but similar task, best span labeling models detecting a single “propaganda” category in English news articles achieved a span-F1 of 0.52 (Da San Martino et al., 2020). While direct comparison is not possible, we take these results as evidence that our approach (average span-F1 of 0.57 for English and 0.50 for Spanish) achieves solid performances on a challenging task.

Regarding RQ2, we can say that while the performance of the proposed approach is not below the top performance expected from a similar task, there is room for further improvement, especially for the less frequent categories. Techniques that could improve performance on the less frequent categories include data augmentation (Feng et al., 2021) and active learning (Erdmann et al., 2019).

While narrative categories in Section 4.2 were detected on the token level, here we will tackle their text-level detection. For each category we assigned binary labels to texts, labeling a text as positive if the category was present in the text, i.e. represented by at least one Gold Standard span. Such per-category binary variables can convey useful information: the presence of spans labeled as Facilitator or Campaigner is a signal of inter-group conflict (for details, see Section 5).

The models and their configuration are the same as for the Conspiracy vs. Critical (CC) classification in Section 4.1, as well as the training and evaluation. The results in Table 5 show that binary detection of the narrative categories is, on average and for most of the categories, a harder task than CC classification (Table 3). Relative model performances are the same: English models obtain better results than Spanish, with DeBERTaV3 being the most successful, and BERT outperforms RoBERTa in Spanish.

The results in Table 5 also show that the binary per-category scores correlate very well with the per-category scores of the sequence labeling models. This is expected since the per-category labeling scores correlate well with the number of spans annotated with a category, which correlates with the number of positive examples for the per-category classification. Similarly as for sequence labeling, the smallest Facilitators and Objectives categories are the most challenging ones and have similar scores.

Regarding RQ3, these results show that text-level detection is a more viable option in applications that require this level or granularity, and that good results can be achieved for most narrative elements. However, the Facilitators and Objectives categories remain a challenge. One could likely improve performance on these less frequent categories by using techniques such as data augmentation Feng et al. (2021) and active learning Settles (2011).

The results indicate that the Facilitators (F) and Objectives (O) categories are the most challenging to detect (Table 5). We focus the error analysis on the Facilitators span since it conveys important information about inter-group conflict (Sections 3, 5). We hypothesize that the model confuses the Agent (A) and Facilitator (F) labels, which have a similar role in terms of the “friend vs. foe” discourse. To examine their relationship, we introduce, for each of the categories, a text-level variable that codifies classification outcomes: False Negative, Correct and False Positive. For both categories and for each language, we calculated the outcomes of the best-performing classifiers (see Table 6). Predictions for all five folds are pooled to cover the entire dataset. Applying the chi-square test, which assesses the independence between two categorical variables, we found that the classification outcomes for Facilitators (F) are not independent of the outcomes for Agent (A) ($p<0.001$).

Next we perform the analysis of the residuals, which measure how much a particular combination of outcomes influences the dependence of the variables – large residuals influence the dependence more, and vice versa (Sharpe, 2015). Table 6 shows that there are 81 English texts with missed Facilitators and falsely detected Agents, and that this combination strongly influences the dependence between variables (high residual of 4.3). In our context, the values of the residuals show that there is a strong correlation between Facilitators being missed (False Negative) and Agents being incorrectly identified (False Positive), for both languages. For Spanish, incorrectly identifying Facilitators (False Positive) was significantly linked to missing Agents (False Negative). These results show a tendency to misclassify Facilitators as Agents in both languages, and to misclassify Agents as Facilitators in Spanish. Future models should tackle this confusion, possibly by applying a multi-task approach with a loss function modified to penalize the problematic cases.

Firstly, we investigate the difference in the amount of anger and violence words between conspiratorial and critical texts. Since these variables are not normally distributed according to the Kolmogorov-Smirnov test ($p<0.001$), we used the Mann-Whitney U test for comparison of the two groups.

As we observe in Figure 4, conspiratorial messages convey significantly more anger than critical messages, both in Spanish (U=2352490.5, df=3, $p<0.001$) and in English (U=2046581.5, df=3, $p<0.001$), which confirms [H1]. Similarly, we observe that conspiratorial messages convey significantly more violent words both in Spanish (U=2046572.5, df=3, $p<0.001$) and in English (U=2030201.5, df=3, $p<0.001$), which confirms [H2].

Next we test whether a larger degree of IGC is present in conspiratorial messages than in critical messages. To this end, we perform a chi-square test and a residual analysis which helps to interpret the chi-square test: the size of the residual indicates a major contribution of a cell to the chi-square significance (for more details, see Sharpe (2015)). The results confirm a significant relation between the IGC variable, and the conspiracy vs. critical variable, both for Spanish ($\chi^{2}$ = 182.02, df = 3, $p<0.001$) and for English ($\chi^{2}$ = 159.16, df = 3, $p<0.001$). For both languages, the C4 category (which is a clear indicator of IGC) occurs significantly more in conspiracy than in critical messages, while the opposite is true for C1 (indicator of no IGC), which supports [H3]. Table 7 contains the statistics for the combined texts of both languages.

Finally, we tested whether the use of anger and violence words is higher in messages in which IGC is expressed. We use the Kruskal-Wallis test to compare the presence of anger and violence words in the different categories of the IGC variable. We observed a statistically significant effect of the IGC variable on the measures of anger and violence words, both in Spanish (H=96.289, df=3, $p<0.001$), and in English (H=79.462, df=3, $p<0.001$). As we can see in Figure 5, there is a significantly larger amount of both anger and violence words in messages which have the highest IGC (C4), and a lesser amount in the messages which have the lowest IGC (C1), supporting [H4].