Sina at FigNews 2024:
Multilingual Datasets Annotated with Bias and Propaganda

Our understanding of "bias" is based on the work done by the United Nations Committee on the Elimination of Racial Discrimination and the European Commission against Racism and Intolerance European External Action Service (n.d.). We define the notations ‘bias’ and ‘propaganda’ based on the UN and EU accounts, as:

Bias: is generally understood as an inclination or prejudice towards or against a particular person or group, often in a way considered to be unfair. In other words, it is an unreasonable preference or dislike that prompts someone to behave in a discriminatory way, often based on unfair judgment. This bias is typically based on prohibited grounds of discrimination such as race, religion, language, nationality, ethnicity, social background, gender, and others.

Classifications of Bias: we adopted the same classes provided in the Shared Task: (1) Biased against Palestine,(2) Biased against Israel, (3) Biased against others, (4) Biased against both Israel and Palestine, (5) Not Applicable, (6) Unclear, and (7) Unbiased. We also introduced a new feature called "Type of Bias", which can be either: (a) $Explicit$ (\<تحيز صريح>) if it is obvious and evident in the post, (b) $Implicit$ (\<تحيز ضمني>) if it is clear but not evident in the post, and (c) $Vague$ (\<تحيز مبهم>) in case of indirect and ambiguous bias. This feature is important from a methodological viewpoint as it encourages the annotators to think more during classification. If a post contains biased content but not in a direct way it can be accounted as implicit.
Propaganda: misleading ideas or statements that can distort the truth or omit facts to promote a specific political or social agenda. These ideas are typically published by media outlets. For example, propaganda can take the forms of exaggeration, minimization, spreading doubts, name-calling, labeling, or intentional vagueness. All these forms have the common intention to spread false information and obscure facts.

Classifications of Propaganda: We adopted the four classes provided in the Shared Task: (i) Propaganda, (ii) Not propaganda, (iii) Not Applicable, and (iv) Unclear.

Additionally, we added a new column to classify Propaganda into three types: (1) Propaganda must be deleted: if it contains evident harmful content that poses risks to the safety and security of individuals or groups; (2) Propaganda may be deleted: if we cannot easily judge whether it is propaganda, depending on a specific context; and (3) Propaganda not to be deleted: if it is not clear and lacks harmful consequences and therefore does not warrant deletion.

Remark: Since the data was collected from Facebook posts some cases contain quoted content (e.g. an unbiased post quoting biased content). It was established in the guidelines that a post should not be classified as bias or propaganda based on its quotation, but rather on the post itself.

An Example of the guidelines mentioned earlier regarding quoted content is as follows: “Hamas and Islamic Jihad spare no effort to exploit religious institutions for terrorist purposes,” the IDF said in a statement. This post is annotated as unbiased because it is a direct quote and does not include any additional commentary or interpretation.

To evaluate the quality of our annotations, we used the $F1$-score and Cohen’s Kappa Cohen (1968) to compute the agreement between the annotators. The results are shown in Table 1.

The task organizers allocated 100 posts ($10\%$) from each batch for IAA, including $20$ posts randomly selected from each language. Overall, we annotated $12,000$ posts, resulting in an IAA dataset of $1,200$ posts. These were distributed among our $10$ annotators following this scheme: (1) each annotator received $240$ posts, (2) each post was annotated by two different annotators, and (3) the $240$ posts assigned to each annotator were distributed among four other annotators. Consequently, each pair of annotators had $60$ posts in common.

All vs. Binary IAA: to evaluate whether a (dis)agreement was dominated by a certain class, we mapped all labels into binary categories: ($Bias$ oder $NotBias$ $and$ $others$) and ($Propaganda$ oder $NotPropaganda$ $and$ $others$). Table 1 demonstrates no class dominance because All and Binary evaluations are close to each other.

Looking at all Cohen’s scores in Table 1, the average is $0.808$ for bias, which is a "very good" agreement, and $0.7015$ for propaganda, which is a "good" agreement overall. Agreement on propaganda was more challenging but the results are enhanced when it is considered as a binary.

Annotation Phase: The dataset consisted of $12$ batches, comprising $10,800$ posts from the Main sheet, and $1200$ posts from the IAA sheet. The annotation was carried out in two phases:

1. 1.

   Phase One: We distributed Batch01 and Batch02, each with $180$ posts, among team members. To ensure consistency with the guidelines, an expert reviewed all student annotations for these batches and provided feedback.

2. 2.

   Phase Two: we assigned each annotator $450$ posts from two different batches. This step allowed us to complete the annotation of all $12$ batches (i.e. $12k$ posts).

   Set quality standards

   To set quality standards among annotators, after the annotation process was complete, each pair of annotators who had annotated the same data held meetings to review the selected posts they disagreed on. They discussed their differences, and if they reached an agreement, they would change the label accordingly. If they could not agree, they kept the original label.

Table 2 displays the final results provided by the shared task organizers. Our Sina team achieved the third and second place in the IAA Quality and Quantity tracks for the Bias and Propaganda subtasks, respectively. In addition to third place in Propaganda Guidelines.

Table 3 and Table 4 illustrate the distribution of the bias classes and types of bias across languages respectively. Table 3 shows that about $27\%$ of the posts are biased against Palestine and $63\%$ of the posts are unbiased. Most of the bias against Palestine originated from French posts. Table 4 gives more statistics about the types of bias. As shown in this table, most of the posts annotated as $Explicit$ bias are in Hebrew.

For propaganda results, Table 5 illustrates the distributions of propaganda classes across languages, which shows that $31$% of the posts ($3333$) are annotated as "Propaganda", and $66\%$ ($7084$) are "Not Propaganda". The majority of the propaganda originated from French posts. Table 6 illustrates the distribution of the type of propaganda classes among languages. As shown in the table posts that were classified as propaganda must be deleted were in French with 348 posts.