Nullpointer at CheckThat! 2024: Identifying Subjectivity from Multilingual Text Sequence

The first step in the pipeline is data preprocessing, which involves cleaning and transforming the raw text data into a suitable format for model. The preprocessing steps include:

• •

  Demojization: We convert emoji characters into their text descriptions using a demojizer to ensure consistent input to the model.

• •

  Removing users and links: We remove user mentions and URLs from the text, as they are not relevant for subjectivity classification.

• •

  Handling poorly formatted TSV files: Some of the provided TSV files were poorly formatted, so we use a custom dataset class to handle the processing instead of relying on the pandas library.

We also experiment with additional preprocessing techniques such as part-of-speech (POS) tagging and attention masking, but find that they do not significantly improve the performance of the model.

For the subjectivity classification task, we choose to fine-tune pre-trained Transformer models that have been previously trained on sentiment analysis tasks. Specifically, we use the ’MarieAngeA13/Sentiment-Analysis-BERT’ model, which is a BERT-based model fine-tuned for sentiment analysis. We find that this approach of using a model already fine-tuned for a related task (i.e., multi-task learning) yields better results compared to fine-tuning a pre-trained model from scratch. The code and data can be found in the GitHub repository https://github.com/Abrar-Abir/CLEF2024task02.

The training was conducted on a remote Dell server running the latest Ubuntu 22 OS with 512 GB RAM and 24-core CPU. The server was equipped with NVIDIA A100 GPU with 80 GB GPU memory. We employ several training strategies to improve the performance of the model listed below.

• •

  Label mapping: The pre-trained sentiment analysis model is designed for three-class prediction (positive, neutral, negative), while our subjectivity classification task requires only two classes (subjective and objective). We experimented with different label mappings and found that mapping subjective to negative sentiment and objective to positive sentiment yielded the best results.

• •

  Confidence weighting: For the English dataset, we incorporate the confidence level information provided in the dataset (in the ’solved_conflict’ column where 1[true] means conflict was resolved i.e., higher annotation confidence and vice versa). We assign 20% higher weight for the training losses- coming from the annotations with higher confidence (i.e, 1.2 weight)- before passing the losses to the loss function so that backpropagation prioritizes minimizing loss for higher confidence annotation compared to their counterparts.

• •

  Hyperparameter tuning: We experiment with different hyperparameter settings and find that a batch size of 16, learning rate of 2e-5, and training for 20 epochs yields the best performance.

To handle the multilingual nature of the task, we employ machine translation to convert non-English data into English. We use the Google Translator API through the deep translator library for this purpose. While we also experiment with fine-tuning language-specific pre-trained models for non-English languages, we find that translating the training and test datasets to English and using the English model yields better performance. These preprocessing, model selection, and training strategies form the core of the subjectivity classification system. In the following sections, we detail our experimental setup and present the results of the approach.

The dataset for the Subjectivity Subtask consists of sentences from news articles in five languages: Arabic, Bulgarian, English, German, and Italian. Additionally, there is a multilingual dataset that combines all five languages. Table 1 shows the distribution of objective and subjective sentences in the training and test sets for each language. Across all languages, the percentage of objective sentences is higher than that of subjective sentences, with the imbalance being more pronounced in the training sets. This imbalance poses a challenge for subjectivity classification systems, as they need to learn from skewed data distributions. For the Arabic language, the training set comprises 1185 sentences, with 905 being objective (76.37%), and the test set includes 748 sentences, of which 425 are classified as objective (56.81%). In Bulgarian, the training set contains 729 sentences, where 406 are objective (55.69%), and the test set consists of 250 sentences, with 143 objective sentences (57.2%). The English dataset includes 830 sentences for training, with 532 labeled as objective (64.09%), and the test set has 484 sentences, with 362 objectives (74.79%). For the German language, the training set comprises 800 sentences, of which 492 are objective (61.5%), and the test set contains 337 sentences, with 226 being objective (67.07%). For the Italian language, the training set includes 1613 sentences, with 1231 objectives (76.31%), and the test set comprises 513 sentences, with 377 objectives (73.4%). The multilingual dataset combines all five languages and comprises 5159 sentences in the training set, of which 3568 are objective (69.16%). The test set contains 500 sentences, evenly split with 250 objective sentences (50%) and 250 subjective sentences (50%). This comprehensive dataset provides a robust foundation for developing and evaluating systems that distinguish between subjective and objective statements in news articles across multiple languages.

We evaluate our subjectivity classification model using various performance metrics, including macro-averaged F1-score, precision, recall, and accuracy. Table 2 presents the results for each language and the multilingual dataset. Our model achieves the best performance on the German dataset, with an F1 Macro score of 0.79 and an accuracy of 0.81, indicating high prediction correctness. The multilingual dataset obtains good performance, with an F1 Macro score of 0.71, an F1 SUBJ of 0.69, and an accuracy of 0.71. The model also shows good performance for the Italian language with an F1 Macro score of 0.74 and strong subjective class metrics, with an F1 SUBJ of 0.64. On the other hand, the model struggles the most with the Arabic dataset, obtaining an F1 Macro score of 0.49 and an accuracy of 0.52. The performance is relatively lower than in other languages, which shows the difficulty in identifying subjective data. The model performs well in Bulgarian, achieving an F1 Macro score of 0.72 and high subjective class performance with an F1 SUBJ of 0.69. For English, the performance is moderate to good, with an F1 Macro score of 0.68. The model handles subjective data in English relatively better, with an F1 SUBJ of 0.54, precision (P SUBJ) of 0.52, and recall (R SUBJ) of 0.64. The overall accuracy for English is 0.64.

In summary, the model shows the highest performance in German, followed by Italian and Bulgarian, with Arabic being the most challenging language for the model. The performance in English is moderate, and the overall multilingual performance is strong, suggesting the model’s effectiveness across multiple languages but with some variability in specific language performance.

• •

  F1 Macro: The macro-averaged F1 score, which is the harmonic mean of precision and recall across all classes.

• •

  P Macro: The macro-averaged precision.

• •

  R Macro: The macro-averaged recall.

• •

  F1 SUBJ: The F1 score for subjective classification.

• •

  P SUBJ: The precision for subjective classification.

• •

  R SUBJ: The recall for subjective classification.

• •

  Accuracy: The overall accuracy of the model.

We compare the performance of our subjectivity classification model with that of other participating teams in the CheckThat! Lab at CLEF 2024. Table 3 shows the official results for each language and the multilingual dataset. Our team achieves the highest rank in the German and multilingual categories, with Macro F1 scores of 0.7908 and 0.7121, respectively. We also secure the second position in Arabic and Bulgarian. For Arabic, our model achieved second place with a Macro F1 score of 0.4908 and a SUBJ F1 score of 0.37. In Bulgarian, our model also secured second place with a Macro F1 score of 0.7169 and a SUBJ F1 score of 0.69. In Italian, our model ranks third with a Macro F1 score of 0.7430 and a SUBJ F1 score of 0.64. In the English category, our model ranks ninth with a Macro F1 score of 0.6893 and a SUBJ F1 score of 0.54.

These results showcase the competitiveness of our approach in the shared task, especially in the German and multilingual categories. They also indicate areas for improvement, particularly in English, where our model’s performance is lower than other teams. Overall, our team’s participation in the ArAIEval shared task demonstrated strong performance across multiple languages, securing top ranks in several categories and showcasing our model’s capabilities in multilingual data and subjective data evaluation.