dzStance at StanceEval2024: Arabic Stance Detection based on Sentence Transformers

Stance detection is a pivotal task in Natural Language Processing (NLP) that involves determining the position or attitude expressed in a text regarding a specific topic or entity. This task is crucial for a variety of applications, including sentiment analysis, opinion mining, and social media monitoring, where understanding public sentiment and opinion is essential Alturayeif et al. (2022). With the exponential growth of user-generated content on social media and online news platforms, there is a pressing need to develop sophisticated tools that can analyze and interpret the myriad of perspectives present in these texts.

The Mawqif 2022 shared task, an initiative focused on Arabic stance detection, addresses this need by challenging participants to detect stances towards three contemporary topics: COVID-19 vaccine, digital transformation, and women empowerment. This task is significant not only because it addresses critical societal issues but also because it highlights the complexities of processing Arabic text, which is characterized by its rich morphology, diverse dialects, and intricate syntax Alturayeif et al. (2022).

Traditional methods for stance detection have largely relied on feature extraction techniques such as Term Frequency-Inverse Document Frequency (TF-IDF). These methods transform textual data into a numerical format, allowing machine learning models to process and analyze the text. Our initial experiments leverage TF-IDF features due to their simplicity and proven effectiveness in various text classification tasks Abbas et al. (2019); Lichouri and Abbas (2020).

Despite their utility, TF-IDF-based methods have limitations, particularly in capturing the deeper semantic relationships and contextual nuances within text. To address these limitations, recent research has increasingly focused on deep learning techniques. Long Short-Term Memory (LSTM) networks have demonstrated their ability to handle sequential data and capture dependencies in text, which are crucial for understanding stances Alturayeif et al. (2023); Lai et al. (2020).

The advent of transformer-based models, such as BERT and Sentence Transformers, has revolutionized the field of NLP by offering robust methods to capture semantic and contextual information in text. These models utilize self-attention mechanisms to understand the relationships within text, making them particularly effective for nuanced tasks like stance detection Reimers and Gurevych (2019); AlDayel and Magdy (2021). Transformers have set new benchmarks in various NLP tasks by leveraging their ability to generate dense, context-aware representations of text, facilitating a deeper understanding of the underlying meaning and intent.

Several studies have explored the use of BERT and other transformers for stance detection, demonstrating their superiority over traditional methods in capturing subtle and complex stances in text. For example, Alshahrani et al. AlDayel and Magdy (2021) showed that BERT-based models outperform traditional approaches in detecting stances in English social media texts. However, these models often require extensive computational resources and fine-tuning, posing challenges for their application in resource-limited settings Alturayeif et al. (2023).

Building upon these advancements, our study explores a comparative analysis between traditional feature extraction methods and modern deep learning approaches for Arabic stance detection. In our experiments, we employ both TF-IDF and Sentence Transformers to detect stances towards the selected topics. Our participation in the Mawqif 2022 shared task allowed us to evaluate these methodologies rigorously, where our team, dzStance, achieved competitive results across the three topics, demonstrating the effectiveness of our approaches.

In the following sections, we will delve deeper into our methodology and findings. Section 2 provides an overview of the Mawqif dataset used in our study. Section 3 describes our approach to stance detection, including the feature extraction techniques and model architectures we employed. Section 4 presents our experimental results and discusses their implications. Finally, Section 5 concludes the paper by summarizing the key insights and contributions of our work.

The Mawqif dataset Alturayeif et al. (2022), utilized in the StanceEval 2024 shared task Alturayeif et al. (2024), serves as a crucial resource for advancing natural language processing (NLP) in the domain of stance detection. This dataset comprises over 4,000 annotated text samples that encapsulate diverse stances—favorable, opposing, or neutral—on pertinent topics such as COVID-19 vaccine, digital transformation, and women empowerment.

The significance of the Mawqif dataset lies in its ability to provide a comprehensive view of how different opinions and attitudes are expressed in Arabic text. This makes it invaluable for researchers who aim to evaluate and enhance stance detection models. By leveraging such a dataset, one can explore and refine models to better understand and process nuanced stances within varied contexts.

Table 1 offers a detailed breakdown of the dataset, illustrating the distribution of tweets across the specified topics. The dataset comprises a total of 3,502 tweets, with the distribution as follows:

A closer look at the data statistics reveals notable class imbalances. For instance, the COVID-19 vaccine category includes nearly equal proportions of favorable (43.53%) and opposing (43.48%) tweets, with a smaller fraction being neutral (12.85%). In contrast, the digital transformation topic shows a predominance of favorable stances (76.77%), with fewer opposing (12.41%) and neutral (1.92%) tweets. Similarly, the women empowerment category also leans heavily towards favorable stances (63.95%), followed by opposing (31.18%) and neutral (4.96%) tweets.

Such imbalances can pose significant challenges for model training and evaluation. Models trained on datasets with skewed class distributions may become biased towards the majority classes, leading to suboptimal performance on minority classes. Therefore, addressing these imbalances is critical to ensure the development of robust and fair models. Techniques like data resampling, class weighting, and the use of advanced algorithms capable of handling imbalance are essential strategies to mitigate these effects and enhance overall model performance.

By understanding and leveraging the characteristics of the Mawqif dataset, researchers can effectively tackle the complexities of stance detection, contributing to the broader field of Arabic NLP and enabling more accurate and nuanced analysis of opinions and attitudes expressed in text.

In our proposed system ¹¹1https://github.com/licvol/dzStanceEval_2024, we explore two distinct methodologies for feature extraction: a weighted union of TF-IDF features Lichouri et al. (2023) and Sentence Transformers. These techniques offer complementary advantages, leveraging the strengths of traditional feature representation and cutting-edge deep learning architectures.

Experiment 1

: Traditional Machine Learning with TF-IDF Features In our first experiment, we focused on extracting features using the Term Frequency-Inverse Document Frequency (TF-IDF) approach, which is widely used in text classification tasks. We utilized scikit-learn’s FeatureUnion module to combine different TF-IDF features, capturing both character-level and word-level information Lichouri et al. (2021).

• •

  N-gram Range: We experimented with various n-gram ranges to understand their impact on model performance. The ranges included:

  • -

    (1,1): Unigrams

  • -

    (1,2): Unigrams and bigrams

  • -

    (1,3): Up to trigrams

  • -

    (1,4): Up to 4-grams

  • -

    (1,5): Up to 5-grams

  • -

    (1,6): Up to 6-grams

  • -

    (1,7): Up to 7-grams

  • -

    (1,8): Up to 8-grams

  • -

    (1,9): Up to 9-grams

  • -

    (1,10): Up to 10-grams

• •

  Weighting: To further enhance the TF-IDF features, we incorporated weighting schemes. The weights were varied from 0.1 to 1.0 in steps of 0.1 to determine the optimal balance for capturing the intricacies of Arabic text. The best-performing weight, denoted as tw1, was identified and applied consistently across subsequent experiments (see Table 2).

• •

  Classifier: For classification, we employed the Linear Support Vector Classifier (LSVC) with a regularization parameter set to $C=4$. This choice was based on its ability to handle high-dimensional feature spaces effectively, which is crucial when dealing with the extensive n-gram features produced by TF-IDF.

This combination of weighted TF-IDF features and LSVC forms the baseline for our stance detection system, aiming to capture both the surface-level and deeper linguistic patterns in Arabic text.

Experiment 2

: Leveraging Pre-trained Language Models (PLMs) In the second experiment, we explored the use of advanced pre-trained language models (PLMs) to enhance stance detection capabilities further. These models are pre-trained on vast amounts of text data and are adept at generating rich semantic representations of words and sentences.

• •

  Sentence Embeddings: We utilized Sentence Transformers, specifically the xlm-r-bert-base-nli-stsb-mean-tokens model, which excels at producing dense vector embeddings that encapsulate the overall meaning of sentences. These embeddings were used as input features for the classification task.

• •

  Classifier: For classification, we opted for a Logistic Regression (LR) model configured with the following hyperparameters:

  • -

    max_iter=1000: This parameter sets the maximum number of iterations for the solver to converge.

  • -

    multi_class=’multinomial’: This setting enables the classifier to handle multiple classes simultaneously, which is crucial for stance detection where multiple stance labels exist.

  • -

    solver=’lbfgs’: The solver used for optimization, chosen for its efficiency in handling multiclass logistic regression problems.

  This approach leverages the powerful representations learned by Sentence Transformers, which are adept at capturing semantic nuances and contextual relationships within the text. By integrating these embeddings with a logistic regression classifier, we aim to improve the model’s ability to discern subtle stance indicators in Arabic text.

Combining these methodologies, our system aims to balance the strengths of traditional feature extraction with the advanced capabilities of modern pre-trained models. This hybrid approach is designed to address the linguistic complexity and variability of Arabic, providing robust stance detection across different contexts.

This section evaluates the performance of our stance detection system on the test set using the F1-score as the primary metric. We conducted two sets of experiments to compare the effectiveness of different feature extraction techniques and model configurations. Following these, we analyze our competitive performance in the stance detection challenge.

In this paper, we introduced dzStance, our solution to the StanceEval 2024 shared task on Arabic stance detection. Leveraging Sentence Transformers in conjunction with Logistic Regression, our approach achieved competitive results with an overall average F1-score of 71.77%. This performance positioned us 13th among all participating teams, highlighting the effectiveness of advanced embedding techniques and robust classification algorithms in handling the complexities of Arabic stance detection. The success of our approach underscores the significance of pre-trained models like Sentence Transformers for capturing nuanced semantic relationships within Arabic text across diverse topics.

Looking ahead, further investigation into why the Logistic Regression model outperformed traditional methods such as Linear Support Vector Classification (LSVC) could yield insights through deeper hyperparameter tuning and broader evaluation on varied datasets. Additionally, exploring hybrid approaches that integrate TF-IDF features with advanced embedding models may offer enhanced model robustness and accuracy in Arabic natural language processing tasks. By making our code and methodologies openly accessible, we aim to foster reproducibility and encourage ongoing advancements in Arabic stance detection research, paving the way for more sophisticated and effective models in the future.