Detecting rumors in social media using emotion based deep learning approach

Textual feature extraction

The proposed work focuses on textual data, acknowledging its primacy in conveying meaning and context within social media posts. Therefore, to extract meaningful features from the textual content of tweets, we leverage the power of transformer-based deep learning models, explicitly employing the well-established BERT (Bidirectional Encoder Representations from Transformers) architecture (Kenton & Toutanova, 2019).

The basic architecture of BERT is described in Fig. 4. The transformer architecture consists of encoder and decoder components. The encoder generates distinct continuous vector representations by processing the text of a tweet. The decoder then uses these vectorized embeddings to predict the desired outputs. The proposed work utilizes BERT because it is pre-trained on extensive data, around 3.3 billion words from Wikipedia and BooksCorpus. The model consists of a large number of encoder layers, a feed-forward network, and attention heads, as depicted in Fig. 4. The BERT language representation paradigm employs a deep architecture of stacked transformer encoder layers to generate contextualized word embeddings for each input token.

This study represents each tweet as a sequence of words denoted as $W{d}_i={\left\{w_i^x\right\}}_{x=1}^Z$, where Z represents the word count in the tweet. These words sequence $\left(W{d}_i\right)$, forms the textual modality $\left(T_i^t\right)$ for a tweet $\left(T_i\right)$. We employ the distilBertTokenizerFast model from the pre-trained transformer architecture to create the embedding representation. This tokenizer adds two special tokens, CLS (Class), at the beginning and SEP (Separator), at the end of each tweet’s sequence.

For a given tweet $\left(T_i\right)$, we provide the input $\left(T_i^t\right)$. This input is then processed to generate a sequence of integer-based tokens D_t shown in Eq. (1). (1)${\displaystyle D_t=distilBertTokenizerFast\left(T_i^t\right)}$ As depicted in Eq. (1), the distilBERT tokenizer aids in getting tokens in the form of integer sequences denoted as D_t. For any tweet $\left(T_i\right)$, the output is tokens and can be demonstrated as $D_t={\left\{d_i^x\right\}}_{x=1}^l$, where l denotes the length of sequence. In our work, we are taking a fixed sequence length of 60 for each tweet, i.e., l = 60. Padding will be done for the tweets having a length of less than 60. Further, the tokens will be passed from the Encoder model to get the embedding vector for each token, as shown in Eq. (2). (2)${\displaystyle E_m=BERT{\left\{d_i^x\right\}}_{x=1}^l}$ Equation (2) illustrates the generation of the embedding vector E_m when passed through the encoder model, i.e., BERT. Here, $E_m={\left\{e_x\right\}}_{x=1}^d$, where d is the dimension of size 768. The demonstrated process was textual feature extraction.

Sentiment feature extraction

The extraction of sentiment features from textual modality is substantiated to gather contextual insights into the tweet. To extract the sentiment tags, we implement a module from the natural language processing toolkit, namely TextBlob. The module is pre-trained on a variety of datasets.

TextBlob leverages a lexicon-based sentiment analysis approach and initially determines the intensity (positive or negative orientation) of individual words in a sentence. Lexicon-based approaches involve the use of a pre-built dictionary that categorizes words as positive or negative.

The generation of tags involves using a sentiment label analyzer as reflected in Eq. (3). (3)${\displaystyle S_v=SentimentIntensityAnalyzer\left(\right)}$ Furthermore, polarity scores are generated as shown in Eq. (4) followed by estimating tags from the calculated scores. (4)${\displaystyle S_d=S_v.polarity\text{\_}scores\left(T_i\right)}$ Equation (4) represents the use of the analyzed score in the generation of the polarity score, which will be converted to provide us with the final sentiment label. The algorithm below outlines the procedure for sentiment tag extraction and subsequent incorporation into the original textual features.

 
_______________________ 
Algorithm 1 Sentiment feature extraction module____________________________________ 
Input: ex : TextualModalityTi 
Output: Si: Sentiment Label 
function  Sentiment(ex) 
  1:  Sv = IntensityAnalyzer() 
  2:  Sd = Sv.polarity_scores(Ti) 
  3:  Si = polarity_to_text(Sd) 
  4:  return Si_______________________________________________________________________________________    

Emotion feature extraction

The novel framework utilizes the extraction of the emotion tags from the tweets. The emotion extraction module leverages the six emotion tags: joy, sadness, anger, fear, love, and surprise. We employ a RNN-based deep learning model, i.e., Bi-directional LSTMs, to associate each tweet with an emotion. Leveraging the ‘Emotions dataset for NLP’, we classify each tweet into one of six pre-defined emotion categories based on its textual content. Figure 5 accordingly depicts the process employed for emotion extraction from the textual data of tweets.

To extract emotion tags, we utilize the RNN-based module, namely Bi-directional LSTM, consisting of a cell, input gate, output gate, and forget gate (Van Houdt, Mosquera & Nápoles, 2020). Every gate follows a function for the calculation of output values corresponding to it. Equations (5), (6) and (7) represents the required functions. The module has recurrently connected blocks. The Eq. (5) describes the forward pass limiting the input in the RNN network where the current input is, say x^t and z^t−1 is the output of LSTM in the last iteration. (5)${\displaystyle y^{\left(t\right)}=\mathcal{F}\left(W_y{x}^t+R_y{z}^{t-1}+b_y\right)}$

The $\mathcal{F}$ in Eq. (5) usually is tanh whereas W_y and R_y are weights associated with x^t and z^t−1 respectively, while b_y represents the bias weight vector. The given function allows the calculation of the forward pass of the LSTM architecture. For the next connected block, the current input is combined with the previous layer’s output, as shown in Eq. (6) followed by the removal of information from the previous cell, i.e., g^(t), following the same procedure as input gate on current input, previous cell output and the state c^t−1. The τ is always sigmoid and p, W, R are weights at respective stages. (6)${\displaystyle i^{\left(t\right)}=\tau \left(W_i{x}^t+R_i{z}^{t-1}+p_i\odot c^{t-1}+b_i\right)}$ Equation (6) represents the function of the input gate. By combining the y^(t), i^(t) and g^(t) we can calculate the cell value as c^(t) = y^(t)⊙i^(t) + c^(t−1)⊙g^(t). This c^(t) corresponds to the function of the cell gate. Finally, the output of the recurrent model can be described as in Eq. (7). (7)${\displaystyle O^{\left(t\right)}=\tau \left(W_o{x}^t+R_o{z}^{t-1}+p_o\odot c^t+b_o\right)}$

Our emotion extraction module leverages two LSTM networks, one in the forward direction and the other in the backward direction, to capture the contextual insights of the tweet in order to generate the output label. Figure 5 accordingly depicts the process employed for emotion extraction from the textual modality. The process initiates with passing the raw text T (representing individual tweets) as input to the model. T, when tokenized, gives T₁, which provides us with an embedding vector by utilizing the GloVe model of the gensim library. GloVe stands for Global Vectors for word embedding. It is a pre-trained model trained on extensive text data; utilizing this, we get our embedding vectors for available textual modality (Pennington, Socher & Manning, 2014).

Furthermore, the embedding passes through bidirectional recurrent blocks of dimension 100, 200, 100 as depicted in lines 3, 5, 7 of the algorithm. Finally, the output of the last bidirectional LSTM layer is passed through a fully connected dense layer of dimension 6 followed by softmax activation function to get label E_i where E_i ∈ joy, sadness, surprise, love, anger, fear.

The algorithm discussed below shows flow the model follows.

 
__________________________________________________________________________________________________________ 
Algorithm 2 Emotion extraction module________________________________________________ 
Input: T: Tweet from Emotion dataset 
Output: Ei : Emotion label 
function  Emotion(T) 
  1:  T1 ← clean_text(T) 
  2:  T1 ← TokenizeEmbed(T) 
  3:  g1 ← Bi(LSTM(T1,100)) 
  4:  g1 ← dropout(0.2) 
  5:  g2 ← Bi(LSTM(g1,200)) 
  6:  g2 ← dropout(0.2) 
  7:  C ← Bi(LSTM(g2,100)) 
  8:  Ei ← Dense(6,activation = “softmax”) 
  9:  return Ei_________________________________________________________________________________________________________    

PHEME

The dataset is based on actual life incidents that happened around the world; the events are defined as hashtags, namely #charliehebdo, the incident of firing in France, and #ferguson, an incident of killing a black person in the USA. The dataset was formed consisting of a total of nine events. The tweets were taken from around 25,691 Twitter (X) users. This work utilizes the events mentioned above.

Twitter24

The novel dataset named “Twitter24“ has been curated from the real-time tweets extracted manually from the social media platform Twitter (X). The dataset consists of only textual modality. It consists of tweets from popular user accounts like “Narendra Modi”, “Virat Kohli“, focusing more on information circulating in India. The labels are assigned manually and the correctness is established by utilizing fact checking website i.e., “Boom Fact Check”. The purpose of this dataset is to validate the performance of SEMTEC model on real-time data.

As referred in Table 2, the “PHEME” dataset consists of a total of 62,445 tweets and “Twitter24” consists around 4,829 tweets which are distributed in two labels, i.e., rumor and non-rumor. Three mutually exclusive training, testing, and validation sets are created from the tweets with tweet share as 70%, 20%, and 10%, respectively.

For training our RNN-based deep learning module, “Emotion dataset for NLP” is utilized.

Table 3 illustrates an overview of the dataset. The dataset aggregates a total of 20,000 tweets, categorized into six different classes, namely joy, sadness, anger, fear, love, and surprise, with tweet counts as 6,761, 5,797, 2,709, 2,373, 1,641 and 719 respectively.

Software requirements

This section details the computational environment that facilitated the research and enabled the achievement of the presented results.

Table 4 illustrates all the software parameters used for setting up the running environment of the proposed SEMTEC method.

Hardware requirements

The following section details the hardware requirements used in this study to ensure the reproducibility of the presented results. We focus on the critical hardware components that significantly impact the performance of our experiments. Additionally, we acknowledge that similar configurations with comparable capabilities might achieve similar outcomes, aiming to broaden accessibility for researchers with varying resource constraints.

Table 5 demonstrates the specifications of the local system utilized in fulfillment of the proposed SEMTEC method.

FastText classifier

FastText Classifier (Joulin et al., 2016) represents text data as a bag of words and employs a linear classifier following training. This approach aligns with establishing simple machine-learning models as strong baselines for text classification tasks.

GAN-GRU

The GAN-GRU (Ma, Gao & Wong, 2019) method is based on Generative Adversarial Network(GAN). It employs a generator to introduce conflicting and uncertain perspectives into the original tweet thread, leading the discriminator to learn from more complicated examples.

TDRD

The TDRD (Topic Driven Rumor Detection) method extracts the post’s topic to derive the tweet’s label. Xu, Sheng & Wang (2020) first automatically perform topic classification on source microblogs, and then they successfully incorporate the predicted topic vector of the source microblogs into rumor detection.

BiGCN

BiGCN (Bi-directional Graph Convolutional Network) (Bian et al., 2020) method utilizes both propagation and dispersion for rumor detection. The model incorporates both features by operating from bottom-to-top and top-to-bottom propagation of rumors. The up-down GCN(UD-GCN) incorporates the propagation features, whereas bottom-up(BU-GCN) deals with the dispersion of rumor.

GCAN

The GCAN (Graph Aware Co-attention Networks) (Lu & Li, 2020), a neural network-based method, predicts whether the tweet is accurate and simultaneously generates explanations highlighting the evidence from suspicious retweeters and the concerning words they use.

GACL

GACL (Graph Adversial Contrastive Learning) (Sun et al., 2022) deals with poor generalization in conventional models, where the module of contrastive learning extracts similarities and differences among tweet threads. Furthermore, the AFT(Adversial Feature Transformation) module generates conflicting samples to extract event-invariant features. Table 6 compares features among the existing research and the proposed SEMTEC method.

Existing rumor detection methods primarily rely on classification approaches, focusing on features extracted from follow-up comments to the initial tweet as indicated in Table 6. However, these methods often neglect the potential value of the primary tweet itself for early rumor detection, particularly in real-time scenarios. This paper introduces SEMTEC, a novel approach that moves beyond classification and emphasizes the importance of the primary tweet. SEMTEC leverages a comprehensive feature set that incorporates functionalities employed in prior work and introduces additional features to enhance real-time detection accuracy.

Evaluation of emotion extraction methods

To optimize the work and increase the efficiency of SEMTEC, we focused on identifying the best method for emotion extraction from the textual modality. We experimented with RNN-based deep learning methods and encoder-based deep learning methods. Standard BERT is utilized as an encoder model for the comparison. It takes into account the ambiguous meaning of the textual modality that enhances its performance in NLP tasks (Kenton & Toutanova, 2019).

Figure 6 shows that the RNN-based methods best identify the emotion from the available textual modality. Bi-LSTMs are acknowledged for their capability to capture contextual information due to their ability to process sequences in both forward and backward directions. This characteristic allows the model to extract emotions while considering the surrounding context within each tweet. This led us to move forward with Bidirectional LSTM (RNN-based method) for emotion extraction.

The experiment compared the stated emotion extraction methods. A standard optimization algorithm, Adam, was employed alongside categorical cross-entropy loss and a softmax activation function. The training process was executed for approximately 35 epochs.

Effectiveness comparisions

To evaluate the efficacy of our proposed approach for rumor detection, we compare its performance to existing techniques. We evaluate the performance of various techniques based on established metrics like Precision, Recall, F1-score, and Accuracy. Leveraging the publicly available “PHEME“ dataset and the real-time dataset “Twitter24”, we illustrate the effectiveness of our model by comparing its results to those obtained using previously employed techniques.

Table 8 shows how significantly better our SEMTEC model performs on the datasets, as mentioned earlier, than the prior techniques. The performance of our SEMTEC model on different metric parameters, namely Precision, Recall, and F1-score, is 0.91, 0.92, and 0.92, respectively, on the PHEME dataset. In terms of accuracy, we achieve a surge of around 0.7 from the best existing method.

Figure 7 illustrates a comprehensive visualization of the variations in F1-score across different models. Our model achieves superior performance on PHEME due to its incorporation of emotion and sentiment features alongside a contextual analysis of textual modalities, as opposed to current techniques, which rely on the textual content of social media posts. Figure 8 illustrates the qualitative assessment of the proposed work. The SEMTEC surpasses the existing research, and this can be justified by the feature set and deep learning models utilized in this work.

Furthermore, we have compared our work with the existing classifier. The classifiers can be divided into machine learning-based classifiers like support vector machine (SVM), Random Forest(RF) and deep learning-based classifiers like Transformer and Bi-directional LSTMs.

As illustrated in Table 9, the SEMTEC method outperforms the standard classifiers with a significant difference. The findings justify the relationship between the semantic attributes and the veracity of the tweet, which aids in the classification task. Figure 9 visually illustrates the performances of standard classifiers on “PHEME” dataset. Our findings suggest that rumor detection extends beyond a simple classification task.

To validate the performance of our proposed SEMTEC method, we further experimented with the novel “Twitter24” dataset. The experimentation demonstrates that the proposed SEMTEC method surpasses the existing standard methods used for classification by around 2%. Table 10 illustrates the findings highlighting the superior performances.

Performance gain analysis

In this section, we analyze the effectiveness of our SEMTEC method with a combination of various features utilized in our work. The performance of our model is the outcome of integrating emotion features, sentiment features, and contextual analysis of text. Ajao, Bhowmik & Zargari (2019) also showed the interconnectedness of semantic features with detecting fake news. We conducted more experiments to justify that the proposed SEMTEC method establishes the relationship between semantic attributes and the veracity of the tweet. To justify the findings and the relationship between the semantic features, i.e., sentiment and emotion features, we conducted an ablation study demonstrating semantic features effect on the identification of rumor.

Figure 10 demonstrates our method’s performance by including various features. We discuss our proposed model’s performance by including emotion and sentiment tags along with the textual modality. The emotion feature directly conveys the tweet’s objective. This variant is mentioned as SEMTEC. This model performs better than the prior SEMTEC - (E), where only text and sentiment tags were used. SEMTEC - (E+S) illustrates the textual modality without any features. This work presents the results achieved in terms of accuracy. This enhancement can be attributed to the incorporation of emotion and sentiment tags as they facilitate a deeper understanding of the sentence semantics, which further prove significant in predicting the sentiment polarity of the post.

Analysis of curated dataset with emotion labels

In this section, we discuss the curated “EmoPHEME” dataset. Our proposed SEMTEC method utilizes emotion tags extracted using RNN based deep learning model, i.e., Bi-directional LSTM. This module, trained on the “Emotion dataset for NLP”, enables the generation of emotion labels for the “PHEME” dataset, capturing the emotional aspect of the tweets. The emotion extraction module facilitates exploring new informational dimensions within the transformed “PHEME” dataset named “EmoPHEME”. This enhanced dataset offers potential applications in training and testing models designed for emotion-related sentiment analysis tasks. Figure 11 visualizes the distribution of the emotion labels across the new “EmoPHEME” dataset. The labels, namely joy, anger, sadness, love, surprise, and fear, have percentage division as 29%, 20%, 27%, 5%, 1% and 18%, of the total tweets, respectively.

Pros and practical implementation of SEMTEC method

• The SEMTEC method reveals the contextual relationship hidden inside the root or main tweet. It considers the emotional aspect of the tweet, aiding in correctly providing a label to the tweet.

• The proposed method can be used in various social media platforms and publication industries for rumor identification and detection.

Cons or under considered aspect of SEMTEC

• In previous literature, the authors utilized follow-up comments as a propagation feature. Our proposed work mainly focused on the root or main tweet. The propagation feature can be utilized as an additional feature in the extension of this work.

• The real-time streaming of data involves processing data as soon as it gets pushed on any platform. The model pipeline should not hold any lagging(time delay) when processing this kind of data. Our SEMTEC method involves numerous modules, such as preprocessing and feature extraction units, before providing a label to the tweet. This might cause a delay in processing data in real time.