Employing Sentence Space Embedding for Classification of Data Stream from Fake News Domain

At the core of Natural Language Processing lies the challenge of converting natural language content into a numerical feature space that preserves the document’s semantic information. The bag-of-words method, a fundamental text feature extraction technique, calculates specific word occurrences within individual samples [21]. This approach forms the basis for more advanced methods such as bag-of-n-grams, which retain the contextual information of specific words and thereby a certain semantic association [11]. The tf-idf method [33] further enhances this vector-based approach by incorporating two key statistics: Term Frequency (tf), which measures word frequency within a document, and Inverse Document Frequency (idf), a logarithmic measure of word uniqueness within a corpus.

In pursuit of the need to reduce dimensionality and the feature vector notation in continuous space, the continuous bag-of-words approach was invented, which, along with the continuous skip-gram model, is more widely known as Word2Vec (W2V) [25]. This method allows the determination of a vector representation of a given length for each word in the corpus. Despite its innovation, this approach (i) fails to handle languages that are highly morphologically rich, and (ii) determines embeddings only based on local word relationships in the corpus. The answer to the first problem is the FastText approach [3], which, relying on the Word2Vec idea, performs additional word segmentation into character-n-grams, significantly enhancing the context during processing. In response to the second drawback, the Global Vectors (GloVe) [31] model was developed, which is also based on Word2Vec, but in addition, extends the model to include general statistics from the processed corpus using a global word-word co-occurrence matrix.

Among vectorization methods, Large Language Models (llms) are at the top-notch among the most commonly used approaches [13]. They use an artificial neural network structure called a transformer as their basis [37]. Their significant advantage is the employment of self-attention heads that powerfully extend the context, and this, in combination with the processing of massive linguistic resources, generates promising text representations [26]. One of the most popular large language models is Bidirectional Encoder Representations from Transformers (bert) [8], which is designed to pre-train deep bidirectional representations from unlabeled text. As a result, it can capture the dependencies present throughout the text. Additionally, there are no words but subword units called WordPieces at the foundation of the bert model.

However, despite the many advantages of large language models, they have a fair amount of complexity, affecting the time required to determine vectors. A MiniLM model [40] is an interesting approach where the authors train a reduced model using knowledge distillation from the bert, maintaining a quality similar to the original. The primary transformer idea draws inspiration from the sequence processing concept – the seq2seq model [36]. The expected input is a sequence of tokens forming a longer text. In contrast, a more intuitive solution would be to employ word embeddings in processing a single word into a vector space.

Access to massive volumes of data and increased processing power promotes the usage of deep learning techniques. They are the foundation of multimodal data processing, which is primarily reliant on computer vision tasks, where deep methods frequently outperform canonical approaches [29]. Numerous scientific articles confirm that convolutional networks are successfully utilized for image, video, natural language [12], and audio classification (e.g., in spectrogram form) [34] in both unimodal and multimodal settings. Deep networks facilitate transfer learning, enabling models to apply previously learned information to the task at hand [42].

Although the term multi-dimensional encoding is mainly used for tabular data, the Sentence Space proposed by Kim [15] can be considered as its equivalent for text corpora. Using sentence space, text data is transformed into an image in which each row contains embeddings of individual words for each text sample. This approach is clearly dependent on the configuration of the convolutional neural network architecture used, as noted and studied in their work by Zhang and Wallace [41], analyzing possible configurations of one-layer cnns. In turn, Le et al. [22] analyzed the effect of the depth of convolutional neural networks on sentence space. Lately, an extension of the original concept of text encoding to a two-dimensional discrete digital signal was proposed by Soni et al. [35] in the form of TextConvoNet. This approach extracts the n-gram features within a sentence and captures the n-gram features between sentences in the input text data, resulting in a three-dimensional representation.

Despite over three decades of research and numerous techniques now accessible, classifying drifting imbalanced data streams remains one of the important machine learning topics. Methods designed for this type of data can work in online manner, where each instance is analyzed individually, or on batches of data, where the stream is processed in non-interlacing windows. This work focuses on batch processing, which, because to the larger training set, may provide improved classification quality in the current concept, but is accompanied with a delayed reaction due to having to wait for the next data batch to be available [1].

Data imbalance is a prevalent problem in data streams, where the imbalance ratio can be static or dynamic [2]. Most real streams do not have a fixed imbalance ratio, and their properties might change over time [39]. As a result, data stream classification methods should achieve good classification quality regardless of class distribution, however most techniques built for imbalanced data streams produce unsatisfactory results when class sizes are similar. At the same time, algorithms designed with balanced data in mind, frequently have difficulty with the correct classification of data streams with skewed class distribution [6]. Methods designed for for dealing with imbalanced data are separated into two main groups: (i) data-level approaches and (ii) algorithm-level techniques [1]. The first group focuses on data preprocessing, canonically by employing oversampling or undersampling, to change its characteristics prior to classification attempt to alleviate the bias towards majority class, whereas algorithm-level approaches focus on modifying classification algorithms’ training phase.

The most prevalent methods for imbalanced data stream classification employ classifier ensembles coupled with data preprocessing techniques. By assuring diversity, constantly updating the classifier pool, and combining the available models, it is feasible to increase the generalization ability and allow for dynamic adaptation to changes in the stream’s characteristics [5]. Among the established algorithms, we can distinguish Learn++.CDS (Concept Drift with Smote) and Learn++.NIE (Nonstationary and Imbalanced Environments) by Ditzler and Polikar [9]. Learn++.CDS extends the Learn++.NIE (Non-Stationary Environments) algorithm by employing the smote in attempt to balance the number of samples in each class, while Learn++.NIE utilizes a penalty constraint to balance classification accuracy on all classes, while also employing a bagging-based sub-ensemble. Wang et al. proposed the Oversampling Online Bagging (oob) and Undersampling Online Bagging (uob) algorithms dedicated for online data stream processing, extending Online Bagging by altering the Poisson distribution $\lambda$ parameter according to the current imbalance ratio [38]. Cano and Krawczyk developed Kappa Updated Ensemble (kue) [6], which combines batch-based and online processing on feature subspaces. kue uses the Kappa statistic to dynamically weight and select base classifiers. The same authors introduced also the Robust Online Self-Adjusting Ensemble (rose) for non-stationary data stream classification [7]. This method trains online learners based on data views, ensuring a diverse classifier pool. It also employs drift detectors to respond quickly to changes in data distribution and proposes effective strategies for dealing with data imbalance. Wozniak et al. used built-in mechanisms (e.g., weighting and aging of classification models) to establish a self-updating classifier pool that can adjust its lineup in response to changes in imbalance ratio and concept drift. Klikowski and Woźniak trained one-class classifiers using clustered data [16], while Zyblewski et al. proposed to combine Dynamic Classifier Selection with data preprocessing techniques for imbalanced data stream classification [43].

Data All of the research was conducted using Fakeddit’s multimodal dataset, which presents a real-life fake news classification task broken down into two, three or six classes [28]. This dataset consists of more than one million posts on 22 different subreddits of the social networking platform Reddit and includes text and image modalities, supplemented by metadata about the posts and their authors. For the purposes of this study, a single binary data stream was prepared, in which consecutive texts were sorted accordingly to their creation timestamp. Of the entire dataset, 682,996 multimodal samples were used (the rest have only one modality). The decision to limit to multimodal samples only is linked to the facilitation of extending the presented research to include the image modality. The data stream was divided into 2731 data chunks, containing 250 samples each. Relevant to the research course is the fact that the resulting data stream is characterized by a dynamic imbalance ratio, which changes in successive batches to the point where, at about 3/4 of the length of the stream, a minority class transitions into a majority class. The exact changes in the prior probability of class membership are shown in Fig. 3.

Experimental protocol & reproducibility All experiments were carried out using the Test-Then-Train protocol to guarantee a robust experimental evaluation, were implemented in Python and can be replicated using the publicly available GitHub repository¹¹1https://github.com/w4k2/sentence-space-stream. Implementation of state-of-the-art algorithms were based on stream-learn [18], scikit-multiflow [27], and PyTorch [30] libraries. The classification quality evaluation of the algorithms was based on the standard metrics used in the task of imbalanced data classification, i.e. balanced accuracy score (bac), recall, specificity, precision, $F_{1}$ score, $Gmean$, and $Gmean_{s}$.

As can be seen in Figure 4, across all data chunks, the results of the first three methods GloVe, MiniLM, pre-trained Word2Vec are similar, with the only deviating method being partial-fit Word2Vec trained in each chunk. This discrepancy can be explained by the dictionary’s limitations built on the training data, consisting of article titles, primarily short texts. In contrast, the MiniLM model and Word2Vec pre-trained (word2vec-google-news-300) based on the Google News dataset, as well as GloVe (glove-wiki-gigaword-300), which means they were all built on larger volumes of data compared to the own trained Word2Vec.

Following this experiment, it was decided to use the GloVe vectors for further research, which is a newer method than Word2Vec that processes additional global information and, at the same time – unlike MiniLM, dedicated to sequential processing – was designed to process single words. In addition, the MiniLM model requires more computational complexity when determining the representation vector for a given word, which for Word2Vec and Glove methods is limited only to reading values from the vector array. With equal effectiveness, the exclusion of MiniLM is fully justified.

The preliminary part of main comparative experiment has begun from analysis of feature space reduction influence. It was conducted by Principal Component Analysis (pca) – projecting the original embedding space down into 100 features. As it can be observed in Figure 5, influence of this simple reduction is mostly cosmetic, always laying in one percent margin of a difference, so it is justified to reduce problem representation for canonical models, since representation gets smaller while change in quality is negligible.

The main comparison of recognition efficiency, taking into account the deep strategy already proposed in this work (sss), is presented in Figure 6, divided into four empirical analyzes according to the extraction strategies used for canonical methods. In the case of each approach, the inglorious laggard turns out to be the nie method, which maintains its model at the level of a random classifier for a very long time, and in two cases (Glove and MiniLM) raises it only slightly in the final phase of the stream, after the prior concept drift shown in Fig. 3.

The weakest method in the main rate (excluding nie) turns out to be Learning++CDS, which for all extractors except MiniLM clearly leans towards randomness. All analyzed reference solutions appear to be very sensitive to concept drift occurring around the chunk 2,000. The ranking of kue, rose and raw Hoeffding Tree methods depends on the extraction strategy used, but the overall distribution of their effectiveness is rather similar, with a slight advantage for rose.

sss as a deep learning strategy shows a noticeable advantage over all canonical solutions throughout the entire data stream, being the only one in the competition to maintain an average recognition efficiency of 80 percent of balanced accuracy score. This observation is supported by the extended analysis of metrics in the form of radar (Figure 7), where an advantage of sss can be observed in each of the simple and aggregate metrics used. The outlier specificity result for the nie method results solely from its complete inability to learn in the analyzed problem environment, which for the dominant majority of processing time induces decisions towards one of the problem classes, without a generalized relation to given bias.

Among the analyzed extraction methods, MiniLM comes minimaly to the fore, as it was the only one that allowed the nie strategy to noticeably rise from the random classifier level in the final part of the stream, and the cds to compete with the rest of the competition.

The results of the third experiment – showing the time complexity for extracting, training, and testing the algorithm – are shown in Figure 8. As we can see, in the case of preprocessing, all reference methods show uniform computation time (lines overlap), and only the proposed approach deviates from this tendency and performs extraction faster. Additionally, it should be noted that this happens despite the time measurement considering the transition to image representation. The method owes it using the GloVe technique for sss, which in Experiment 1 had the best results for set space encoding. The reference methods, however, use the MiniLM transformer in accordance with the outcome of Experiment 2.

More variability can be observed in the training and testing processes – in both, a single Hoeffding Tree processes the fastest. In turn, training in a single epoch is the slowest for rose, and prediction – for nie. It is also reflected in the accumulated time graph, which indicates that the only method ahead of sss in the entire processing is the Hoeffding Tree. Still, it should be borne in mind that a single classifier is considered here, not an ensemble.

Therefore, the proposed method is ahead of classifiers ensembles used for data streams even though it is based on convolutional networks and requires transformation from text into an image. At the same time, despite the lowest processing time compared to state-of-the-art ensemble algorithms, sss offers the highest generalization ability. All this means that sss can be successfully used in real-life batch-based data stream classification tasks.