Empirical Evaluation of Public HateSpeech Datasets

Hatespeech is gaining increasing attention from industry, government organisations and academia. The proliferation of information published on social media platforms provides the means to create datasets for processing, analysing, detecting and classifying hatespeech content. Variations in the available datasets (e.g, in size, topic, domain, language, hatespeech categories, platform, content type, etc.) could be beneficial. However, it can make it challenging for researchers to determine which dataset is suitable for training machine learning algorithms for hatespeech classification. For example, a dataset based only on Twitter content may, or may not, be suitable for producing a generalisable machine learning model that perform well on non-twitter data, such as YouTube comments.

The aim of this study is to evaluate multiple publicly available dataset to assess their suitability for training and testing deep learning algorithm for hatespeech classification. We have selected ten datasets from hatespeechdata.com website, which is a widely used platform for hosting hatespeech dataset. Our goal in evaluating multiple dataset is to examine two application aspects of each dataset: (i) to examine the suitability of a dataset in testing the performance of a deep learning based system for hatespeech classification, and (ii) to examine the suitability of a dataset in producing generalisable deep learning model by training a deep learning algorithm on a dataset but testing it on other dataset with different domain, text, genre, and size. The selected dataset names, platforms where the data are collected from, and the publication years of the dataset are presented in Table 1. We chose the evaluated datasets based on several characteristics: different platforms, dataset size, content type, length of individual text entry, and publication time. Examining dataset with content extracted from various platforms helps us to evaluate the generalisability of deep learning algorithms. Similarly, the different years were chosen to ensure that the evaluation would be generalisable, as some hateful terms might be more popular in specific years. We selected datasets with different sizes because machine learning algorithm performance is usually dependent on dataset size.

The labelled content in each selected dataset varies from one dataset to another. The proposed dataset by [13], has various categories of abuse (e.g., targeted Identity, affiliation, and person), and counter speech. The published dataset by [5] has three types of content (classes) i.e., Hatespeech, Offensive language and Neither. The datasets, from [14] and [16], have only two classes: Hate/Not Hate and Offensive/Vulgar. Kennedy et al.’s dataset [2] content is focused on particular categories (such as religions, three hate classes for races , and a hate score to indicate the hate level). The dataset from [3] contains text and images. They are labelled as “Hate” or “Not-Hate”.“Hate” content is further divided into five classes of different types of hate. Waseem and Havoy’s dataset [9], has three labels: “Sexism”, “Racism”, and “None”.

The content of the available dataset has been labelled with different types of hatespeech. Therefore, there are inconsistent labels between the dataset. In order to design and evaluate a supervised deep learning hatespeech classifier trained on the available dataset for binary hatespeech classification, we have converted the various labels in the dataset into either “Hate” or “Not-Hate” labels. Therefore, each dataset has one of two classes (“Hate” or “Not-Hate”). This approach enables us to make the labels in all the selected datasets uniform.

The available datasets have multiple types (classes) of hatespeech content. Each dataset contains different types of hatespeech, which results in inconsistent hatespeech labels between the datasets. One of the main reasons that published hatespeech datasets have different labels for hatespeech content is because there is no uniform consensus in the research community on the different classes of hatespeech, which is a challenge that requires further effort from the research community to address. Since there is no consensus in the research community on the different types of hatespeech, authors of the published datasets have labelled their data with different classes of hate. Some datasets contain fine-grained categories of hate, where the victims are targeted based on their race; religion; sexuality; ethnicity; gender etc. [2, 13]. Other datasets contain broad categories such as “hateful”, “abusive” or “neutral” [3, 16, 9]. Table 2 presents a summary of some of the categories of hatespeech in each dataset. Some datasets contain very few and general hatespeech content (such as “hate” or “offensive”, as in the dataset published by [14] and  [5]) whereas other datasets have many and fine-grained hatespeech types such as the dataset published by [2] which includes various subtypes of hatespeech based on gender or religion.

Prior to exploring and analysing the content of the selected datasets for this study, we have binarised the labels. The objective is to ensure all the datasets contain consistent labels (“Hate” or “Not-Hate”). The label binarization process is performed as follows:

• •

  Merge fine-grained labels of hatespeech to broad labels. If the label of content indicates any type of hatespeech, then we convert it to a broad label “Hate”. If the label indicates the content is “neutral” or “not hateful”, then we convert to “Not-hate”.

• •

  Drop ambiguous labels by discarding any content in a dataset where the content has an ambiguous label, such as “abusive”, because such content may not be considered hateful.

• •

  We convert content that is labelled as “neutral”, “not-hate”, or “not-abusive” to the“Not-hate” label.

Table 3 presents some statistical information on each dataset after we have binarised the labels as “Hate” or “Not-Hate”. The table contains the followings: sample size for each dataset based on the content split between “Hate” and “Not-Hate”, total unique word count for “Hate” and “Not-Hate” content, the mean and median, the min/max and the variance and standard deviation.

The label binarization process provides us with a dataset containing consistent labels of “Hate” or “Not-Hate”, which we can use to evaluate their suitability for training and testing a deep learning algorithm for binary hatespeech classification. We evaluate a baseline deep learning system on each dataset to assess its performance in two tasks: (i) performing binary classification of hatespeech (i.e., classifying text as either “Hate” or “Not-hate”), and (ii) performing transfer learning classification to test the generalisation of the system where we train the system on one dataset and test it on multiple other dataset.

One of the major issues with all the public datasets that negatively impact the performance of machine learning algorithms is the imbalance in the features. The sample size for different categories of text is often uneven, with some categories having more content than others. Most of the datasets appear to have more text related to “Not-Hate” categories than “Hate”. To address the data imbalance problem, we balance the sample size of “Hate” and “Not-Hate” content for each dataset before we use them to design and evaluate a baseline deep learning system.

The available datasets are imbalanced, as they contain unequal sample size for different text categories. As we discussed in Section 3.2, we have binarised the labelled content in each dataset so that they contain only “Hate” and “Not-Hate”.

As it can be seen in Table 4, the differences in the sample size for different labelled content is large in all the datasets. Such differences in sample sizes negatively affect the training of machine learning algorithms (including deep learning algorithms), as the algorithms become biased towards the majority sample. To balance the sample size of “Hate” and “Not-Hate” content in each dataset, we apply under-sampling methods, reducing the size of the majority sample to match the size of the minority one. The columns in Table 4 show the balanced sample size of “Hate”/“Not-Hate” content for each dataset.

To gain a better understanding of the nature and prevalence of hatespeech, this study utilised a quantitative approach to analyse the frequency of hate terms in both hateful and non-hateful speech. Our analysis began with collecting, 1523 frequently used hate terms from Hatebase²²2Available at: https://hatebase.org/, a publicly available database of hatespeech terms. We then conducted an analysis of the usage frequency of these terms in both types of speech. To ensure the comparability of the two counts, we utilised the balanced datasets.

We conducted a T-test for the means of the counted hate terms frequency using python scientific computing package [20]. The T-test generated two matrices for the comparison, the $t$-value and the $p$-value, providing a quantitative measure of the significance of differences in the usage of hate terms between hateful and non-hateful speech. This allowed us to compare the usage of the hate terms in both types of speech within each dataset, and to draw conclusions about the prevalence and nature of hatespeech in our datasets. Fig 1 presents a block diagram of the used T-test procedure.

Text preprocessing: cleaning and normalization. Since the content of the published dataset is collected from different online platforms (e.g., Twitter, Facebook, YouTube comments, etc.), they have different features such as structure, topic, user writing style, etc. We stored each dataset in comma separated value files (CSV) with two columns (text and label). The text column contains the text content and the label column contains one of two values, “hate” or “not-hate”. We have performed the following transformations on the text before training and evaluating our model:

• •

  Lower casing. We convert all the text to lower case English characters.

• •

  Removing non-English text. We remove content that is not part of the English alphabet.

• •

  Normalising emojis by replacing with token “$<$EMOJ$>$”.

• •

  Normalising tag. We transform all hashtags to the token “$<$HASHTAG$>$” and all usernames to the token “@USER”.

• •

  Removing duplication. We remove sequentially duplicated items such as words, spaces, characters etc.

• •

  Removing punctuation. We remove all the English and non-English punctuation.

• •

  Removing stop words. We remove all stop words such as ‘a’, ‘the’, ‘of’ etc.

• •

  Normalizing URL. We transform all hyperlinks and website address to the token “$<$URL$>$”.

• •

  Normalising HTML elements. We convert all named and numeric character references from HTMLs such as “>” and “&#amps;” in the text to their corresponding Unicode characters “$<$” and “&”, respectively.

• •

  Removing new line in text. We remove all new line in each text to create a single line text.

Deep learning model implementation. We have implemented a baseline deep learning text classification system. We have trained and tested the system on ten publicly available datasets.

Our model is based on a widely utilised deep learning algorithms for text classification. We have also utilised a general-purpose language model that has contributed to many natural language processing tasks.

After preprocessing and normalizing the text in each dataset 3.5) we have tokenised the input data (textual information) using Bidirectional Encoder Representations from Transformers model (BERT) [21], which is an essential step for training any deep learning algorithms. The BERT model is a multilingual language model trained on a very large text data. BERT model is a multi-layer bidirectional Transformer, which is a deep learning model used in several Natural Language Processing tasks  [22].

Next, the vector representation of the data is processed by a dropout layer with a dropout rate of 0.3. This step involves randomly excluding 30% of the training data during the training phase of the model in order to prevent the algorithm from memorizing the data pattern from the dataset, as is usually referred to as overfitting.

The output from the dropout later is used as input to a single deep learning dense layer of neural network. We optimised the model learning capacity using Adam optimizer with learning rate of $2e-5$. During the training phase, we computed the training error rate using a Binary Cross Entropy Loss function. Fig 2 shows the components of our model architecture.

BERT assigns special tokens to sentences, the “CLS”, “SEP” and “PAD” token. The “CLS” token is to indicate for the model to identify the start of a sentence. The “SEP” token to indicate the end of a sentence for the model. Therefore, using “CLS” and “SEP” informs the model of the start and end of a sentence. We use the “PAD” token is used to assign extra empty token(s) to sentences in order for all input sentences to have equal token length. For example, if we restrict the model to learn from sentences of 50 tokens long, and a sentence contains 30 tokens we would be appended to it to 20 empty tokens (“PAD”) to make it a 50 tokens long sentence. We set the maximum length of sentences to 128 tokens. We add padding token “PAD” to sentences with less than 128 tokens in order to reach 128 tokens. Any sentence over 128 tokens long will be truncated. Moreover, BERT employs attention mask where tokens that represent words are masked with the value of 1 and tokens that represent nothing (e.g., “PAD”) are masked with the value of 0. This masking mechanism allows BERT to determine what token(s) to retain or discard during the learning phase of the model. Each sentence, after being separated into individual words, will be encoded into vectors. We use the bert-base-cased pre-trained version for our model. BERT has been trained on 110 million parameters. The main advantage of using a pre-trained model (such as BERT in our study) is the significant reduction in training time.

Deep learning model Evaluation. We follow a standard approach for evaluating a machine learning algorithm for a supervised binary classification task. The evaluation metrics used in this study are based on three standard measures for classification task: Recall, Precision and Weighted F1-score.

We empirically evaluated ten different datasets using a baseline classifier. For each dataset evaluation, we divided the dataset into two parts: a training set and a test set, ensuring that the content of the two sets was distinct. We allocated 80% of each dataset to the training set for training the baseline classifier and 20% to the test set for evaluating the classifier’s performance. within the training set, we further partitioned the data by allocating 90% for training the deep learning architecture and reserving 10% for validating the training accuracy during the training stage.

We conducted two types of evaluations on each dataset:

• •

  Mono-dataset evaluation. In the model evaluation process, we trained and tested our baseline classifier, individually on each dataset, to classify their content as either “Hate” or “Not-Hate”. The aim of this experiment is to assess the suitability of each dataset for binary classification of hatespeech content.

• •

  Generalised learning evaluation. In this evaluation, we used the baseline classifier that we used for the mono-dataset evaluation, but we tested it differently. In this evaluation, we have conducted multiple experiments on the baseline classifier. In each experiment, we trained the model on one dataset and tested it on nine other datasets. For this experiment, our approach is: given dataset $d$ is a member of a set of dataset $D^{i,...,n}$, we trained our model on $d^{i}$ and tested it on all the dataset in $D$ except $d^{i}$. We trained the baseline classifier on the train set of dataset $d^{i}$ and tested it on the test set of the dataset $D^{j...n}$, where $d^{j}$ is the dataset in $D$ that is not the same as the test set of the dataset $d^{i}$, as demonstrated in Algorithm1. This approach is similar to the transfer learning approach, however, the classifiers here are not fine tunned on the test sets.

  This evaluation method allows us to assess the dataset’s suitability for training a deep learning model that can be effectively generalised to other datasets, which may exhibit different features compared to the one used for training. These variations may include differences in data content published on diverse online platforms, encompassing variations in users’ writing styles, content topics, dataset sizes, and other relevant characteristics. We utilise this evaluation to determine each dataset’s suitability for producing a deep learning model capable of generalising to classifying “Hate” and “Not-Hate” content published on different online platforms.

We have conducted multiple empirical evaluations of our baseline classifier, which we described in section 3.5, on ten publicly available datasets. In this section, we report the empirical evaluation outcomes of the effectiveness of each dataset for training and testing a baseline deep learning classifier to examine the suitability of different public dataset for hatespeech classification.

For each dataset evaluation, we conducted two experiments: (i) we evaluated the suitability of each dataset for training and testing a baseline classifier for the binary classification of hatespeech content. The training and testing samples are from the same dataset. Thus, we refer to this experiment as “mono-dataset experiment”, and we refer to the baseline classifier in this experiment as “mono-dataset classifier”. (ii) The second experiment involved testing the suitability of each dataset to produce a generalised baseline classifier, which has the same architecture as the mono-dataset classifier but trained and tested in a more generalisable approach. We trained the baseline classifier on a dataset and tested it on nine other datasets, excluding the dataset that we used for training the classifier. We refer to this experiment as “generalised learning experiment”, and we refer to the baseline classifier as “generalised classifier”.

For each experiment–due to space limitation– we report the system performance using weighted F1-score, which is based on the harmonic mean of recall and precision.

Table 5 presents the performance of the mono-dataset classifier. Out of the ten selected dataset,the model performs best when trained on the dataset published by Davidson et al. [5] achieving a weighted F1-score of 0.930. The second-best performance is based on the Suryawanshi et al. [8] dataset with a weighted F1-score of 0.902, which is slightly behind Davidson et al. [5]’s. The classifier performed worse when trained and tested on the dataset published by [3], producing a weighted F1-score of 0.697.

The classifier achieved moderate performance (between 0.81 and 0.88 of weighted F1-score) when evaluated on the following datasets: Qian et al. [14] (0.817), Kennedy et al. [2] (0.841), Waseem and Havoy [9] (0.879) and Salminen et al. [6] (0.884). Furthermore, the classifier produced a weighted F1-score between 0.71 and 0.78 when trained and tested on the following datasets: Vidgen et al. [13] (0.740), Gibert et al. [15] (0.777) and Vidgen et al. A [16] (0.789).

Table 6, shows the ranking of different datasets in ascending order based on the classifier’s performance, measured by weighted F1-score.

The dataset from Gomez et al. [3] is the least effective for training and testing a baseline neural network classifier, despite being a large dataset, which, theoretically, should be beneficial for deep neural network algorithm training. Davidson et al. [5]’s dataset is ranked first for training a baseline classifier, with Suryawanshi et al. [8]’s dataset (ranked second) narrowly behind. Several other datasets performed moderately: Salminen et al. [6] (0.884), Waseem and Havoy [9] (0.879), Kennedy et al. [2] (0.840) and Qian et al. (0.816). The dataset from Vidgen et al. A [16], Gibert et al. [15], and Vidgen et al. [13] produced weight F1-score of 0.789, 0.777 and 0.740, respectively.

In comparison with the reported hatespeech classifiers proposed by some of the authors of the published datasets, Table 7 presents the recall, precision, and f1-score of our baseline system and other published binary hatespeech classifiers. The table demonstrates that in some cases, our baseline system trained on the binary labels of the selected datasets outperforms the binary systems proposed by the dataset authors.

In this experiment, we evaluated our baseline classifier by training it on one dataset and testing it on nine other datasets, excluding the dataset used for training. We repeated the experiment for all the datasets. The results from this experiment provide a clear indication of the suitability of each dataset for producing a classifier that can be generalised to unseen data. We refer to the baseline classifier in this experiment as “generalised classifier”³³3Note: this type of evaluation could also be referred to as “transfer learning”.

Table 8 shows the performance of the classifier during this experiment. The first column contains the dataset used for training the classifier. The other columns (column 1 to 10) contain the datasets used for testing the performance of the classifier. An asterisk ‘*’ in each column indicates the classifier is not tested on the dataset specified in that column. For example, the ‘*’ in the first row of the second column indicates the classifier is trained on Davidson et al.’s [5] dataset but not tested on that dataset. This is due to the nature of transfer learning method. Thus, there is one ‘*’ in each row. The numbers in the rows represent the performance of each classifier when tested on all the datasets except the one that is used for training.

The database published by [14] is one of the most suitable for producing a generalised deep learning baseline classifier that performs well on multiple hatespeech datasets. When the baseline classifier was trained on Qian et al. [14]’s dataset, it performed well on four out of nine datasets (namely, Davidson et al. [5], Waseem and Havoy [9], Vidgen et al. A [16], and Salmenin et al. [6]). The classifier’s performance on this dataset is shown in bold in the 10th row of Table 8.

Second to Qian et al. [14]’ dataset is Vidgen et al. B [13]’s dataset, which performed well on three out of nine datasets (namely, Vidgen et al. [16], Qian et al. [14], and Gibert et al. [15]). The classifier performed well on only one out of nine datasets when trained on the following dataset: Davidson et al. [5], Waseem and Havoy [9], and Gibert et al. [15]. The classifier trained on Salmenin et al. [6], Gomez et al. [3], Kennedy et al. [2], Vidgen et al. [13], and Suryawanshi et al. [8] did not perform better than those trained on other datasets. However, it should be noted that their mean score across the nine datasets affected their ranking performance.

Table 9 shows the ranking of the generalised classifier performance for each dataset based on the overall mean performance across the nine datasets used for testing the classifier. The average mean score of the classifier indicates the generalization level of the classifier when trained on each dataset for hatespeech binary classification, i.e., the suitability of a dataset for training a baseline deep learning classifier in a generalised learning setting.

The classifier ranked first when trained on the Qian et al. [14] dataset and tested on nine other datasets, achieving 0.656. The classifier trained on the Vidgen et al. B [13] dataset ranked second with a mean of 0.649Ṫhe classifier performed worst when trained on Gomez et al. [3] dataset and tested on the other nine datasets, achieving the lowest mean weighted F1-score of 0.363. This poor performance indicates that the Gomez et al. [3] dataset is the least suitable for producing a baseline deep learning classifier in a generalised learning setting.

The other datasets (Vidgen et al. A. [16], Salminen et al. [6], Gibert et al. [15], Davidson et al. [5], Kennedy et al. [2], Suryawanshi et al. [8] and Waseem and Havoy [9]) achieved mean weighted F1-score between 0.532 and 0.599, as shown in Table 9.

We have calculated two statistical measures ($P-test$ and $T-test$) using the content of each dataset based on the labels “Hate” or “Not-Hate”. Details of our approach to computing these statistics are presented in section 3.4. Since machine learning algorithm performance is based on its learning from identifiable patterns in a given dataset, the $p$ value offers good indication on the available patterns in the ten datasets we have chosen for this study. A $p$ value of 0 indicates that the patterns in the dataset occurred by chance, which may reflect poor dataset annotation. A $p$ value of 1 indicates there is no difference in the patterns in the dataset. Our data analyses results are shown in Table 10.

The result of our analyses highlighted that the dataset from [3] has consistently performed poorly in both of the experiments: mono-dataset experiment and generalised learning experiment. From table 10 it can be noted that $p$ the value for this dataset is very close to 1 (0.9947), which means there is no recognisable pattern between “Hate” and “Not-Hate” content in this dataset. The lack of distinguishable patterns in the dataset highlights the main reason for the baseline classifier failing to learn sufficiently from this dataset, hence performing poorly, producing a weighted F1-score of 0.363 in the generalised learning experiment.

In contrast, as can be seen from table 9, the classifier produced a weighted F1-score of more than 0.5 for all those datasets with $p$ value 0.0001 and <0.1011, indicating that the model learned sufficient patterns to produce a weighted F1-score of over 0.531.

In supervised text classification, machine learning algorithms learn from a set of labelled data. Any given labelled dataset contain annotation errors due to many reasons (e.g., annotation procedure, annotator competency, data quality checking, ambiguities in natural language, etc.). Thus, machine learning algorithms are expected to make mistakes since they learn from annotated data. We use confusion matrices to highlight the classification errors the classifier made in each experiment (mono-dataset and generalised learning experiments). Due to space limitations, we provide comprehensive details on the classification errors of the best and worst performing classifiers compared to other models.