LionGuard: Building a Contextualized Moderation Classifier to Tackle Localized Unsafe Content

The importance of content moderation has led to a plethora of works focused on the detection of toxic and abusive content (Nobata et al., 2016; de Gibert et al., 2018; Chakravartula, 2019; Mozafari et al., 2020; Vidgen and Yasseri, 2020). Bidirectional Encoder Representations from Transformers (BERTs) (Devlin et al., 2018) first emerged as powerful word embeddings that could be fine-tuned for downstream tasks like hate speech detection. Lee et al. (2021) used BERT embeddings in the Hateful Memes Challenge (Kiela et al., 2021), while Liu et al. (2019) combined BERT and a Long-Short Term Memory model on the OffensEval dataset (Zampieri et al., 2019). Caselli et al. (2021) re-trained BERT on offensive Reddit comments, building a shifted BERT model, HateBERT, that outperformed general BERT in hate speech detection.

Moderation APIs have become increasingly popular due to the ease at which they can be integrated into applications. Such APIs aim to be universally applicable to different languages and domains. Jigsaw (2017) developed Perspective API, which uses multilingual BERT-based models that are then distilled into single-language Convolutional Neural Networks (CNNs) for each language supported. Markov et al. (2023) developed OpenAI’s Moderation API, which uses a lightweight transformer decoder model with a multi-layer perceptron head for each toxicity category. However, one concern amidst the increasing adoption of moderation APIs is how strikingly different toxicity triggers are across the Western and Eastern contexts (Chong and Kwak, 2022), underscoring the importance of localized content moderation.

Adapting toxicity detection to Singlish, Zou (2022) used a CNN to detect hate speech from Twitter data. Haber et al. (2023) curated a multilingual dataset of Reddit comments in Singlish, Malay and Indonesian and found that domain adaption of mBERT (Devlin et al., 2018) and XLM-R (Conneau et al., 2020) models improved F1 performance in detecting toxic comments. Prakash et al. (2023) analyzed multimodal Singlish hate speech by creating a dataset of offensive memes. Our work contributes to this space by establishing a more systematic approach to detecting unsafe content with automated labelling and by developing a contextualized moderation classifier which outperforms existing generalized moderation APIs.

Despite requiring more time and resources, human labelling has frequently been used to generate gold standard labels for toxic speech, particularly via crowdsourcing (Davidson et al., 2017; Parrish et al., 2022). However, Waseem (2016) found that amateur annotators were more likely than expert annotators to label items as hate speech, causing poorer data quality. Considering the scale of data required for building safe LLMs, automated labelling has emerged as an alternative to human labelling. Bai et al. (2022) used Constitutional AI to automatically perform evaluations with Claude, and then trained a preference model using the dataset of AI preferences. Chiu and Alexander (2021) found that with few-shot learning, GPT-3 can be used to detect sexist or racist text. Plaza-del arco et al. (2023) also found that zero-shot prompting of FLAN-T5 produced favorable results on several hate speech benchmarks. Inan et al. (2023) proposed an LLM-based input-output safeguard model, LlamaGuard, which classifies text inputs based on specific safety risks as defined by prompts. Unlike existing works that rely on a single model for automated labelling, we combined several LLMs to provide more accurate and reliable labels, leveraging the collective wisdom and knowledge of several safety-tuned LLMs.

To build a dataset of Singlish texts, we collected comments from HardwareZone’s Eat-Drink-Man-Woman online forum and selected subreddits from Reddit on Singapore.⁹⁹9r/Singapore, r/SingaporeHappenings, r/SingaporeRaw The former is notorious in Singapore as a hotspot of misogynistic, xenophobic, and toxic comments,¹⁰¹⁰10https://www.ricemedia.co/pretty-privilege-bbfa/ while the latter is a popular online forum for Singapore-specific issues. We collected comments on all threads between 2020 and 2023 from both forums, resulting in a dataset of approximately 8.9 million comments.

However, upon manual inspection of the data, only a small minority of the comments were unsafe. Both forums have a wide range of topics which are not always controversial or harmful, and forum moderators often remove the most toxic comments.

To ensure sufficient unsafe texts for a balanced dataset, we used entire threads that discussed controversial topics in Singapore or contained offensive words (see Appendix A), which were more likely to be unsafe. This resulted in approximately 400,000 texts, which we randomly subsampled to a smaller set of 69,000 potentially unsafe texts. We then randomly sampled another 69,000 texts from the remaining dataset that had not been identified as likely to be unsafe, for greater heterogeneity in topics and language, resulting in a final training dataset of 138,000 texts (examples in Appendix B).

Next, we referenced the moderation categories defined in OpenAI’s Moderation API, Jigsaw’s Perspective API and Meta’s LlamaGuard, and took into consideration Singapore’s Code of Internet Practice and Code of Practice for Online Safety.¹¹¹¹11https://www.imda.gov.sg/-/media/imda/files/regulations-and-licensing/regulations/codes-of-practice/codes-of-practice-media/code-of-practice-for-online-safety.pdf

We defined seven categories of safety risks for LionGuard. Brief descriptions of each category are listed below, while the full definitions are available in Appendix C. The key differences between our safety risk categories and OpenAI’s, Jigsaw’s and Meta’s are summarized in Table 1.

1. 1.

   Hateful: Content that expresses, incites, or promotes hate based on race, gender, ethnicity, religion, nationality, sexual orientation, disability status, or caste.

2. 2.

   Harassment: Content that expresses, incites, or promotes harassing language towards any target/individual.

3. 3.

   Encouraging public harm: Content that promotes, facilitates, or encourages harmful public acts, vice or organized crime.

4. 4.

   Encouraging self-harm: Content that promotes or depicts acts of self-harm, such as suicide, cutting, and eating disorders.

5. 5.

   Sexual: Content meant to arouse sexual excitement, such as the description of sexual activity, or that promotes sexual services (excluding sex education and wellness).

6. 6.

   Toxic: Content that is rude, disrespectful, or profane, including the use of slurs.

7. 7.

   Violent: Content that depicts death, violence, or physical injury.

We then automatically labelled our Singlish dataset according to our safety risk categories using LLMs. Automated labelling with LLMs is increasingly popular given vast improvements in instruction-following with recent LLMs (Ouyang et al., 2022; Muennighoff et al., 2023; Xu et al., 2024).

To verify the accuracy of our automated labelling, we internally labelled 200 texts that served as our expert-labelled dataset. The dataset was handpicked by our team with a focus on selecting particularly challenging texts that were likely to be mislabelled. This consisted of 143 unsafe texts (71.5%) and 57 safe texts (28.5%).

Architecture: LionGuard, our moderation classifier, comprises two components: an embedding and classifier model. The embedding model generates a vector representation of the text, which the classifier model uses as inputs to generate a moderation score. This simple architecture enables us to test different embedding and classifier models to find the best-performing combination for LionGuard.

Embedding model: Our approach compared general embedding models against finetuned models. We chose BAAI General Embedding (BGE) (Xiao et al., 2023) given its strong performance on Hugging Face’s leaderboard for embeddings,¹⁶¹⁶16https://huggingface.co/spaces/mteb/leaderboard HateBERT (Caselli et al., 2021), as well as SingBERT (Lim, 2023). We also experimented with masked language modelling (MLM) on these embedding models on a separate sample of 500,000 texts from our initial dataset of 8.9m texts for 30 epochs. Ablation studies were also conducted with BGE-small, BERT-base and BERT-large embedding models.

Classifier model: We selected our classifier models based on different levels of model complexity to reveal any differences in performance due to the number of parameters. In order of complexity, we chose a ridge regression classifier, XGBoost classifier, and a neural network (consisting of one hidden and one dropout layer). We carried out hyperparameter tuning for the XGBoost and neural network classifier. More details on the hyperparameter search and the final set of hyperparameters are provided in Appendix F.

Training: We developed two versions of LionGuard: a binary classifier (to detect if a text is safe or unsafe), and a multi-label classifier (to detect if a text fulfills any category in our safety risk taxonomy defined in 4.2).

For the binary classifier, we limited the training data to texts where there was consensus among the LLMs on the label (i.e., unsafe or safe). This resulted in a smaller dataset of 99,597 texts (72.2%).

For the multi-label classifier, we trained a dedicated classifier model for each category. We included texts where there was a consensus for that category, which enabled us to maximize the use of our limited number of positive labels. Apart from the toxic category, there was consensus on over 96% of the labels for each of the other categories.¹⁷¹⁷17For the toxic category, the consensus rate was 72.4%. Although this meant there was less training data for the toxic-specific classifier, there was still more than enough training data (around 99,900 texts). Moreover, the toxic category also had more positive labels than the other categories.

Evaluation: Due to the heavily imbalanced dataset, we chose the Precision-Recall AUC (PR-AUC) as our evaluation metric as it can better represent the classifier’s ability to detect unsafe content across all score thresholds. PR-AUC was also used by OpenAI (Markov et al., 2023) and LlamaGuard (Inan et al., 2023) in their evaluations.

Benchmarking: After identifying the best classifier, we compared LionGuard with Moderation API, Perspective API and LlamaGuard. Both APIs provided a score, while LlamaGuard provided the probability of the first (classification) token, which we used to calculate the PR-AUC. We benchmarked them on both the binary and multi-label experiments.

We compared labels provided by the LLMs with labels annotated by crowdsourced human labellers to further validate the accuracy of LLM labels. We worked with TicTag, a Singapore-based annotation company, to crowdsource for workers who could understand Singlish. These workers accessed the labelling task via TicTag’s mobile application (see Appendix G), and were provided extensive instructions, including the safety risk taxonomy and examples. They could choose "I Don’t Know" if they did not understand the text. 95 workers labelled 11,997 unique texts randomly drawn from our final dataset (see subsection 4.3.4), with each text labelled by 3 different workers. The demographic profile of the workers were reflective of Singapore’s population characteristics (see Appendix G.1).

Of the 11,997 texts, we found that crowdsourced human labellers had low concurrence (i.e., inter-rater agreement). As seen in Table 3, human labellers only had full concurrence on binary labels 52.9% of the time. Concurrence on binary labels was significantly lower than individual categories as we found that human labellers tended to label more texts unsafe, resulting in lower concurrence on generally safe texts. On less contentious categories like self-harm, public harm, sexual and violence, concurrence occurred more than 85% of the time. In contrast, toxic and hateful categories had less than 75% concurrence. Even with detailed instructions and strong quality control measures, the inherent subjectivity of labelling harmful content makes it challenging to achieve consensus among non-expert human labellers.

For sentences with concurrence among all human labellers and all LLM labellers respectively, we found that the human labels generally have high concurrence with LLM labels (see Table 3), with the concurrence rate exceeding 90% for all categories. This suggested that where human labels were consistent, LLMs were relatively accurate in providing labels aligned with human judgment. However, in contentious and ambiguous cases where human labels are inconsistent, evaluating the accuracy and concurrence of LLM labels vis-à-vis human labels is an area for future work.

Model experimentation results (see Table 4): On the binary label, we found that the classifiers which used BGE Large performed significantly better than those which used HateBERT and SingBERT. Based on our ablation study with BERT-base, BERT-large and BGE-small models (see Appendix H), which all performed poorly, we posit that the number of parameters and type of pre-training embeddings are critical in improving performance. As for the classifier model, the ridge classifier performed comparably to XGBoost and the neural network for all embedding models despite its relative simplicity. We also found that MLM finetuning on the embedding models had a negligible effect on performance (see Appendix H).

For the multi-label classifiers, we similarly found that the classifiers which used BGE Large were the best performers by a large margin. Likewise, the ridge classifier performed best, indicating that a simple classification layer is sufficient for good performance, given a complex embedding model.

Overall, the best performing combination was the BGE model combined with the Ridge classifier. We used this combination for LionGuard, our moderation classifier.

Benchmarking results (see Table 4): We found that LionGuard significantly outperformed Moderation API, Perspective API and LlamaGuard.

On the binary label experiments, LionGuard’s PR-AUC score of 0.819 is higher than OpenAI’s 0.675, Perspective’s 0.588 and LlamaGuard’s 0.459.¹⁸¹⁸18Additionally, LionGuard scored lower than Moderation API on precision (0.63 vs 0.74) but significantly higher on recall (0.81 vs 0.56) when using 0.5 as the prediction threshold. Likewise, for multi-label classification, LionGuard outperformed on all categories. The difference in performance is especially clear for the harassment, sexual, toxic and violent categories, with the performance more than doubled.