3M-Health: Multimodal Multi-Teacher Knowledge Distillation for Mental Health Detection

Recent studies in mental health classification from social media content have incorporated diverse social media components. These components encompass various elements, including historical posts, conversation trees, social and interaction graphs, user or post metadata information, and profile pictures or posted images (Cao et al., 2019, 2022; Lin et al., 2020; Sawhney et al., 2021a, b). However, those additional sources will not always be available in the dataset due to data privacy restrictions or user preferences. This complicates research reproducibility since each study selects features based on what social media components are available to them. Hence, our research focuses on exploring mental health detection by analysing only social media textual posts, which is a compulsory component of text-based social media posts related to mental health. Based on the textual aspect, existing studies have worked on frequency- or score-based emotion features (Aragón et al., 2023; Zogan et al., 2022). More recent works use the fine-tuned contextual embeddings on emotion-based tasks as the emotion features for mental health classification (Lara et al., 2021; Sawhney et al., 2021a). These studies mainly focus on identifying and fixedly matching one type of emotion to each word or entire textual content. On the other hand, our research highlights the complexity of human emotions wherein a single word could be associated with multiple types of emotions by integrating the emotion-enriched features generated through a multi-label, corpus-based representation learning framework. In addition, we propose a new way to include acoustic features generated by original textual posts in this task, motivated by the proven effectiveness of vocal biomarkers in psychological distress and other medical conditions indication (Iyer et al., 2022). This would simultaneously process the emotion-enriched features as additional information. To the best of our knowledge, there have been no attempts to create a new modality from the text-only input to achieve unimodal text-based mental health risk detection tasks.

To integrate the multimodal knowledge efficiently, we design our model with a knowledge distillation idea to compress a complex and large multimodal integration model into a smaller and simpler one while still retaining the accuracy and performance of the resultant model. Knowledge distillation (Hinton et al., 2015) involves transferring knowledge from a teacher model to a student model, commonly applied to compress large models by mapping intermediate layer outputs (Chen et al., 2021; Jiao et al., 2020) or minimising KL-divergence in class distribution (Mirzadeh et al., 2020). Traditionally, knowledge distillation is used within the same modality. However, recent approaches extend it to different modalities (Yang and Xu, 2021; Ni et al., 2022; Li et al., 2020). Some studies explore collaborative learning with multiple teachers for improved compression, such as (Wu et al., 2021) and (Pham et al., 2023) in language and vision models. (Tan et al., 2018) focuses on multilingual language translation, and (Vongkulbhisal et al., 2019) applies multi-teacher knowledge distillation to unify classifiers trained on distinct data sources. Inspired by this, we propose a new Multimodal Multi-Teacher Knowledge Distillation framework for mental health risk detection.

This section articulates each teacher model’s objective and construction process. Teacher fine-tuning can be found in Section 3.2.

Researchers (Jiao et al., 2020; Wu et al., 2021) emphasise the significance of distilling knowledge from the hidden states of a teacher model for effective student instruction. In this section, we describe the independent fine-tuning process of each teacher model.

We performed fine-tuning on the text-based teacher, utilising the pre-trained language model, for the mental health classification task with labels $C=\{c_{1},c_{2},...,c_{|C|}\}$. This process enables the teacher to learn and adapt to the nuances of mental health-related contexts within each dataset.

For the emotion-based teacher, following the generation of emotion-rich representations for each post and its words, these embeddings serve as inputs for fine-tuning a Multi-Layer Perceptron (MLP) for mental health classification, operating over the labels $C$.

We employ the Audio Spectrogram Transformer (Gong et al., 2021) (AST) for the audio-based teacher to classify each generated audio into mental health risk classes. AST is a transformer-based model that takes a sequence of audio spectrogram patches as inputs. An audio waveform is first converted into a 128x100t spectrogram based on a sequence of 128-dimensional log Mel filterbank features computed with a 25ms Hamming window every 10ms. Such a spectrogram is then split into a sequence of N 16x16 patches of images with an overlap of 6 in both time and frequency dimensions. A special token [CLS] is added to the beginning of the sequence of spectrogram patches. After passing through transformer encoder layers, the [CLS] embedding is fed into a linear layer with sigmoid activation to classify mental health risk class labels $C$.

Every teacher model is individually constructed and fine-tuned to facilitate optimal learning. We are aware of the concerns raised by some researchers (Wu et al., 2021) highlighting the potential inconsistency in the feature space when different teachers are separately pre-trained with distinct settings and then fine-tuned independently. Based on our initial testing, co-finetuning multimodal teachers yields little improvement; in fact, it tends to result in lower performance. We speculate that integrating multimodal information may not perform optimally during co-finetuning.

For the student model, we use a single modality involving textual posts as input for a pre-trained BERT, which processes the sequence of tokenised words. The student model performs the mental health risk classification task over the same class labels $C$, with knowledge distilled from the text-based, emotion-based, and audio-based teacher models. To incorporate the acquired knowledge from these various multimodal sources, the student model is trained to minimise the distillation loss given by $L=L_{task}+L_{kd}$. Here, $L_{task}$ represents the cross-entropy loss between the student model’s predictions and the ground truth of mental health risk categories, while $L_{kd}$ stands for the Kullback-Leibler (KL) divergence between the student model and the teacher models’ predictions. Given the presence of multiple teacher models, we calculate $L_{kd}$ by averaging the predicted probability distributions from all three teacher models.

We evaluate our proposed model using four publicly available datasets related to mental health on social media. Table 1 and Figure 2 provide a summary of statistics and an illustration of class distribution.

The TwitSuicide Dataset⁶⁶6Data available upon request. (Long et al., 2022) replicates the data collection, processing, and annotation methods of (O’Dea et al., 2015). A sample of 660 tweets is annotated into three risk levels. The Strongly Concerning (SC) class is assigned to posts with a convincing display of severe suicidal ideation, while Safe to Ignore (SI) shows no evidence of suicide risk. If it doesn’t fall into other categories, a post remains in the Possibly Concerning (PC) class.

DEPTWEET⁷⁷7https://github.com/mohsinulkabir14/DEPTWEET (Kabir et al., 2023) is collected from Twitter using seed terms based on the Patient Health Questionnaire (PHQ-9). The dataset comprises 40,191 tweets; however, only 5,128 tweets were retrieved during this study. The labels include Non-Depressed (ND), Mild (MI), Moderate (MO), and Severe (SE), maintaining an imbalanced class distribution, with around 80% labelled as ND and less than 2% SE.

The Identifying Depression Dataset⁸⁸8https://github.com/Inusette/Identifying-depression (IdenDep) (Pirina and Çöltekin, 2018) consists of 1,841 Reddit posts, with “depression indicative” (DE) posts sourced from the Depression subreddit and non-depressive (NDE) posts from the “family” and “friendship advice” subreddits. No further manual check was done on the samples, increasing the probability of false negatives.

The SDCNL Dataset⁹⁹9https://github.com/ayaanzhaque/SDCNL (Haque et al., 2021) involves distinguishing between Reddit suicide-related and depression-related posts. The dataset contains 1,895 nearly balanced posts labelled as Suicide (SUI) or Depression/Not Suicide (DEP) based on their subreddit. In accordance with (Benton et al., 2017), all posts are de-identified before any analysis, audio generation, and model training.

We provide a detailed breakdown of text and audio statistics for each dataset class in Tables 2 and 3 to provide more information regarding the nature of each class, which may influence model learning and performance. Notably, DEPTWEET and IdenDep datasets have highly skewed data, with 82.16% and 70.23% on a single class, respectively. Figure 3 illustrates the differences in the generated audio in terms of duration. The Reddit-based datasets, IdenDep and SDCNL, are significantly longer than the Twitter-based datasets, possibly providing more auditory information inferred from the textual posts.

In order to generate the best possible audio to represent each textual post in our benchmark datasets, we performed a theoretical and practical comparison between five publicly accessible text-to-speech and text-to-audio generative APIs.

1. (1)

   Tacotron2¹⁰¹⁰10https://github.com/NVIDIA/tacotron2 (Shen et al., 2018) uses a recurrent neural network architecture to predict mel spectrogram sequences from text followed by a modified WaveNet vocoder.

2. (2)

   SpeechT5¹¹¹¹11https://github.com/microsoft/SpeechT5 (Ao et al., 2022) unifies modalities with a shared encoder-decoder architecture that uses cross-modal vector quantisation for speech and text alignment.

3. (3)

   SpeechBrain¹²¹²12https://github.com/speechbrain/speechbrain/ (Ravanelli et al., 2021) is a speech toolkit offering various speech related tasks. Their text-to-speech model is based on Tacotron2 but is trained further on the LJSpeech (Ito and Johnson, 2017) and LibriTTS (Zen et al., 2019) datasets.

4. (4)

   Balacoon¹³¹³13https://huggingface.co/balacoon/tts packages offer lightweight and fast text analysis and speech generation going against larger but slower TTS models. It sacrifices multi-speaker and multi-lingual features for lightning-fast speed on the CPU. The detailed model architecture was not publicly available at the time of this paper’s writing.

5. (5)

   Bark¹⁴¹⁴14https://github.com/suno-ai/bark has a GPT-based architecture using a quantised audio representation that does not require the use of phonemes allowing it to generalise beyond speech, thus making it a text-to-audio model.

Upon comparison of the five generators, we use Bark due to the expressiveness of the audio generated by the model. While the other models suffer from a robotic delivery of the generated speech, not verbalising numerical figures, and reading of emoticons as individual punctuations (e.g. “>:|” as greater than, semicolon, pipeline), Bark produces the most naturally sounding audio recognising textual markers like “,” for pauses, “–” and “…” for hesitations, capitalisation for emphasis (e.g. goodbye vs. GOODBYE), and sentence punctuations for produce tonal shifts (e.g. huh? vs huh!). Bark also verbalises non-speech sounds such as [laughter], [laughs], [sighs], [music], [gasps], [clears throat], haha, uhm, waaah, and ooh. Bark’s ability to infer and convey emotions from only an input text would be valuable to our mental health risk detection model since it can provide additional emotional cues from the generated sound.

We assess the performance of our model by comparing it to previously published results using post-only¹⁵¹⁵15In contrast to studies incorporating other components such as posted images or user network and activity. and post-level classification on the same datasets, employing identical class labels and similar evaluation setups. We use results reported in the following studies as our baselines: Bi-LSTM Char+Word (Long et al., 2022) for TwitSuicide; MLP (Tadesse et al., 2019), and EAN (Ren et al., 2021) for IdenDep; and GUSE-DENSE (Haque et al., 2021), and AugBERT+LR (Ansari et al., 2021) for SDCNL. For the DEPTWEET dataset, we use the published DistilBERT code from (Kabir et al., 2023) to replicate baseline results for the retrieved dataset. In addition, we provide strong baselines from fine-tuning state-of-the-art PLMs: BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019), MentalBERT, and MentalRoBERTa (Ji et al., 2021). All PLM baselines follow the training setup used by (Long et al., 2022) with a batch size of 8 and a learning rate of 1e-04 trained for three epochs. Given the class imbalance, we evaluate our system based on macro F1 (F1m) and weighted F1 (F1w) scores, followed by accuracy and class F1 scores.

We evaluate our model following established evaluation setups from previous literature using the same datasets on the same classification task setup for fair benchmark comparisons. We use 10-fold cross-validation for TwitSuicide and IdenDep, while a train/test split is used for SDCNL and DEPTWEET. Original data splits are retained when provided; otherwise, the data is randomly split (Table 1). When none is given, 10% of the training set is used for validation.

Hyperparameter tuning is done per dataset and model setup using Optuna¹⁶¹⁶16https://optuna.org/ for 50 trials optimising weighted F1 scores. Detailed search space and best-found hyperparameters may be found in the Supplementary Material. Text-based teachers are trained using ReLu and a max length of 256. Audio-based teachers are trained for 25 epochs with a 5-epoch early stop, a 512 max length, and using the ReduceLRonPlateau scheduler. All inputs for the AST model are normalised to zero mean and 0.5 standard deviation. Emotion-based teachers are trained for 100 epochs with a 10-epoch early stop and a 256 max length. The student models are tuned and trained with distilled knowledge from the fine-tuned teachers using a max length of 256. All tuning was done using a 90:10 validation split and was conducted separately from the final model construction. All models are trained using an Adam optimiser on an NVIDIA TITAN RTX machine.

We compare our model with fine-tuned PLM baselines and several published baselines that use the same mental health detection datasets and evaluation setup. Note that we select post-only mental health detection models as mentioned in Section 1 and 4.3. We comprehensively evaluate the overall performance and class performance in Table 4.

Overall, our model outperforms all baselines on all four benchmark datasets. What should be noted is that our model does not have to be trained with all three different teachers to achieve the best results. As illustrated in Table 4, we have four datasets, the initial two originating from Twitter and the latter from Reddit. Our model demonstrates superior performance, even with partial teacher combinations. The datasets from Twitter produce the best results with the combination of Text and Emotion, whereas the Reddit-based datasets perform the best with Text and Audio Knowledge Distillation. Note that we detail the efficacy of each modality teacher combination on different datasets in Section 5.3. In addition, our model trained with all three modalities still outperforms the other baseline models in most cases and shows greater performance in identifying certain classes. Especially for the Moderate (MO) class of the DEPTWEET dataset, our model trained with all three modalities achieves a 34.34 F1 score, while our model trained with partial modalities only achieves a 12.31 F1 score. All the other pre-trained baseline models fail to recognise the MO class. Hence, we can conclude that learning from different modality teachers helps our model achieve much better performances than the baseline models that learned from only textual inputs. Such improvement is more noticeable on datasets with shorter texts. Specifically, our model’s best performance is 8.36% and 8.07% higher than the best-performing baseline model on the macro F1 and weighted F1, respectively, on the TwitSuicide dataset. For DEPTWEET, IdenDep, and SDCNL datasets, the best performances of our model are 6.03%, 5.40%, 4.40%, and 2.88%, 4.50%, 4.37% higher than the best-performing baseline model on the macro and weighted F1 scores, respectively.

To ensure the feasibility of our audio modality for mental health detection, we give an illustrative visualisation of the audio embeddings, which are generated by input text and learned via the Audio Spectrogram Transformer (AST). We conduct Principal Component Analysis (PCA) to visualise the acquired audio embeddings and their corresponding mental health class labels. In order to emphasise the distinguishability of the embeddings, we select samples from both the least and most concerning labels in each dataset, as shown in Figure 4. For each dataset, we group all the generated audio based on durations of 0-to-10-second and 10-to-25-second length¹⁷¹⁷17Note that most generated audios are less than 25 seconds.. For each of these two audio groups, we generated the corresponding spectrograms and randomly selected ten audio samples for each group to visualise the first two principal components after performing PCA. In Figure 4, we annotate each sample with the post ID and the audio duration for detailed comparison. Tables 5 to 7 contains de-identified and masked post contents for TwitSuicide, DEPTWEET, and IdenDep, respectively. SDCNL samples may be found in the Supplementary Material.

The visualisation shows that our audio embeddings can show a noticeable separation between mental health classes for all four datasets. In datasets derived from Twitter, shorter audio samples display more pronounced distinctions between the most and least concerning classes, whereas, in datasets from Reddit, this separation becomes more evident in longer audio segments. We assume that this is primarily due to Twitter posts being generally shorter in length, whereas Reddit posts tend to be longer.

To examine the efficacy of each teacher modality and their combinations, we evaluate and explore the performance by adding an extra emotion-based teacher, an audio-based teacher, or both alongside the text-based teacher. The results are presented in Table LABEL:tbl:ablationstudyteachers.

In general, the multi-teacher structure outperforms the use of a singular text-based teacher, although the effectiveness of different modalities varies across datasets. For the Twitter-based datasets, applying the emotion-based and text-based teachers together achieves the best results. In contrast, for Reddit-based datasets, the audio- and text-based teachers have better performances. This difference may be attributed to the longer posts in Reddit-based datasets, resulting in longer audio (Table 1) that contains more acoustic information beneficial for the audio-based teacher.

Moreover, due to the lengthier nature of posts in Reddit-based datasets, more SenticNet lexicon tokens are likely to be matched compared to Twitter-based datasets. This results in a higher number of generated emotion labels during the learning process of the multi-label emotion-based teacher. In Figure 5, we compare the count of generated multi-label emotion classes utilised to train the emotion-based teacher across all four datasets. It is evident that a greater proportion of posts in Reddit-based datasets match all seven emotion labels (shown in Section 3.1.2) compared to Twitter-based datasets. This potential increase in the number of matching emotion labels may present challenges in distinguishing between different emotions during the training of the emotion-based teacher, potentially impacting downstream mental health classification, especially for ambiguous classes such as Suicide (SUI) and Depression (DEP) in the SDCNL dataset.

We can conclude that using multimodality teachers generally helps detect mental health, and these findings also suggest varying effectiveness of different modalities across datasets with distinct characteristics, offering valuable insights into selecting suitable modalities for improved performance in future scenarios.

We compare the impact of various pre-trained language models on the effectiveness of the text-based teacher. According to the results in Table LABEL:tbl:ablationstudytextteachers, BERT performs the best in terms of the weighted F1 across all four datasets. However, the domain-specific PLMs, such as MentalBERT or ClinicalBERT, perform better for most of the more concerning classes. For instance, ClinicalBERT outperforms BERT by 6.66% in the F1 score for the Strongly Concerning (SC) class in the TwitSuicide dataset, and MentalBERT achieves a 41.38% F1 score for the Severe (SE) class in the DEPTWEET dataset, surpassing all other language models that fail to recognise it. To ensure optimal performance of the text-based teacher, we specifically employ MentalBERT for experiments on the DEPTWEET dataset, while BERT is used for the other three datasets. Nonetheless, the overall performance of the text-based teacher is not significantly impacted by the choice of pre-trained language models. Notably, more performance enhancement stems from the inclusion of different modalities, as mentioned in the previous sections.

We examine different combinations of multimodal inputs for the student model in Table 10 in order to explore the optimal input for a knowledge distillation for the student model. In this scenario, we concatenate emotion embeddings, audio embeddings, or both with textual post embeddings from pre-trained BERT and then pass them to the transformer layer after a linear layer projection. We also test a randomly initialised vanilla transformer compared to pre-trained BERT for the student model. The results indicate that unimodal textual post inputs outperform the concatenation of multimodal inputs for the student model. Moreover, pre-trained BERT yields better results than the randomly initialised vanilla transformer across all datasets. These outcomes underscore the effectiveness of multi-aspect knowledge acquired from the multi-teachers, efficiently guiding the student to achieve robust performance with only textual inputs.

We further investigate the impact of different patch size values for the audio-based teacher model, Audio Spectrogram Transformer (AST) (Gong et al., 2021), while maintaining a consistent setup for the student model. Figure 6 shows each dataset’s weighted F1 score for each patch size value tested. TwitSuicide, DEPTWEET, and IdenDep datasets show a relatively stable performance between 2 to 64 patch sizes; however, for SDCNL, performance improvement may be achieved using a patch size of 32. This improvement may be due to the higher variance in audio duration of the outliers in the SDCNL dataset compared to the other three datasets, as shown in Figure 3. Despite being shorter on average length and duration than the other Reddit-based dataset, SDCNL has audio samples with longer durations, which may have benefited from a patch size 32. However, a sharp decline in performance could be expected when the patch size is increased to 64.