A Depression Detection Method Based on Multi-Modal Feature Fusion Using Cross-Attention

The dataset employed in our research is Weibo User Depression Detection Dataset (hereafter referred to as WU3D), compiled by Wang et al.[7]. Specifically, within the Weibo platform, each user possesses a unique ID, and the WU3D dataset accesses the homepage of each user through web crawling techniques to gather information. With a focus on ensuring authority and high reliability, the dataset underwent dual scrutiny by psychologists and psychiatrists for labeling accuracy. The information collected for each user is illustrated in Fig. 1.

Attention is a mechanism, which has found widespread application in Natural Language Processing (NLP) and computer vision tasks. At its core, the idea is to enable models to selectively focus on the most crucial parts of the input information during processing, rather than treating all input elements equally or averaging their importance. This mechanism mimics the way humans concentrate their attention on salient details when processing complex information, thereby enhancing the efficiency and performance of models.

Transformer: Transformer is a groundbreaking sequence transduction model, first proposed by Vaswani et al. in their seminal paper ’Attention Is All You Need’ in 2017[8]. It has utterly transformed the field of Natural Language Processing (NLP), reshaping the landscape dominated by Recurrent Neural Networks (RNNs) and their variants, such as Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRUs). These models traditionally struggled with computational inefficiency when handling long sequences. Vaswani and colleagues addressed this by abandon the loop structure altogether, relying exclusively on self-attention mechanisms. This innovation demonstrated superior performance compared to the best RNN models of the time in machine translation tasks, while also significantly accelerating training speeds.

Cross Attention: Cross Attention is a specialized form of attention mechanism that primarily deals with dependencies between two different sequences[9]. Unlike self-attention, which focuses on interdependencies among elements within the same sequence, cross attention operates through a ’query-answer’ paradigm, enabling one sequence to adjust its representation based on the content of another sequence. In this paper, to uncover the relationships between word features and statistical characteristics, the cross attention mechanism has been employed.

Large-scale pre-trained models (PTMs), benefiting from intricate pre-training objectives and substantial parameter counts, excel at extracting knowledge from vast amounts of both labeled and unlabeled data. They encapsulate this abundant knowledge within their extensive parameters, which, when fine-tuned for particular tasks, can greatly enhance a wide array of downstream applications. This efficacy has been amply validated through rigorous experimentation and empirical assessments[10].

MacBERT: MacBERT[5] is an improved version of BERT, incorporating Error-Correcting Masked Language Model (MLM as Correction, abbreviated as Mac) into its pre-training regimen, addressing the inconsistency issue between ”pre-training” and ”downstream tasks.” This model replaces the conventional MLM task with the MLM as Correction (Mac) task, thereby mitigating discrepancies between the pre-training and fine-tuning stages.

This module specializes in processing user text information originating from social media platforms, encompassing multiple dimensions of users, including nicknames, personal profiles, and posted tweets, among others. The process begins by integrating these various categories of information, concatenating them to form a continuous long-text sequence. This design is intended to capture and preserve the full picture of users’ online behavior, as each component may harbor clues about their personality traits, emotional states, or social habits.

The constructed long-text sequence is then fed into a pre-trained MacBERT model. As an enhanced version of the BERT model, MacBERT introduces error-correcting masked language modeling tasks during pre-training, optimizing the model and reducing inconsistencies between pre-training and downstream tasks. This process enables not only the learning of lexical meanings but also the understanding of contextual nuances, extracting profound semantic information embedded within the text. Through MacBERT’s processing, the raw text information is transformed into high-dimensional word embedding vectors. These vectors carry rich semantic and contextual information, furnishing high-quality input data, referred to as input 1, for subsequent natural language processing tasks.

This input 1, derived from deep learning model transformation, embodies a comprehensive understanding and parsing of user textual expressions. It not only facilitates the identification of key terms but also delves into the text to uncover emotional tendencies, underlying themes, and intricate patterns of interpersonal interactions. In applications such as depression detection, this nuanced text analysis capability is particularly crucial, enabling the recognition of subtle linguistic cues associated with depressive sentiments and providing robust data support for mental health assessments.

This module is responsible for extracting statistical information about users. Based on the research of previous work[7][11][12][13][14][15][16], we have adopted six statistical features: the proportion of negative emotional tweets, the proportion of original tweets, the ratio of posts made during late-night hours, the frequency of posts per week, the standard deviation of posting times, and the proportion of posts that include images. The detailed descriptions of these statistical features are provided in Table I. Detailed calculations for each of these statistical features along with their comprehensive descriptions will be elaborated upon in Section IV-B. Following the manual extraction of these statistical features, they are fed into a fully connected network for encoding, serving as input 2, tailored to conform to the dimensional specifications required by the subsequent cross-attention mechanism module.

Cross-attention[17] is a mechanism predominantly used in Transformer models, calculating attention across two distinct sequences to manage semantic relationships between them. It finds applications in tasks such as machine translation, image captioning, and video-text alignment. This mechanism expands upon self-attention, enabling the model to dynamically aggregate information at each position of one sequence (the query sequence) based on the content of another sequence (the key-value sequence).

The cross-attention mechanism is a specialized form of multi-head attention, where the input tensor is divided into two parts, $X_{1}\in\mathbb{R}^{n\times d_{1}}$ and $X_{2}\in\mathbb{R}^{n\times d_{2}}$.One part serves as the set of queries, while the other part acts as the key-value set.Its output is a tensor of size $n\times d_{2}$, where for each query vector, there are attention weights assigned to all the key vectors. Fig. 3 illustrates the computation process of cross-attention, where through matrix operations and the softmax function, the model learns to selectively gather pertinent information from the key-value sequences based on each element in the query sequence. This enhances the model’s capability in handling cross-sequence dependencies, thereby boosting its performance. Such a mechanism is particularly crucial in dealing with multimodal tasks, as it enables the model to flexibly integrate features from disparate modalities. For instance, in the joint analysis of image and text, image features can serve as the key-value sequences, while the textual descriptions act as the query sequence. The cross-attention mechanism then facilitates precise alignment and information fusion between the two, thereby enhancing the interpretability and accuracy of the combined multimodal analysis.

Specifically, let $Q=X_{1}W^{Q}$ and $K=V=X_{2}W^{K}$, the computation of the cross-attention is as follows:

Where $W^{Q}\in\mathbb{R}^{d_{1}\times d_{k}}$ and $W^{K}\in\mathbb{R}^{d_{2}\times d_{k}}$ represent learned projection matrices, $d_{k}$ denotes the dimensionality of the key-value set (which is also the dimensionality of the query set).

Below, we will elaborate on the computation process of cross-attention, as depicted in Figure 10086. This figure illustrates a mechanism known as ”Cross-attention,” a commonly employed technique in many modern Natural Language Processing (NLP) tasks. Within this process, V, K, and Q represent three matrices that collectively determine the attention scores. Here is a concise explanation of each component:

• •

  V Matrix (Value): The Value matrix encapsulates information from the input sequence, which has been encoded for the purpose of computing attention scores. Typically generated by an embedding layer of the input sequence, this layer is responsible for transforming raw text into numerical representations.

• •

  K Matrix (Key): Similarly derived from the input sequence, the Key matrix serves to ascertain which positions are more significant during the calculation of attention scores. Elements in the K matrix are compared against those in the Q matrix to decide which positions ought to receive heightened attention.

• •

  Q Matrix (Query): The Query matrix, another derivation from the input sequence, is utilized to inquire about the most relevant positions within the K matrix. Through similarity matching between elements in the Q matrix and those in the K matrix, it facilitates the determination of attention scores.

• •

  Attention Scores: Attention scores are the outcome of the interaction between the V, K, and Q matrices. By assessing the similarity between the Q and K matrices, attention scores are computed. These scores are subsequently employed to weight and combine values from the V matrix, yielding the final output.

In practical applications, the cross-attention mechanism typically involves the following steps, as exemplified by Formula 1:

1. 1.

   Linear Transformation: Initially, the V, K, and Q matrices undergo linear transformations, often through multiplication with weight matrices, to ensure they all possess the same dimensionality. This preprocessing aligns them for the subsequent computations.

2. 2.

   Inner Product Operation: Subsequently, each element in the Q matrix is paired with every element in the K matrix through an inner product operation, yielding a scalar value. These scalars quantify the degree of similarity between each element in the Q matrix and its corresponding elements in the K matrix.

3. 3.

   Softmax Function: The outcomes from the inner products are then fed into a softmax function. This operation scales the results such that each position’s attention score falls within the interval $[0,1]$, with the sum of all scores equaling 1. Consequently, a probability distribution is obtained, reflecting the relative importance of each position.

4. 4.

   Weighted Summation: Finally, the values in the V matrix are aggregated via a weighted summation process, where the weights are determined by the attention scores. This means that positions with higher attention scores contribute more significantly to the resultant output.

The cross-attention mechanism is particularly valuable in neural network architectures like the Transformer, as it enables the model to focus on different segments of the input sequence. This is crucial for comprehending complex contextual dependencies and long-range relationships. Consequently, incorporating cross-attention as our multimodal feature fusion module proves highly efficient; it empowers the model to allocate attention resources flexibly, thereby achieving enhanced performance.

The Multilayer Perceptron (MLP)[18], short for Multilayer Perceptron, is a type of feedforward artificial neural network model that serves as an extended version of the single-layer perceptron. By incorporating one or more hidden layers, MLP enhances the learning capacity and expressive power of the network. This structural enhancement enables MLPs to fit complex nonlinear functional relationships, thereby excelling in solving many practical problems, particularly in classification and regression tasks.

In this paper, MLP is employed as the final classification head module, comprising two essential components: linear transformation layers and activation functions. Specifically, the MLP consists of two layers of linear transformations, with a Rectified Linear Unit (ReLU) activation function inserted between them. Renowned for its simplicity and efficacy, the ReLU activation function effectively alleviates the vanishing gradient problem, facilitating the training of deeper networks, as defined by the following equation:

where $\mathrm{MLP}(\cdot)$ represents the multilayer perceptron (MLP), $x$ represents the input feature vector, $W_{1}$ and $W_{2}$ are the weight matrices of the two linear transformation layers, and $b_{1}$ and $b_{2}$ respectively denote the bias terms for these two layers. The ReLU function is defined as $f(x)=\max(0,x)$, which maps all negative inputs to zero while preserving positive values unchanged. This approach maintains the nonlinearity of the network while circumventing the issue of gradient saturation in the negative value region.

Equipped with this architecture, the MLP is capable of learning high-level abstract features from the input data and, through its final output layer, provides category predictions. In the context of depression detection methodologies, the MLP serves as the decision layer, integrating multimodal features fused via the cross-attention mechanism. Through nonlinear transformations, these features are mapped onto a classification probability space, thereby facilitating accurate judgment regarding whether a user is experiencing depression. This design not only enhances the model’s ability to recognize complex emotional and behavioral patterns but also ensures that the model exhibits good generalization performance, capable of making accurate predictions on unseen data.

In this section, we delve into the inner world and behavioral manifestations of individuals with depression from a psychological perspective, to present readers with a vivid mental health landscape. Depression, a prevalent mental health issue worldwide[19], not only impacts an individual’s emotional experiences but also significantly alters their cognitive functions, social behaviors, and even physical health[20]. Our aim, through meticulous data analysis, is to illuminate the distinctions between depression and other mental health states, thereby enhancing public comprehension of this complex psychological condition[21].

1. 1)

   Analysis of Characteristics in Individuals with Depression: The subjective experience of individuals with depression is often characterized by pervasive negative emotions, including sadness, hopelessness, self-deprecation, and loss of interest[22]. These emotions are not confined to specific situations but permeate throughout their daily lives, markedly diminishing their quality of life and sense of well-being. From a cognitive standpoint, those with depression frequently exhibit difficulties in concentrating, memory impairment, and diminished decision-making abilities – cognitive dysfunction that may be linked to alterations in the functioning of the prefrontal cortex and hippocampus regions of the brain. Furthermore, a pessimistic cognitive bias is prominent, manifesting as the overgeneralization of negative experiences and holding bleak outlooks for the future, which constitute key features of depression[23].

2. 2)

   Changes in Social Behavior: Social skills deficits are common among depressed patients, but little attention has been paid to this aspect of depression[24]. Within the realm of social interaction, individuals with depression tend to withdraw from their social circles and decrease engagement with others. Depressive symptoms can make individuals more sensitive to experiences of social exclusion and acceptance in their daily lives[25]. This retreat may stem from a lack of motivation to socialize, fear of being a burden to others, or an inability to derive pleasure from previously enjoyed social activities. Analysis of online behavior reveals that the social media usage patterns of those with depression also diverge; they often post more negative content, engage less in liking and commenting, and exhibit increased activity during late-night hours. These behavioral patterns, to some extent, mirror their psychological state.

3. 3)

   Comparison between the General Population and Individuals with Depression: In contrast, the general population exhibits a greater diversity in emotional expression, capable of experiencing and expressing a wide range of emotions, including positive ones such as joy and gratitude. Regarding cognitive function, they generally possess better control over attention, memory efficiency, and problem-solving capabilities. Socially, they tend to maintain stable social networks, engage in various social activities, and derive support and satisfaction from these interactions. Furthermore, their online behavior is more active and diverse, encompassing sharing daily life updates, participating in discussions, and fostering a broad network of social connections.

According to Figure 4, we observe two social media users who have posted multiple tweets, which indicate they might be experiencing symptoms of depression. Let’s analyze these tweets from a psychological perspective:

Firstly, the sentence, ’I’m struggling to catch my breath again, living through the same repetitive days with exhaustion. And, well, the world has actually gone mad.’ suggests that the person (Xiaoqi wants to sleep) feels utterly exhausted, which can be a symptom of depression. Following this, ’What kind of existence is depression? I don’t want to explain this to anyone anymore…… My nights are broken, waking up repeatedly, this cycle repeating endlessly,’ indicates a loss of interest and pleasure in life, a common manifestation of depression. Furthermore, ’……Thoughts of self-harm—cutting, jumping from heights, drowning, consuming pesticides or sleeping pills—haunt me……I am only 18, and I feel like I can’t hold on any longer,” reveals the presence of suicidal ideation, which is a severe indicator of a mental health crisis.

Another individual, referred to as (’Crying record book’), also expresses negative sentiments: ’They would be happy if I were gone.’ And, ’I wish I could choose to end my life on my own terms.’ These statements similarly demonstrate a sense of despair towards life and suicidal tendencies.

Taken together, these remarks all point towards the likelihood of depression. Depression is a treatable condition, and through our work, we are able to identify tendencies toward depression in its early stages, providing warning signals to professional caregivers to seek assistance, prompting intervention, and offering necessary psychological support or treatment recommendations. Timely intervention can improve the quality of life and prevent severe outcomes. Moreover, in extreme cases, it can avert tragedies such as suicide, saving precious lives.

In our study, to conduct a comprehensive analysis of users’ emotional states, we integrate various pieces of information left by users on social media platforms, including their usernames, profile descriptions, and posts or retweets, forming a long text sequence. Notably, these posts comprise not only original content created by users but also retweets that shed light on their interests and emotional expressions. Through such integration, the constructed long text sequence transcends the limitations of individual posts, establishing strong contextual links across multiple tweets. Users may sequentially post tweets at different times to articulate their experiences battling depression or seek solace and express inner distress on social media. Consequently, aggregating this information is crucial for discerning whether a user is undergoing depression. We also observe that posts by depressed users do not always overtly convey distress or relate directly to depression; they may depict routine aspects of daily life. This necessitates meticulous semantic extraction from the text during depression prediction, distinguishing between diverse expressions and identifying subtle signs of depression amidst everyday content.

To address this, we employ the MacBERT model as an upstream tool for word embedding. MacBERT, an enhanced version of BERT, introduces a Masked Correction (Mac) pre-training task for the masked language model, resolving the inconsistency issue between pre-training and downstream tasks. By refining its profound understanding of linguistic contexts, MacBERT enhances performance in downstream tasks. Building upon this, we further leverage manually extracted statistical features, combining them with the word features output by MacBERT through a cross-attention module. The cross-attention mechanism effectively matches points of focus between two distinct feature sequences, fostering complementary information exchange and integration, thereby enhancing feature expressiveness.

Finally, the fused features are fed into a Multilayer Perceptron (MLP) classification model, acting as the decision-maker for the downstream task to yield the final depression detection outcome. Through this meticulously designed process, our system can more accurately identify indicators of depression from intricate social media data, providing robust technical support for early detection and intervention in mental health issues.

Previous studies have defined features that are effective for detecting depressed users, we manually extracted six features, as detailed in Table I.

Not only do we compare against baseline methods, but we also incorporate several currently popular Chinese pre-trained models in combination with our feature fusion network, conducting separate experiments to evaluate the performance of our network.Furthermore, we investigate whether incorporating bi-GRU after the pre-trained model enhances the performance of our model.

1. 1)

   XLNet with MFFN[7]: We reproduced their approach, as outlined in [7], to establish our baseline. They adopted the widely-used pre-trained model, XLNet[28], for extracting text-based word features. Subsequently, They designed a Deep Neural Network classification model, termed the Multimodal Feature Fusion Network (MFFN), which consolidates features from various information sources.

2. 2)

   LERT[29]: LERT is a pre-training model that, in addition to employing masked language modeling, integrates three linguistic features and adopts a linguistically inspired pre-training mechanism. Experimental results demonstrate that the LERT algorithm can significantly enhance the performance of various pre-trained language models.

3. 3)

   PERT[30]: PERT is an innovative pre-training model that does not rely on the use of masked tokens ([MASK]) in conventional masked language models (MLMs) to learn semantic information from text. Instead, PERT adopts a methodology based on permuted language modeling to autonomously capture textual meaning. This approach involves rearranging the sequence of input text and then tasking the model with predicting the correct position of each token in the original text. Through this process, the model is compelled to learn deeper semantic structures and dependencies within sequences without explicitly concealing parts of the input.This method has demonstrated performance enhancements across a range of Chinese and English natural language understanding (NLU) tasks, indicating that pre-training through permuted language models effectively enhance the model’s capability in understanding and generating text, paving the way for new pre-training techniques in the exploration of natural language processing.

In this subsection, we provide a detailed analysis of the experimental results obtained by combining different pre-trained models with our cross-attention based feature fusion network. Additionally, we examine the impact of incorporating bi-GRU following the pre-training model on the experimental results.The results are presented in Table II and Fig 5. Specifically, note that CA in the table denotes our method of fusing multimodal features using cross-attention.

• •

  Table II illustrates the experimental outcomes of various approaches, with and without the integration of Bidirectional Gated Recurrent Units (bi-GRU). Evident from the table, among all methods, MacBert+CA (our proposed method) excels in both accuracy and F1 score, irrespective of the inclusion of bi-GRU. This suggests that our method possesses strong generalization capabilities and robustness. Strikingly, however, for all methods, the addition of bi-GRU leads to a noticeable decrease in accuracy, leading us to conclude that bi-GRU has an adverse effect on these pre-trained models.

• •

  Fig. 5 illustrates the evolution of validation set accuracy throughout the training process. Notably, apart from the baseline model, all other models incorporate our multimodal feature fusion approach, which employs cross-attention for integration and utilizes an MLP (Multi-Layer Perceptron) for the final classification task. The baseline model (red curve) experiences a decline in accuracy during the early stages of training, but it unexpectedly begins to rise after Iteration 1000, eventually stabilizing at a level that, while consistent, remains lower than those of other enhanced models (with the exception of PERT+bi-GRU). Initially, the Lert (blue curve) and Pert (green curve) models exhibit similar performance; however, as training progresses, Lert surpasses Pert in terms of accuracy. Upon integrating the bi-GRU structure, as seen in Lert+bi-GRU (blue curve) and Pert+bi-GRU (another green curve), their accuracy rates generally decrease, yet both demonstrate robust stability throughout the training process.

• •

  Macbert (yellow curve) and Macbert+bi-GRU (yellow curve) emerge as the top performers among all models. Specifically, Macbert achieves high accuracy very early in the iteration phase and sustains this peak performance throughout the entire training period. The experiment suggests that adopting advanced techniques like Lert and Macbert can significantly enhance the validation accuracy of the model. Pert, on the other hand, performs marginally worse than other models in this task, even falling below the baseline accuracy at times, aligning with previous observations that its effectiveness can be task-dependent and caution is advised in its application.

In order to comprehensively understand the impact of individual components within our proposed models, we conducted an ablation study. Specifically, we fed the word embeddings processed by MacBert into both bi-GRU and Transformer modules separately for feature extraction, thereby investigating their respective influences on the model’s performance. The results of this ablation experiment are depicted in Table III.

The comparison between the bi-GRU and Transformer modules reveals that the Transformer module marginally outperforms the bi-GRU module. However, it is noteworthy that the standalone MacBert model generally excels over the MacBert models augmented with either the bi-GRU or Transformer modules, affirming the efficacy of MacBert in this particular task. The performance of the MacBert network with the Transformer module added and without any additional module is nearly equivalent, exhibiting accuracies (ACC) of 0.9495 and 0.9490, and F1-scores of 0.9469 and 0.9465, respectively. Our ablation study underscores the significant role each component plays in the holistic performance of the model.

We conducted a case study to explore the accuracy and robustness of our model.

We have provided three examples for analysis, as shown in Table IV, Table V, and Table VI. Let us proceed to analyze their respective outcomes:

1. 1.

   Case 1: Table IV presents the profile information and tweets of a user named ’Director Ren Youbing’ who identifies himself as the ’Director of the Depression Research Institute.’ Through his profile description and the content of the illustrated tweets, signs of mild depression are noticeable. Remarkably, ’Director Ren Youbing’ exhibits a degree of self-awareness, openly acknowledging his mental health state. In one particular tweet, he inquires, ’I’ve posed the question: What are some self-healing methods for mild depression?’ This demonstrates not only his awareness of his condition but also a proactive approach in seeking remedies through social media platforms.

   For users like ’Director Ren Youbing’, whose profiles and tweets frequently incorporate negative terms such as ’depression,’ our model demonstrates a high level of sensitivity and precision. It is capable of accurately identifying depressive tendencies and swiftly converging on predictions, highlighting the efficacy and reliability of our model in assessing mental health indicators within online contexts.

2. 2.

   Case 2: Table V introduces an individual with the username ’Mr. Yusan is also begging for him today,’ showcasing how relying solely on a limited number of tweets for psychological profiling can lead to misinterpretations. The user’s nickname, profile description, and selected tweets illustrate this point. Particularly, the user’s profile statement, ’We can make it right $\sim$,’ embodies a profound sense of optimism, powerfully expressing the user’s positive attitude and belief in the face of life’s challenges. The profile summary, in its concise yet potent form, not only outlines the user’s basic profile but, in this case, subtly reveals their inherent resilience and determination to improve their circumstances. This further adds to the complexity of understanding their psychological state.

   For instance, the tweet, ’After two days and nights of excruciating pain, my little angel was born – a testament to beauty blossoming amidst agony,’ reveals that the user is actually a new mother rejoicing over the arrival of her child after the ordeal of childbirth. Another tweet, ’With the deadline looming, burning the midnight oil seems inadequate compared to seasoned scholars’ composure, yet the view outside my window reminds me that amidst this rush, the beauty of life still deserves appreciation,” demonstrates that despite intense work pressure, the user finds solace in natural scenery, transforming stress into motivation. Terms like ’little angel’ and references to ’beauty’ convey an upbeat sentiment.

   Contrastingly, another tweet states, ’I fear I might be suffering from depression, plagued by inner turmoil all day, even entertaining strong suicidal thoughts. What should I do?’ This clearly expresses negative emotions and psychological distress. By comprehensively analyzing the content and varied emotional hues of this user’s tweets, our model not only detects direct cues like ’depression’ but also uncovers underlying depressive tendencies from seemingly positive expressions. Through integrating multi-faceted information, our model achieves a more precise assessment. This capability highlights the robustness of our model, which maintains a high accuracy in identifying depressive tendencies amidst the complexity and variability of social media text data.

3. 3.

   Case 3: In contrast to the cases presented in Table IV and Table V, Table VI showcases a user profile and tweet samples of an individual whom our model predicts to have a “normal” mental state. While one of the user’s tweets, ”I really can’t stand the sound of teeth grinding. It keeps me awake and now I’m hungry too,” hints at experiencing insomnia, particularly when coupled with the statistical feature that most of their tweets are posted late at night, this initial observation may suggest a preliminary suspicion of depressive tendencies.

   However, upon deeper analysis of subsequent tweets, we find that the user, in fact, harbors an optimistic outlook towards the future, filled with anticipation and vision for life. This stark contrast underscores the superficiality of initial judgments. This process further validates the robustness of our model—it refrains from fixating on singular pieces of information or momentary emotional expressions. Instead, it holistically evaluates the user’s diverse tweets across different times and contexts, conducting a meticulous, multidimensional assessment that significantly reduces the risk of misclassification. This example affirms the model’s precision and profound understanding in the realm of complex emotion recognition and mental health assessment.

Early studies, such as those by Choudhury and colleagues[12], employed an elaborate manual feature engineering process to extract features from Twitter users’ behavioral patterns, providing inputs for subsequent depression detection models. This approach relies on researchers’ prior knowledge of depressive manifestations to manually design features that reflect depressive tendencies. Li et al.[31] utilized sentiment analysis techniques, in conjunction with word embedding methods, to assess the depressive tendency of tweets. This underscores the importance of textual content, particularly emotional expression, in identifying depressive states, further affirming the pivotal role of text-based features in online mental health analyses. Shen et al.[11], in their research, considered not only textual information but also integrated other data types (such as user behavior data), adopting multimodal methods to construct a more comprehensive user profile. This indicates that incorporating diverse information sources can effectively enhance the accuracy and robustness of depression detection.

Recent research has shifted towards employing deep learning, notably neural network models, for processing social media data and identifying depressive users. The pioneering work by Mustafa and colleagues[32] marks a transition in the field of online social network (OSN) depression detection, shifting from conventional machine learning to the deep learning paradigm. This transition showcases the capability of automatically learning complex and high-dimensional features, thereby enhancing both the efficiency and accuracy of detection.Lin et al.’s work[33] illustrates how cross-disciplinary technological innovation, specifically the integration of state-of-the-art NLP pre-traind models with image processing techniques, can advance mental health monitoring. They employed BERT (Bidirectional Encoder Representations from Transformers)[34], a groundbreaking pre-trained model, which through deep bidirectional context understanding, generates high-quality word embeddings (i.e., high-dimensional vectors) that capture the nuanced contextual meanings of words in sentences, vastly outperforming previous bag-of-words models or simplistic word embedding approaches.Specifically, Wang et al.[7] constructed a dataset, the Weibo User Depression Detection Dataset (WU3D), and employed the popular pre-trained model XLNet to extract text-based word features. They then devised a Deep Neural Network classification model, the Multimodal Feature Fusion Network (MFFN), which integrates features derived from diverse information sources, further accomplishing the classification task. Their approach demonstrated remarkable performance on the test dataset. The dataset utilized in this paper is the publicly available WU3D dataset provided by their team.