ConText at WASSA 2024 Empathy and Personality Shared Task:
History-Dependent Embedding Utterance Representations
for Empathy and Emotion Prediction in Conversations

Given dyadic conversations, the tasks consist in:

• •

  Track CONV-turn: Modelling empathy, emotion polarity and emotion intensity of each utterance in a conversation. Each utterance in a conversation was annotated with these 3 traits, on a scale or real numbers from 0 to 5.

• •

  Track CONV-dialog: Modelling perceived counterparty empathy of each interlocutor in a conversation. Each interlocutor of a dyadic conversation rated the perceived counterparty empathy, on a scale of integers from 1 to 7.

Embeddings from Pre-trained Language Models (PLMs) are the most commonly used state-of-the-art approaches in these tasks. RoBERTa (Liu et al., 2019) is a PLM suceeding from BERT (Devlin et al., 2019), pre-trained to perform language modeling to learn deep contextual embeddings, i.e., vectors representing the semantics of each word or sequence of words. DeBERTa (He et al., 2020) differentiates from the previous PLMs by introducing disentangled attention and an enhanced mask decoder. Longformer (Beltagy et al., 2020) was conceived for tackling long texts, using modified attention mechanisms, acting on both local and global scale.

We now describe how we obtain embedding representations with the PLM. These processes are depicted in Figure 1. For each track we leverage different representations:

• •

  Track CONV-turn: we feed as input to the PLM the utterance we intend to classify, $u_{i}$, concatenated with its conversational context corresponding to a number $c$ of previous utterances in the conversation, $(u_{i-1},u_{i-2},...,u_{i-c})$. Concretely, we feed $u_{i}$ to the model, preceded by the [CLS] token and suceded by the [SEP] token, followed by the previous turns $u_{i-1}$ up to $u_{i-c}$, separated by the [SEP] token. An utterance consists in a sequence of $w_{it}$ tokens representing its $T_{i}$ words:

  $$u_{i}=(w_{i1},w_{i2},...,w_{iT_{i}})$$

  (1)

  The motivation behind feeding previous context turns lies within the fact that empathy and emotion are deeply context-dependent. Similar to human judgement in which these traits are better evaluated given the conversational context, the language model also benefits from this knowledge.

• •

  Track CONV-dialog: we use the same backbone architecture but instead of feeding a certain number of previous utterances of the conversation, we feed all the utterances and by the order of which they were written. We also add as input in the beginning a token corresponding to the interlocutor for which we are predicting the trait.

  Since we are predicting an interlocutor trait from a dialog, it is our contention that feeding all that dialog utterances provides the language model with the most complete information and that adding a token identifying the interlocutor for which we are predicting the trait makes the model establish a distinction between the interlocutors, which is necessary since the same dialog is used twice to predict both interlocutors’ trait.

From the obtained embeddings we can extract a suitable representation for the sentence. Choosing all tokens from all layers would yield an extremely memory demanding classification layer and may not yield the best model performance. Thus we choose the first embedding from the last layer L, the [CLS], as in Equation 2:

The regression module that follows the PLM is a linear fully connected layer, applying a linear transformation to the pooled encoder output data, changing the dimension of this data from the PLM hidden size to 1:

We then minimize the Mean Squared Error (MSE) loss between the predicted value and the gold label:

The models used are RoBERTa, DeBERTa and Longformer, all in both base and large versions from the Transformers library by Hugging Face Wolf et al. (2020). The Adam Kingma and Ba (2014) optimizer is used with an initial learning rate of 1e-5 and 5e-5, for the encoder and the regression head, respectively with a layer-wise decay rate of 0.95 after each training epoch for the encoder. The encoder is frozen for the first epoch. The batch size is set to 4. Gradient clipping is set to 1.0. As stopping criteria, early stopping is used to terminate training if there is no decrease after 5 consecutive epochs on the validation set over MSE, for a maximum of 40 epochs. The checkpoint used for obtaining the results on the test set is the one that achieves the lowest MSE on the validation set. When running training to determine which backbone model for each trait or number of context-turns on the CONV-turn track for each trait should be used for training the final model, we use the provided validation set as test set and perform a 90:10 split of the provided training set into training and validation sets.

Our code is publicly available¹¹1https://github.com/patricia-pereira/wassa-sharedtask.

The shared task dataset consists in empathic reactions to news stories and associated conversations, containing dyadic conversations in reaction to news articles where there is harm to a person, group, or other Omitaomu et al. (2022). These conversations are turn-level annotated in perceived empathy, emotion polarity, and emotion intensity, and dialogue-level annotated in terms of perceived counterparty empathy.

Given a conversation, the data is processed and fed to the model to train as depicted in Table 1:

We now report the results of our approach for Track CONV-turn on the validation set. For representative purposes we only report the backbone model that yielded the best results, RoBERTa-large.

The lowest performance in all traits is the one obtained without introducing any context turns, highlighting the importance of considering context. The general tendency is for the performance to increase with the progressive increase of the number of context turns, up to a performance peak which is trait specific, and then for it to decrease. For the purpose of the shared task only one random seed was used to generate results, but we use 5 random seeds in our previous work Pereira et al. (2023) to validate this tendency. The peak of performance is obtained with 6 context turns for Emotion Polarity, 2 turns for Emotion Intensity and 4 turns for Empathy.

We now report the results of our approach for Track CONV-turn on the test set and compare it with the results of other teams.

Our team ranked first place amongst nine teams, with an average Pearson score of 0.626. Some teams scored Pearson scores very close to ours while some teams scored Pearson scores much lower and even negative. This may indicate a very diverse set of approaches resulting in different scores.

Regarding results for the track CONV-dialog on the validation set, the backbone model which yielded the best result, a Pearson correlation score of 0.3416, was RoBERTa-base. This model has a max token length of 512 and we truncate the input to respect that limit. As we feed all the conversation to the model, which usually exceeds 512 tokens, it could be expected that a backbone model such as the Longformer which has a max token length of 4016 would yield better results. These results could indicate that it is not necessary to feed all the conversation to evaluate perceived counterparty empathy.

We now report the results of our approach for Track CONV-dialog on the test set and compare it with the results of other teams.

We achieved a Pearson score of 0.191, just 0.002 below the top ranking submission, placing our team second in the ranking.

The result on the test set was notably lower than the result on the validation set. This can be due to the different distributions of the sets but also due to the fact that with this small dataset, the provided validation set and the validation set for choosing the model when performing the 90:10 split on the provided training set are not large enough to be representative.