Enhancing Emotion Prediction in News Headlines: Insights from ChatGPT and Seq2Seq Models for Free-Text Generation

We want to explore if we can harness the rich emotional context of the free-text explanation of the emotional reactions when in practice, at inference time we will only have access to the news headlines. Therefore, we turn to generating these free-text emotion explanations from the headlines. Text generation is a natural language processing task that received increasing attention with the appearance of Transformer architecture models including GPT variations Brown et al. (2020) and T5 Raffel et al. (2020). In recent years, text generation has shown promising results in numerous fields of application Li et al. (2021) including machine translation Vaswani et al. (2017); Yang et al. (2020), text summarization Guan et al. (2021), and medical applications Amin-Nejad et al. (2020). More recently, the GPT-based generative chatbot, ChatGPT, gained spotlight as it demonstrated high quality performance at question answering and text generation, including news article summarization Wang et al. (2023). In our study, we aim to explore the use of these large pretrained language models to generate free-text emotion explanations based on headlines and see whether they can help with emotion classification.

For our study of predicting the emotion from news consumers after reading news headlines, we used the BU-NEmo⁺ Gao et al. (2022) dataset which contains 1297 headline and lead news image pairings for gun violence news. Starting with the Gun Violence Frame Corpus (GVFC) dataset which applies framing predictions on news headlines and their lead images related to gun violence Liu et al. (2019a); Akyürek et al. (2020); Tourni et al. (2021), BU-NEmo Reardon et al. (2022) extended upon it by collecting emotion annotations from a crowdsourcing experiment on Amazon Mechanical Turk (MTurk) for 320 gun violence related news headlines. There are 10 emotion annotations for each headline and each annotation consists of 1) an emotion label from the 8 categories (Amusement, Awe, Contentment, Excitement, Fear, Sadness, Anger, and Disgust) Mikels et al. (2005); 2) the intensity of the emotion on a scale of 1-5; and 3) a short free text explaining why the annotator feels the selected emotion, as shown in Figure 1. The experiment was repeated for 3 modalities: Text only (T), Image only (I), and Text+Image (TI), respectively corresponding to when the annotators were presented with only the headline, only the image, and the headline together with the image. BU-NEmo⁺ expanded the original BU-NEmo dataset to include emotion annotations for 1297 headlines. We only used the Text only modality of the data for our study.

For our analysis, we used the full BU-NEmo⁺ and its subset, BU-NEmo⁺-CR, containing 365 news items that show a clear emotional agreement. Gao et al. (2022) highlighted that people have varying emotion reactions towards the same news content in the BU-NEmo⁺ dataset, and it is a challenging task to select a single ground truth emotion label for a given news item. This adds a significant layer of complexity and variance to the emotion prediction task with single label classification.

Moreover, due to the nature of the gun violence news Guo et al. (2021, ), the dataset is imbalanced towards the negative emotions. Therefore, it is important to take the majority baseline into consideration during evaluations of the models’ performances.

The BU-NEmo⁺ study used headlines as the input to the emotion classification model. However, relying solely on the headline comes with the disadvantage that headlines often carry limited information about the news. Furthermore, predicting the emotional reaction from headlines is especially difficult since the headline text does not usually contain the sentiment information of the emotional reaction, as shown by examples in Figure 2. To tackle this problem, we turn to the emotion explanations. For the first example headline that is about blocking 3D-printed gun blueprints (Figure 2), the emotion explanations justifying why annotators feel a certain emotion towards the news are:

{spverbatim}

1. for now i am content but intrigued to see the next steps 2. the distribution of 3D-printed gun blueprints should not happen 3. I think the judge did a right thing; that is good of the judge 4. Seems like a strange call, but not one that I have enough knowledge to properly have an opinion on 5. I’m a fan of strict gun control 6. It seems like a fair solution for the time 7. I can’t believe that there is even a discussion about distributing blueprints to 3D-print guns 8. It is already far too easy to gain access to guns in this country 9. I am confused as to why this would happen 10. because why are there printed gun blue prints in the first place

It is easy to observe that the emotion explanation has the advantage of containing the desired emotional information that is easier to classify than the headline.

We compiled the emotion explanations from BU-NEmo⁺ and refer to this dataset of free-text emotion explanations and their emotion labels as EE (Emotion Explanations). We also created another dataset by concatenating all 10 emotion explanations in BU-NEmo⁺-CR for each news which we refer to as the CEE (Concatenated Emotion Explanations). With the constructed corpus and the emotion labels, we trained a RoBERTa Liu et al. (2019b) classification model¹¹1https://huggingface.co/roberta-base. We refer to this model as CEE. We followed the setup from BU-NEmo⁺’s experiments Gao et al. (2022) and split the data into training / validation / test sets in a ratio of 0.5:0.25:0.25. For a fair comparison, we also included the single label emotion classification with only the headline text, a model which we refer to as Headline, and compared it against our results.

Furthermore, due to the diversifying nature of the emotional reaction, in addition to the exact match accuracy for the dominant emotion for a given news headline, we included the top-2 emotion accuracy. We calculated the top-2 emotion accuracy by checking whether the ground truth dominant emotion is the same as either one of the top-2 predicted emotions.

In our experiment, we found that over 74% of news headlines in the dataset exhibit a second most dominant emotion with a frequency of at least 2 (out of 10 annotations for each headline). As a result, we believe that measuring the accuracy of the top-2 emotions is a more representative metric than the exact match accuracy, capturing the diverse nature of emotional responses.

However, we would like to emphasize that this task of predicting emotions from explanations is easier since at test time, the CEE model is evaluated on concatenated human-generated free-text emotion explanations. For the rest of the experiments in our study, at inference time we will only have access to the headlines. CEE is our proof-of-concept of whether free-text emotion explanations are helpful in emotion classification. Therefore, the performance of CEE serves as an upper bound in our study.

As shown in the results section 3.1., using free-text emotion explanations produces a significant improvement in emotion classification than just relying on headlines. This finding led us to explore whether we can further harness the emotion explanations to predict people’s emotional reactions towards news content.

In practice, at inference time, we wanted to utilize the rich emotional context in the emotion explanations but we will only have access to headlines. Therefore, we constructed a seq2seq (sequence-to-sequence) architecture to map a headline to a collection of emotion explanations. The ground truth emotion explanations that the headlines map to were the CEE corpus we constructed. We trained a sequence-to-sequence transformer Vaswani et al. (2017) model and fed the generated emotion explanations into the RoBERTa classification model trained on the BU-NEmo⁺-CR for emotion classification (shown in Figure 3). We call this model CEE-T (Transformer-generated Concatenated Emotion Explanations). We use the task of generating free-text emotion explanation to draw a parallel to the intermediate reasoning steps that humans perform when conducting a task.

We constructed a model utilizing a seq2seq architecture to generate a collection of emotional explanations that corresponded to headlines in order to leverage the emotional cues presented in the emotional explanations. However, due to the limited size of our training dataset (CEE) for this task, there were limitations in generating free-text emotional explanations that mimic human writing in terms of sentence completeness (as shown in the example in Figure 3).

Therefore, instead of directly generating free-text emotional explanations and using the generated explanations to predict emotions, we treated the free-text emotion explanation generation task as an intermediate or related task for emotion prediction and performed intermediate-task transfer learning Phang et al. (2019). Implementation wise, this means fine-tuning a pre-trained T5 Raffel et al. (2020) model²²2https://huggingface.co/google-t5/t5-base on our headline-to-explanation corpus (CEE), and subsequently fine-tuning the same model for the headline-to-emotion prediction task as shown in Figure 4. The proposed approach does not rely on emotion explanations as direct inputs for predicting emotions. However, through intermediate-task training process, the model parameters can be updated to learn and capture the relationship between the headlines and the emotion explanations written by humans. This learned representation can then be transferred to the target task of emotion prediction, making the training process more effective.

In this section, we describe a way in which emotion explanations generated by the Large Language Model ChatGPT (GPT-4 based) for a news headline can be utilized for emotion prediction. We experimented with different prompting methods (zero-shot and few-shot) for emotion explanation generations with ChatGPT. We also evaluated ChatGPT for a dominant emotion prediction task (Baseline 1) by asking ChatGPT to directly generate the top-2 dominant emotions that readers might feel after reading the headlines using the following prompt:

For each of the headlines below, what are the top 2 dominant emotions (must choose from these 8 emotions: Amusement, Awe, Contentment, Excitement, Fear, Sadness, Anger, and Disgust) that the reader might feel after reading the headlines? The dominant emotion is the emotion most commonly experienced among/by the majority of population/individuals who read the headline. Please only return a list of 91 dominant emotions tuples that the readers might feel corresponding to each of the headlines. For each tuple in the list, the first element should be the most likely dominant emotion and the second element should be the second most likely dominant emotion. Here are the headlines: […]

For the single label emotion classification, the majority baseline which represents the percentage of the dominant class for this dataset is 0.42. As shown from the test time performances in Table 1, the models using emotion explanations show significant performance improvement in both the exact match and top-2 emotions accuracies for the task of single label emotion classification. The CEE outperformed the Headline model by 10 percent points for the exact match accuracy and 13 percent points for the top-2 accuracy.

As mentioned in section 2.2., the task of the CEE model here is intrinsically easier and its performance serves as an upper bound when we have human-generated emotion explanations for each headline at inference time. We are more interested in the real-world scenario where at inference time we only have the headlines to predict emotions. Indeed, the main focus of our study is our novel approach in using generated free-text emotion explanations to enhance the task of emotion prediction from headlines.

As shown in Table 2, intermediate-task training on headline-to-explanation generation task boosts the performance of the emotion classification task by 6 percent points. While this T5 model with transfer learning does not surpass the exact match accuracy of 0.58 achieved by Headline model (Table 1), the results demonstrate that leveraging emotion explanations could enhance the emotion classification performance compared to using the headlines alone. Moreover, the findings show positive transferability between emotion explanation generation and emotion classification tasks.

As shown in Table 3, the emotion classification pipelines using ChatGPT to generate explanations (EE-Chat, CEE-Chat) significantly outperform training our own seq2seq explanation-generation transformer (CEE-T).

When using emotion explanations generated by ChatGPT as inputs for emotion classification, we observe in Table 3 that almost all models that utilize these generated emotion explanations (zero- and few-shot) outperform Baseline 1 (i.e., ChatGPT’s own direct prediction of dominant emotions). Specifically, our best zero-shot model CEE-Chat outperformed Baseline 1 in exact match accuracy by 20 percent points and top-2 accuracy by 13 percent points.

We also compared the performances with the more complex ChatGPT baseline, (i.e., Baseline 2, where we prompt ChatGPT to generate emotion explanations and emotion predictions at once). Our models that utilize ChatGPT-generated explanations consistently outperformed Baseline 2 in the zero-shot setting. However, there was no distinguishable improvement in either of the few-shot approaches.

In addition, we observed that models with few-shot generations of emotion explanations perform noticeably better than the ones with zero-shot generations in both exact match and top 2-accuracies. Between the 2 few-shot approaches, the frame ignorant approach yields better emotion classification performance. We suspect that this is due to the class imbalance of the frames in the original dataset.

Additionally, models trained and evaluated on the Concatenated Emotion Explanations in CEE show better exact match performances than models trained and evaluated on individual Emotion Explanations in EE. However, the performance of EE slightly outperformed that of CEE in the top-2 accuracies few-shot setting (Table 3).

We also conducted McNemar’s test McNemar (1947), a statistical hypothesis test used to analyze the significance of differences between paired categorical data. It assesses whether the differences are due to random variation or indicative of a real effect (P-value ¡ 0.05). We performed McNemar’s test on differences between emotion outputs of models built with only headlines and outputs of models built with ChatGPT-generated emotion explanations. Among the results, model trained with few-shot generated Concatenated Emotion Explanations, without frame information (i.e., the CEE-Chat model) significantly outperforms the prediction of the Headline model (P-value = 0.013). In addition, in terms of the exact match accuracy, the model trained with zero-shot individual Emotion Explanations generation (i.e., the EE-Chat model), shows significant improvement over the Headline model predictions as well (P-value = 0.024). Both notable boosts prove that GPT-generated explanations are valuable for explaining human’s emotion and can act as a temporary replacement for human-generated contents when resources are limited.

Overall, our best performing models: CEE-Chat and EE-Chat with the frame ignorant few-shot ChatGPT generations achieve a top-2 accuracy of 0.85 and an exact match accuracy of 0.66, reaching a similar performance as our upper bound (CEE) while significantly outperforming emotion classification with the headline-only (Headline) model from Table 1. This demonstrates the potential of leveraging large language models like ChatGPT in low-resource scenarios where there may not be enough headline and human-generated emotion explanation pairs to train models that mimic human-generated emotion explanations.

Table 4 provides qualitative examples where our best-performing CEE-Chat model correctly predicts the emotions, whereas the Headline model fails. In the first example, the headline insinuates a sense of heightened tension. This potentially misleads the Headline model since it exclusively relies on the headline’s textual content. In contrast, the CEE-Chat model’s generated emotion explanations offer deeper context, encompassing a broader emotional landscape and thereby ensuring a more holistic understanding. Moreover, by examining the generated emotion explanations, we obtain better insights into why the model predicts a certain emotion. This offers more interpretability of our emotion prediction model, which is an area that many of the neural classification models lack Zafar et al. (2021); Shen et al. (2022). Among the various ways to leverage this interpretability, one is to check for potential bias or controversies within the model by directly inspecting the generated emotion explanations. The second example in Table 4 serves as a notable illustration for this. The mention of Trump, a polarizing figure, can introduce biases as reflected in some the generated explanations. These biases could have steered the model to predict negative emotions such as anger. However, the overall dominant emotion (e.g., contentment) in the generated explanations overrides these minor biases and guides the model to predict the correct emotion, demonstrating the model’s robustness towards controversial headlines.

In addition, to understand the emotion distribution alignment between human and ChatGPT emotions, we computed the average KL divergence across all test headlines between the human-annotated emotion distribution and the ChatGPT generated emotion distribution (from Baseline 2). As shown in Table 5, providing ChatGPT with few-shot examples improves the emotion distribution alignment. Specifically, the frame ignorant few-shot generations are best aligned with our human emotion distribution. This finding aligns with our result in Table 3 which shows that the frame ignorant few-shot generations are best for emotion classification.

We also plotted confusion matrices to examine the distribution of predicted emotion labels for each true emotion label (Figure 8). When comparing the confusion matrices of Baseline 1 Zero-shot and Baseline 1 Few-shot, we observe that few-shot examples led ChatGPT to generate a wider range of emotions. In the absence of few-shot examples, ChatGPT predominantly generated negative emotions for almost all news headlines related to gun violence. Another notable finding is that ChatGPT struggles to distinguish between anger and disgust. While humans feel different emotions based on the subtle nuances in specific news headlines, ChatGPT falls short in this aspect. We also observe that headlines labeled as Awe was not accurately predicted for any of the news headlines. This discrepancy may arise from different areas of focus between humans and ChatGPT when interpreting the news headlines. For instance, when presented with the headline ”Doctors share gun stories, demand action after NRA tells them to ‘stay in their lane’”, majority of annotators felt awe towards doctors advocating for the right cause according to our emotion explanations written by human annotators. In contrast, ChatGPT responded with anger, perceiving the NRA’s action as disregarding the expertise of doctors.