MASIVE: Open-Ended Affective State Identification in English and Spanish

Our goal is to collect data representing the broad set of ways in which humans describe their feelings. We refer to these expressions as affective states VandenBos (2007); this is an umbrella term incorporating multiple kinds of feelings such as emotions and moods. We collect texts with expressions of affective states from Reddit²²2Using the PullPush API at https://pullpush.io/ using a bootstrapping procedure. Beginning with the adjective forms of the Ekman emotions, we search for texts containing forms of “I feel <affect> and…”, “I am feeling <affect> and…”, where <affect> is replaced with each emotion term. Notably, we also search for “I don’t feel <affect> and…” and “I am not feeling <affect> and…” to better capture the diversity of ways in which authors can express feelings. We extract affective state terms that follow the “and” from the retrieved posts to form a new set of search phrases with these terms. We repeat these steps, expanding the pool of query affective states in each round. Our primary assumption is that any adjective conjuncts of the query emotion term are also affective states, regardless of whether they are canonical emotion terms. For example, if "happy" was used to query the text "I feel happy and excited," the term "excited" is both an adjective and a conjunct; the same is true of “light” in “I feel happy and light”. In contrast, in "I feel happy and want to smile", "want" is a verb and would not be considered an affective state. We evaluate of this assumption in subsection 2.2.

In Spanish, we conduct the same procedure using forms of “Estoy <affect> y…”, “Me siento <affect> y…”, and “Estoy sintiendo <affect> y…”. We also seed the process with the most common Spanish translations of the Ekman emotions on Reddit (see Appendix B). Additionally, as Spanish includes both masculine and feminine forms for some terms, we search for both forms where applicable. Finally, we also collect a challenge set including affective state labels associated with regional Spanish varieties, hand-selected by a native Spanish-speaker, to evaluate models’ abilities to generalize to less-represented dialects (see Appendix B).

For both English and Spanish, we run 4 rounds of bootstrapping; for the regional Spanish terms, we run only a single round to avoid introducing non-regional terms. 15 affective states were randomly sampled from both datasets, and all posts containing those 15 affective states were reserved as part of each test set to evaluate models on unseen emotions. Summary statistics describing the English and Spanish splits as well as the regional Spanish challenge set are included in Table 1, and we include a Data Statement in Appendix A.

To validate the assumptions of our bootstrapping procedure and examine how affective states are used in our dataset, we collect human evaluations of the automatically identified affective states. Judgments are conducted by 2 native Spanish-speakers in Iberian Culture studies and 2 native English-speakers in Psychology for Spanish and English respectively. We randomly sample 250 texts from each language’s test set for evaluation such that 50 texts are shared by each pair. Annotators are provided with a full Reddit post with a single automatically-identified affective state highlighted. We ask annotators to judge the term in context on 3 dimensions, beginning with whether the highlighted term reflects an affective state. If a term is judged to reflect an affective state, annotators are asked to judge whether the highlighted term better reflects an emotion or a mood³³3We distinguish emotions – shorter-term feelings triggered by identifiable events – from moods – longer-term feelings not necessarily triggered by an event. and whether the highlighted term is used figuratively (e.g., "blue") or literally (e.g., "sad"). All 3 dimensions are judged on 4-point Likert scales where higher values mean the term primarily reflects an affective state, an emotion, and a literal usage, respectively. Annotators achieved moderate agreement in English ($\kappa=.51$) and substantial agreement in Spanish ($\kappa=.69$). Additional details concerning human annotations are included in Appendix B.

Additionally, we analyze 2 aspects of our dataset that differentiate it from prior emotion detection benchmarks. First, because Spanish is a language with grammatical gender for adjectives, part of the affective state prediction problem in MASIVE includes choosing whether to use the masculine or feminine form in the context of the input. Second, authors in natural settings may also tend to express their feelings by stating how they do not feel (e.g., “I’m not happy, but….”), and we specifically include negations to test models’ capability to contend with this construction in both English and Spanish.

The results of the aforementioned data annotations as well as the automatically extracted statistics are included in Table 2. Human annotation results are reported as the percentage of affective states within the sample; for negations and grammatical gender, we report the percentage of texts in our datasets that include any target negations or any feminine adjectives.⁴⁴4Recall that a single datapoint may have multiple labels joined by and. Large majorities (88% and 72% in English and Spanish respectively) of terms were judged to reflect affective states, validating the contents of MASIVE.

We additionally evaluate the performance of MASIVE-finetuned models on two previously published datasets in both English and Spanish. A key distinction from MASIVE is that these datasets feature limited label sets; we describe our evaluation procedures in subsection 3.2. In English, we evaluate on GoEmotions (Demszky et al., 2020), a commonly-used emotion dataset consisting of Reddit comments; it is originally labeled with 27 distinct emotion categories, though the authors also relabel the data with the Ekman basic emotions. We additionally evaluate on EmoEvent (Plaza del Arco et al., 2020), a dataset with both English and Spanish subsets of Tweets (among other languages) also labeled with the Ekman set.

Finally, we conduct 2 cross-lingual experiments expanding on prior work investigating the use of machine translation and high-resource language models for inference on lower-resource languages (Isbister et al., 2021; Kathunia et al., 2024). In contrast to prior findings, however, we hypothesize that neither translating the training nor evaluation data will be enable competitive performance with models trained on native data. First, using our natural test sets, we evaluate models finetuned on translated data. Second, we evaluate the performance of our native-trained models on translated data, mimicking the translation of lower-resource language data for inference with a model trained on a higher-resource language.

In both settings, we use bilingual Opus-MT models (Tiedemann and Thottingal, 2020) to independently translate the input documents and target affective state labels. We select Opus-MT models as they are accessible, open-source models, reflecting resources that may be used for large scale translation, and are utilized in experiments in Kathunia et al. (2024). Throughout the experiments, models finetuned on translated data are denoted _Tr. Test sets generated through machine translation are similarly denoted as En_Tr and Es_Tr.

We experiment with finetuning small language models on our original and machine-translated data. We also perform experiments with two Large Language Models (LLMs) in a zero-shot setting.

Table 3 presents the performance metrics for finetuned mT5, Llama-3, and Mixtral on our English and Spanish test sets, as well as finetuned T5 for the English test set only. Among multilingual models, mT5 outperforms both LLMs on top-k accuracy for both languages (Takeaway #1), despite having drastically fewer parameters.⁸⁸8Llama-3 occasionally refuses to make a prediction if the content discussed is sensitive (e.g., drug use). Results with invalid responses filtered out are included in Appendix E. Additionally, mT5 achieves the highest top-k similarity scores, except for top-1 similarity in Spanish. Between the LLMs, Mixtral tends to outperform Llama-3. This performance difference may be explained by the difference in size between models, as well as the fact that multilingual data was upsampled in Mixtral’s pretraining compared to prior models.

In English, the large variant of T5 has been shown to slightly outperform mT5 Xue et al. (2021b). While we find a similar difference, the performance gap is notably quite large. Because the remaining experiments include Spanish data, we focus on mT5. We note, however, that dedicated monolingual English models may offer significantly higher performance on ASI (Takeaway #2) and leave further exploration of the differences between monolingual and multilingual models to future work.

While the differences in language and content of the English and Spanish datasets prevent us from making conclusions concerning their relative difficulty, Table 3 also shows that performance in Spanish tends to be higher than in English, despite the better representation of English in pre-training and larger size of the collected English data compared to Spanish. This trend could be due to the larger set of unique affective states in our English data than Spanish, with more nuanced affective states that may be difficult for models to predict accurately.

To evaluate the generalized emotion detection capabilities afforded by finetuning on MASIVE, Table 4 shows the performance of mT5 finetuned on existing English and Spanish emotion benchmarks, both with and without prior finetuning on MASIVE. First, when used as a classifier, we find that mT5 finetuned on MASIVE first achieves a higher macro-F1 for all datasets. This suggests that finetuning on our corpus gives models generalizable knowledge of emotions (Takeaway #3). Because our corpora contain many more affective state labels than the evaluation datasets, models finetuned on MASIVE will include more nuanced terms than basic emotions in the top-k predictions. So, as expected, models finetuned only on the emotion benchmarks achieve higher top-k accuracy and similarity scores, as they are more likely to predict terms within the smaller label sets. The top-k similarity scores for our models, however, remain high, suggesting that the generated affective states are similar to the ground truth basic emotion labels.

To analyze how well models generalize beyond affective states explicitly included in finetuning, we present performance metrics on seen and unseen affective states in both languages in Table 6. In both languages, all models perform considerably better on affective states included in the finetuning data than on unseen affective states. T5^En , however, maintains better performance on unseen affective states than mT5^En, suggesting that monolingual models may better generalize.

In addition to unseen affective states, we present evaluation results on a subset of Spanish affective states which are region-specific in Table 6. Similarly to results on the full Spanish data, finetuned mT5^Es outperforms both LLMs in top-k accuracy and similarity. Notably, the performance of mT5^Es on this regional subset is comparable to its performance on general unseen Spanish emotions (Table 6), while Llama-3 and Mixtral, which are not explicitly finetuned on our corpora, perform significantly worse on the regional subset than they do on the Spanish data as a whole (Table 3). Because top-k accuracy drops significantly on unseen and region-specific affective states (top-k similarity as well, though less so), future work in this area should prioritize a generalized understanding of affective states, including regionalisms (Takeaway #4).

We break down the top-k accuracy and top-k similarity results for each model by grammatical gender and negations in Figure 2. We see again that mT5 outperforms both LLMs across all subsets, and that mT5 often places the gold label among the top 3 or 5 predictions if not the top 1. In particular, mT5^Es performs better on feminine adjectives than masculine adjectives or those with only a single form, and T5^En and mT5^Es perform better on negated targets than non-negated targets (mT5^En shows the same pattern for accuracy, though not similarity). Llama-3 and Mixtral achieve highest accuracy for masculine adjectives and highest similarity for single-form adjectives, while for negations, Llama-3 performs better on non-negations and Mixtral performs slightly better on negations. These results suggest that explicit training on MASIVE may improve performance specifically on unique features of generative ASI (Takeaway #5).

Finally, we evaluate the changes in performance first when using machine-translated finetuning data and alternatively when translating evaluation data in Table 7. First, we find an expected drop in performance when models are finetuned on machine-translated data for both English and Spanish. Interestingly, the drop in accuracy and similarity metrics (90% and 28%, respectively) in Spanish are notably larger than in English (77% and 14%). This could perhaps be explained by the translation model performing better in the Spanish to English direction than English to Spanish, as well as mT5’s ability to better generalize in English than in Spanish.

As an alternative approach to finetuning on translated data, we also consider the case where data may be translated at inference time. In these cases (En_Tr and Es_Tr in Table 7), we find that performance falls. Artifacts of machine translation have been found to impact evaluation of translation models (Freitag et al., 2020), and, similarly, errors and artifacts of unnatural translation may cause these changes in performance. In contrast to prior work suggesting that performance on the target data translated into English is comparable to finetuning on the target language for tasks such as sentiment detection, our results suggest that for our task, machine-translating the evaluation data leads to poorer performance, and translating either at training or inference time result in similar performance (Takeaway #6).