Entity-Level Sentiment: More than the Sum of Its Parts

Since our task is to annotate texts for sentiment towards individual volitional entities, we trained a dedicated named entity recognition (NER) model for Norwegian on the NorNE dataset (Jørgensen et al., 2020), but included only the PER and ORG labels (merging GPE-ORG with the ORG category).

All mentions of an entity were clustered through substring matching, to obtain a list of entities per documents and their mentions. If an entity "John Travolta" was mentioned in a text, the mentions "John" and "Travolta" would be clustered together with "John Travolta". There is in Norwegian little case inflection of proper nouns, besides genitive where the characters "’" and/or "s" are added. Our substring matcher would therefore check if stripping of "s" and "’" would give match. This way, "John’s" would be found to be a substring of "John". We found few clustering errors from this approach. One exception was that "Elisabeth I" was found to be a substring of "Elisabeth II", and the two we therefore clustered and counted as one entity in the text.

For each volitional entity in each document, the task of the annotators is to annotate sentiment at two different levels, as exemplified in Figure 1:

1. Document level: Based on a reading of the entire text with the given entity in mind, label the sentiment that the full text conveys towards that entity.

2. Sentiment-relevant references: For each sentiment-relevant sentence, identify the text span that either directly refers to the entity in question or indirectly contributes to the entity-directed sentiment through a specified semantic relation. The possible relations are, in order of priority:

   1. (a)

      Name mentions, e. g. "Jagger", "Rolling Stones".

   2. (b)

      Coreferences, e. g. "they", "the band".

   3. (c)

      Bridging references. In addition to coreference, we annotate anaphoric relations between entities that are not co-referent, so-called bridging relations. The inventory of relations was motivated by the pilot study described in Rønningstad et al. (2022) and included the relations "member_of", "has_member" and "created_by". When other bridging relations implied sentiment regarding a target, this was annotated under the subsequent point.

   4. (d)

      Whenever a sentence was considered to imply sentiment regarding an entity in any other way than the above mentioned, the entire sentence was labelled with sentiment, but no text span inside the sentence was identified.

The relation categories for the bridging relations were suggested to the annotators from our initial exploration of the data, and in annotation meetings it was established that these categories were relevant and sufficient for the dataset at hand. See the annotation guidelines’ list of terms in Appendix C for further description of coreferences and bridging references.

All sentiment annotations employ a five-category scale, similar to Dufraisse et al. (2023) and Bastan et al. (2020): "Negative–Standard", "Negative–Slight", "Neutral", "Positive–Slight", and "Positive–Standard". For the "Neutral" category, only name mentions are identified, since these could be added in the pre-processing. The other references to an entity were only annotated if they were non-neutral. Annotation was carried out using the Inception tool (Klie et al., 2018). Figure 3 in Appendix A shows example screenshots from the annotation process.

Our annotation guidelines are derived from those of NoReC_fine, which in turn build on the work of de Kauter et al. (2015). An English translation of the guidelines is presented in Appendix C, and we briefly present some of the most central considerations below.

The dataset was annotated by five paid NLP students at the BSc level. All are native Norwegian speakers between 20 and 35 years old. They underwent introductory training and test annotations in preparation for the project. During this introductory training, the annotators contributed towards refining the annotation scheme. All annotations were curated by the first author of this paper, as the project leader.

After manual cleaning, the pre-annotated volitional entities, 2481 documents based on 412 texts remained for further analysis. Final counts for the dataset are presented in Table 1. All annotators took part in the three phases of the project:

1. 1.

   Introductory parallel annotations and discussions. Annotators were initially provided with 75 documents, whereby 2–3 annotators would annotate the same texts. The annotators then inspected each others’ work, the guidelines were discussed and if neccessary adjusted.

2. 2.

   The entire training corpus annotated. The annotators subsequently annotated the 2481 document in the dataset, according to availability, one annotator per document. Each annotator annotated from 200 documents and upwards. The number of documents annotated by each annotator is shown in Table 3.

3. 3.

   Parallel annotation of the test set. Finally, all annotators annotated the test split, as pre-defined in NoReC_fine, in parallel. The test data contains 44 different texts, containing a total of 247 volitional entities. Each text contains on average 28.5 sentences.

4. 4.

   Curation The project leader reviewed all annotations in the dataset. For the training and development splits, there was one annotator to review. The annotations were corrected when neccessary. The amount of document-level annotations corrected by the curator, varied among the annotators from 0.5% to 8.2%. For the test split, all annotators annotated all instances. The curator inspected the majority vote before making the final judgement. The agreements here are shown in Tables 2 and 3.

We present here the annotator agreements, both for the overall sentiment per entity, and for the sentence-level annotations. For these analyses, we remove the intensity levels "Slight" and "Standard", and check for agreement only in terms of the main categories "Positive", "Neutral" and "Negative".

Table 4 shows the distribution of annotations in the dataset, across category and sentiment. It shows that the sentiment-relevant coreferences beyond name mentions are comparably few. In contrast, we find that the bridging relations (created_by, has_member and is_member) contribute quite significantly to entity-directed sentiment in our dataset. These relations constitute 34.8% of the annotated sentiments, almost equally frequent to the sentiment labels that are directly attached to an entity mention. These figures indicate that any approach that labels only sentences for sentiment regarding an entity if that entity is named in the sentence, appear to lose the majority of sentiment signal, which is found in sentences with other relations to the entity.

In this work we are specifically interested in the relations between the high-level and lower-level annotations for each entity. The availability of our dataset enables further analysis of these correlations at the per-entity level. In the following we will attempt to evaluate the effect of each category of sentiment-related sentences and how the aggregation of lower levels of sentiment classifications compare to the document-level score independently assigned by our annotators. We start by aggregating the sentiments for the name mentions only, before we add the remaining available annotations. When referring to the aggregated sentiment score, we here refer to averaging and rounding the non-neutral mentions, whereby we assign the "Standard" sentiments the value of $\pm$2, and slight sentiments are $\pm$1.

In our dataset, almost two thirds of the entities are mentioned only once by their name in a given text. If we, as suggested by Dufraisse et al. (2023) could assume that entities mentioned only once are neutral and not in focus, that would simplify the task considerably. For our dataset, Table 7 shows that although a majority of the entities with only one name mention are neutral, nearly half of the entities receiving a polarity are single mention entities. Discarding these would have meant discarding much valuable sentiment information, and we conclude that entities with one name mention only are worth keeping.

We extract a simplified dataset containing only the entity mentions and their sentence-level sentiments. We train a sequence labeler to identify entities and their three-class sentiment, with evaluation results shown in Table 8. The pretrained model applied was NorBERT3-large¹¹1https://huggingface.co/ltg/norbert3-large (Samuel et al., 2023). The models tested and search space for hyperparameters explored are shown in Table 11 in Appendix B.

As discussed in Section 4, the document-level sentiment can not be fully derived from the set of sentiments regarding each mention of an entity. However, we aggregate the predicted sentiments, similarly to how we aggregated the annotations for each entity mention in Section 4. This approach serves as a naive modelling baseline and an example of the limitations of this approach. The results from aggregating the modelled labels to the document-entity level are presented in Table 9. Table 6 shows that 82.9% of the entities in the training and development splits were correctly classified at the document level when aggregating the true sentiment labels for the entities’ mentions, and serve as an upper bound for this approach. Table 9 shows that when aggregating the predicted labels, 70.9% of the entities in the test split were correctly labeled with this baseline model.

Recent work indicates that ChatGPT and open-source counterparts may be a relevant resource for annotating and labeling English texts (Gilardi et al., 2023; Alizadeh et al., 2024). We therefore constructed a zero-shot dialogue with ChatGPT.

The prompts were what we consider clear and well-posed Norwegian questions about which of the three sentiment categories "Positive", "Neutral" or "Negative" is assigned to a given entity by the text. We performed the dialogue through the web interface with a paid monthly subscription to OpenAI, employing GPT v4 (Achiam et al., 2023).

The initial prompt was the entire text, preceded with this sentence in Norwegian: "In the subsequent text, is the sentiment towards "Kirsten Flagstad" Positive, Negative or Neutral?"
Where "Kirsten Flagstad" is the volitional entity in question. The prompt would be a lengthy answer including reasoning. The next prompt would be, translated: "Please give the answer with one word, Positive, Negative or Neutral". Table 10 shows that this zero-shot approach yielded an accuracy of 73.3%