Deep Content Understanding Toward Entity and Aspect Target Sentiment Analysis on Foundation Models

Sentiment Analysis (SA) is the field of Natural Language Processing (NLP) that aims to extract and analyze sentiments, opinions, attitudes and emotions expressed towards certain entities from a textual data. Traditionally, SA focused on detecting the overall polarity e.g., positive, negative or neutral. As such, it brings limited amount of information, often insufficient for most of the real world applications (Liu, 2012). Hence, in the past decade, the research recognized fine-grained SA frequently named as Aspect Based Sentiment Analysis (ABSA) (Pontiki et al., 2014, 2015, 2016) whose task fundamentally consists of detecting two components which are targets and its corresponding sentiments. According to Zhang et al. (2022) the target can be described as either aspect category or aspect term, sentiment can be described as sentiment polarity that often attaches by opinion term. Depending on the motives, not all authors aim to extract these elements simultaneously, but they would rather focus on identifying specific subgroups and the relations among its elements which are consequently leading to the creation of various so called ABSA subtasks.

However, the transformer architectures (Vaswani et al., 2017) with pre-trained knowledge (aka Foundation Models (FMs)) influence most of the NLP tasks to not require a specific architecture to train from scratch on a large corpus of data, but rather the best performing solutions rely on adaptation of FMs (Bommasani et al., 2022) for downstream tasks. Generally, Large Language Models (LLMs) can achieve high performance on a downstream task relying only on their emergent abilities (Zhao et al., 2023), without their weights being updated. Such adaption for a downstream task is called zero- or few-shot learning (Brown et al., 2020) toward the text generation approaches (aka Generative AI or GenAI). In recent years, GenAI including models like Llama2 (Touvron et al., 2023), Llama3 ¹¹1https://github.com/meta-llama/llama3, Mistral (Jiang et al., 2023), Mixtral (Jiang et al., 2024). These models have gained the popularity in research areas and industries that revolutionized NLP applications by enabling high performance on various tasks through optimized prompting techniques rather than extensive fine-tuning. These models are pre-trained on vast and diverse datasets and can efficiently handle both simple and complex NLP tasks by simply refining prompts. This approach simplifies deployment and enhances adaptability, making instruct models particularly effective for tasks such as classification, text generation, summarization, and translation tasks.

In this work, we propose a fine-grainer detection of aspect category into what we called entity (i.e., meal) and aspect (i.e., freshness) and focused on sentiment polarity revealed based on detected entity and aspect as a triplet output. This introduces a new ABSA subtask called “Entity Aspect Sentiment Triplet Extraction (EASTE)” task. We benchmark our experimentation settings on the ABSA subtask on the dataset published in a shared task; SemEval16 (Pontiki et al., 2016) which is a dataset based on restaurant reviews in English. Our work also tackle these challenges following EASTE settings in different appraoches: 1) a token classification task like Named-entity recognition (NER) by proposing a unified-loss solution based on BERT model (Devlin et al., 2018) with full fine-tuning, 2) zero/few shot learning and fine-tuning text generation models and 3) model alignments via parameter efficient fine-tuning such as LoRA (Hu et al., 2021) and Prefix-tuning (Li & Liang, 2021b) to observe the impacts and performances on complex tasks such as EASTE on instruct-tuned encoder-decoder LLMs.

We position these contributions on multiple fields.

• •

  Creating an EASTE, a novel ABSA task that profoundly targets entity, aspect elements for comprehensive analysis of sentiment.

• •

  Proposing a novel unified-loss approach to solve EASTE task via token classification that is suitable for multi-class classification.

• •

  Deeply exploring various techniques to adapt Language Models for a complex downstream task and achieving a State-of-the-Art (SoTA) results.

In this section, we highlight relevant research that shares the SoTA target sentiment analysis work. Regarding the ABSA subtasks which have been introduced as a task for SemEval14 (Pontiki et al., 2014) and continued to be present on both SemEval15 (Pontiki et al., 2015) and SemEval16 (Pontiki et al., 2016) competitions, most of complex SA tasks were introduced as compound ABSA subtasks (Zhang et al., 2022) which aims to extract simultaneously multiple elements – pairs, triplets or quadruplets from a given sentence or a text. Among for our research relevant tasks such as Aspect Category Sentiment Analysis (ACSA), Aspect Sentiment Triplet Extraction (ASTE), and Aspect Sentiment Quad Prediction (ASQP), we highlight Target-Aspect-Sentiment Detection (TASD).

Target-Aspect-Sentiment Detection (TASD). TASD aims to extract target, aspect, sentiment triplets. Target Aspect Sentiment Detection was introduced by Wan et al. (2020) and for a review sentence $S$ aims to extract all $(t,a,s)$ triplets:

where $t_{i}$ stands for target and is a subsequence of $S$ if explicit and $NULL$ if implicit, $a_{i}\in\{a_{1},...,a_{n}\}$ stands for aspect and $s_{i}\in\{positive,negative,neutral\}$ for sentiment polarity. For example, in the restaurant review sentence ’The food arrived 20 minutes after I called, cold and soggy.’, output of TASD should be {(NULL, SERVICE#GENERAL, negative), (food, FOOD#QUALITY, negative)}

To our knowledge, Brun & Nikoulina (2018) and Wan et al. (2020) have addressed this problem using available parsers and domain-specific semantic lexicons and pre-trained BERT-based architecture respectively. In additional, the work of Zhang et al. (2021) defines the TASD as sequence-to-sequence learning problem and solves it in a generative manner using encoder-decoder T5 architecture (Raffel et al., 2019).

Pontiki et al. (2015) define an entity e, for instance in the domain of restaurants, as either the reviewed entity itself (restaurant), or another relevant entities directly related to it (e.g. food, service, ambience), while the aspect a is a particular attribute (e.g., quality, price, general) of the entity in question. The sentiment polarity s is defined as a polarity of the opinion(s) expressed towards the target. All three pieces of information are extracted from a predefined set of entities $e\in\{e_{1},...,e_{n}\}$ and aspects $a\in\{a_{1},...,a_{n}\}$ for every specific domain of interest, while sentiment polarity $s\in\{positive,negative,neutral\}$ remains the same per domain. Wan et al. (2020) combine what Pontiki et al. (2015) defined as entity and aspect into one element as “aspect”, taking into account only e and a combinations that appear in the analyzed datasets, which might not cover all true possible combinations.

Therefore, we follow the definitions by Pontiki et al. (2015) and define new ABSA subtask which considers entity and aspect as two different elements. Hence, for a given sentence $S$ extracts all (e, a, s) triplets but taking into account that each of the triplets relates to the corresponding target:

where $t_{i}$ stands for target and is a subsequence of $S$ if explicit and $NULL$ if implicit, $e_{i}\in\{e_{1},...,e_{n}\}$ stands for entity, $a_{i}\in\{a_{1},...,a_{n}\}$ stands for aspect and $s_{i}\in\{positive,negative,netural\}$ for sentiment polarity. If we take the same restaurant sentence ’The food arrived 20 minutes after I called, cold and soggy.’, output of EASTE should be {(NULL, SERVICE, GENERAL, negative), (food, FOOD, QUALITY, negative)}. Hence, we break the aspect into two elements as “entity” and “aspect” where each of them comes from different, independent dataset and is combined after its individual prediction in what Wan et al. (2020) define as aspect. This scenario of extraction creates one more level of ABSA task complexity.

Our work relies on the latest benchmark in this category which is SemEval16 (Pontiki et al., 2016). We adopt the review sentences for restaurants domain. For each given sentence $(e,a,s)$ triplets are annotated where $e\in\{e_{1},e_{2},…e_{n}\}$ stands for entity, $a\in\{a_{1},a_{2},…a_{n}\}$ for aspect category, and $s\in\{positive,negative,neutral\}$ for sentiment. $e$, $a$ and $s$ are chosen from different predefined inventories. To each of the triplets, a target term is attached towards which the opinion is expressed. If it is expressed explicitly, the target term is a subset of the words of corresponding review sentence, and NULL if implicitly. The dataset contains 2000 review sentences for train set, and 676 review sentences for test set.

In this section, we discuss our experimental settings to explore EASTE task while the token classification tasks were run locally using Apple CPU and MPS devices. For text generative tasks via LLMs like Llama2, Llama3, and Mixtral models, we use API calls for model inference hosted via IBM watsonx.ai²²2https://www.ibm.com/products/watsonx-ai.

The main indicator of model performance in our experimentation is F1 score. For each of methodology settings, we defined a way to calculate F1 score in order to get comparable results. In general, the (e, a, s) triplets are considered predicted correctly if and only if the prediction is attached to the correct corresponding target term.

Setting 1: As the target term is composed of either one or multiple words if the opinion expressed explicitly towards entity, we consider prediction correct only if 50% or more tokens of the target term are predicted correctly. On the other hand, if the opinion is expressed implicitly we expect model to attach (e, a, s) triplet to cls token.

Settings 2 and 3: We consider prediction correct only if (e, a, s) triplet is generated attached to correct corresponding target term if opinion expressed explicitly towards an entity, otherwise we expect a target term to be predicted as ’NULL’.

Overall, our best performing model is fully fine-tuned Flan-T5-XL achieving F1 score of 76.38 based on proposed EASTE experimental settings. However, a few-shot prompted on recently released LLMs such as Llama2, Llama3, and Mixtral obtain outstanding results on recall, but considerably lower on precision which impact the final F1-score. Additionally, Mixtral performs as second ranked, up close by Llama3 with 70.98 and 70.36 F1 scores respectively. On the other hand, prefix-tuning yielded competitive results where Flan-UL2 emerged as a top-performing model, achieving a F1 score of 65.28. Our exploration of LoRA tuning did not provide the big gap over prefix-tuning, with the best scoring model being Flan-T5-XXL with the F1 score of 69.87. However, the full-tuned encoder models for token classification using unified-loss approach tends to provide reasonably good results due to joint layers giving the high complexity of the prediction. On the other hand, in Table 2 when we cast the unified-loss approach to evaluate in TASD settings where entity and aspect are merged as a single element, it shows that there is a significant improvement up to almost 10% in the classification report. Ultimately, if we split an evaluation category into individual inspection of Entity, Aspect, Sentiment like in Table 3, we can observe that the unified-loss approach using BERT is well performed on pre-defined entity detection with F1 score of 72.39 in token classification settings where aspect and sentiment remains a challenging detection process.

This research introduces EASTE, a novel and complex task for detecting ABSA settings through various NLP techniques. Our results underscore the critical importance of selecting appropriate fine-tuning techniques and prompting strategies tailored to the size and type of LLMs. By employing diverse approaches such as token classification, text generation, and fine-tuning alignments, we assessed model performance across different architectures and sizes. Our findings demonstrate the efficacy of the proposed unified-loss approach in token classification, particularly for less complex tasks like TASD. Additionally, the potential of text generative models combined with advanced prompting strategies is evident. We also highlight the necessity of choosing suitable fine-tuning techniques and model architectures based on task complexity and available resources. This research offers significant contributions to the field of sentiment analysis, providing deeper insights into sophisticated NLP techniques and models for complex sentiment analysis tasks. Our work advances the understanding of deep content analysis for sentiment detection and sets a foundation for future explorations in the realm of natural language processing.

We would like to thank IBM Client Engineering EMEA for providing the GPU infrastructure that made our research possible. Their support enabled us to carry out our experiments efficiently and effectively. The computational resources provided by IBM were instrumental in advancing our work, especially in model alignments and the diverse fine-tuning approaches utilized in this study.