Aspect-Based Sentiment Analysis Techniques: A Comparative Study

Given the current state-of-the-art interest in Large Language Models (LLMs), we opted to include an LLM-based analysis in our comparative study. LLaMA 2 is a collection of second-generation open-source LLMs from Meta that comes with a commercial license. Roumeliotis et al. [23] presented that LLaMA 2 shows a significant leap forward in natural language understanding and generation, by its advanced architecture, large training data and refined training strategies. The architecture of LLaMA 2 is based on the transformer model, a neural network architecture that has proven highly effective in a wide range of NLP tasks. LLaMA 2 employs a multi-layered transformer architecture with self-attention mechanisms. It is designed to handle a wide range of natural language processing tasks, with models ranging in scale from 7 billion to 70 billion parameters.

Fine-tuning in machine learning is the process of adjusting the weights and parameters of a pre-trained model on new data to improve its performance on a specific task. There are three main fine-tuning methods in the context:

1. 1.

   Instruction Fine-Tunning (IFT): According to Peng et al. [24], IFT involves training the model using prompt completion pairs, showing desired responses to queries.

2. 2.

   Full Fine Tunning: Full fine-tuning involves updating all of the weights in a pre-trained model during training on a new dataset, allowing the model to adapt to a specific task.

3. 3.

   Parameter-Efficient Fine-Tunning (PEFT): Selectively updates a small set of parameters, making memory requirements more manageable. There are various ways of achieving Parameter efficient fine-tuning. Low-Rank Parameter (LoRA) [25] and Quantized Low-Ranking Adaptation (QLoRA) [26] are the most widely used and effective.

Traditional fine-tuning of pre-trained language models (PLMs) requires updating all of the model’s parameters, which is computationally expensive and requires massive amounts of data; thus making it challenging to attempt on consumer hardware due to inadequate VRAMs and computing. However, Parameter-Efficient Fine-Tuning (PEFT) works by only updating a small subset of the model’s most influential parameters, making it much more efficient. Four-bit quantization via QLoRA allows such efficient fine-tuning of huge LLM models on consumer hardware while retaining high performance. QLoRA quantizes a pre-trained language model to four bits and freezes the parameters. A small number of trainable Low-Rank Adapter layers are then added to the model. In our case, we created a 4-bit quantization with NF4-type configuration using BitsAndBytes¹¹1https://github.com/TimDettmers/bitsandbytes.

According to Dettmers et al. [26] under the model fine-tuning process, Supervised fine-tuning (SFT) is a key step in Reinforcement Learning from Human Feedback (RLHF). The SFT models come with tools to train language models using reinforcement learning, starting with supervised fine-tuning, then reward modelling, and finally, Proximal Policy Optimization (PPO). During this process, we provided the SFT trainer with the model, dataset, LoRA configuration, tokenizer, and training parameters.

To test the fine-tuned model, we used the Transformers text generation pipeline including the prompt. The LLaMA 2 model was fine-tuned using techniques such as QLoRA, PEFT, and SFT to overcome memory and computational limitations. By utilizing Hugging Face libraries such as transformers²²2https://huggingface.co/transformers/, accelerate³³3https://huggingface.co/accelerate/, peft⁴⁴4https://huggingface.co/peft/, trl⁵⁵5https://huggingface.co/trl/, and bitsandbytes, we were able to successfully fine-tune the 7B parameter LLaMA 2 model on a consumer GPU.

Few-shot learning has become increasingly essential in addressing label-scarce scenarios, where data annotation is often time-consuming and expensive. These methods aim to adapt pre-trained language models (PLMs) to specific downstream tasks using only a limited number of labelled training examples. One of the primary obstacles is the reliance on large-scale language models, which typically contain billions of parameters, demanding substantial computational resources and specialized infrastructure. Moreover, these methods frequently require manual crafting of prompts, introducing variability and complexity in the training process, thus restricting accessibility for researchers and practitioners.

In response to this, Tunstall et al. [27] proposed SETFIT (Sentence Transformer Fine-tuning) which presents an innovative framework for efficient and prompt-free few-shot fine-tuning of Sentence Transformers (ST). Diverging from existing methods, SETFIT does not necessitate manually crafted prompts and achieves high accuracy with significantly fewer parameters. The SETFIT approach consists of two main steps. In the first step, the ST is fine-tuned using a contrastive loss function, encouraging the model to learn discriminative representations of similar and dissimilar text pairs. In the second step, a simple classification head is trained on top of the fine-tuned ST to perform downstream tasks such as text classification or similarity ranking. By decoupling the fine-tuning and classification steps, SETFIT achieves high accuracy with orders of magnitude fewer parameters than existing methods, making it computationally efficient and scalable. In our study, we utilized several available sentence transformers through the SETFIT framework to obtain accuracies for aspect extraction and sentiment polarity identification.

Yang and Li [28] addressed the challenge of the lack of a unified framework for ABSA by developing PyABSA, an open-source ABSA framework. PyABSA integrates ATE and text classification functionalities alongside ASC within a modular architecture. This design facilitates adaptation to various ABSA subtasks and supports multilingual modelling and automated dataset annotation, thereby streamlining ABSA applications.

Moreover, PyABSA offers multi-task-based ATESC models, which are pipeline models capable of simultaneously performing ATE and ASC sub-tasks.To tackle the data shortage problem, PyABSA provides automated dataset annotation interfaces and manual dataset annotation tools, encouraging community participation in annotating and contributing custom datasets to the repository.

In our study, we utilized the PyABSA framework on the SemEval 2014 restaurant and laptop benchmark dataset to evaluate accuracy to be consistent with the practices followed in the literature as shown in Table I. Specifically, we employed the FAST_LSA_T_V2 model with PyABSA, which is included in the english checkpoint, to assess aspect extraction performance.

The first section of Table II shows the performance of Llama-2-7b [29] with QLoRA [26]. These performances were obtained using the L4 GPU. It can be noted that even though the sentiment polarity results are comparable between the two datasets, the aspect extraction on Res-14 is several magnitudes weaker than that of Lap-14.

In the second section of Table II, we provide a comprehensive overview of the accuracies attained by various sentence models using the SETFIT framework [27]. Due to the modular nature of the SETFIT framework, we could fine-tune and test combinations of models. If a model is reported in a single row, it means we have used the said sentence transformer model for both aspect extraction and sentiment polarity identification (eg, BGE [30]). In the cell blocks where a model is followed by other models with $+$ are combinations. For example, the first row of Paraphrase-MiniLM-L6-v2 [33, 32] contains results of that model being used both for aspect extraction and sentiment polarity identification. The subsequent line with +MpNet [34] indicates that Paraphrase-MiniLM-L6-v2 was used for the aspect extraction component and MpNet was used for the sentiment polarity identification component.

At this point, a question may be raised as to why would the aspect extraction have two different values for accuracy (79.80 vs. 85.40) in the two configurations if in both cases the same model (ie, Paraphrase-MiniLM-L6-v2 in this example) was used for that task. The reason is the fact that the fine-tuning is conducted end-to-end in a holistic manner and thus, the choice of the model used for the sentiment polarity identification ends up influencing the ultimate accuracy obtained by the aspect extraction component. It may enhance the result as in the case of Paraphrase-MiniLM-L6-v2 and MpNet. It may also hinder as in the case of LaBSE.

Overall, it can be noted that LaBSE [41] consistently emerges as a standout performer; either by itself or as the aspect extraction component of a pair. It can be argued that this robust performance is owed to its capability to capture nuanced complex information crucial for understanding both aspect-based sentiment analysis and sentiment polarity classification tasks. Specifically on the sentiment polarity classification task, it can be noted that Mpnet and RoBERTa-STSb-v2 [16] elevates performance multiple configurations. Further, the results also reveal domain-specific variations in model performance, as evidenced by the disparity between that Res-14 and Lap-14 results of SPECTER [36].

To give a better overview of how various models perform, we include 1(a) and 1(b) which visualize the SetFit results discussed in Table II. In 1(a), we present a detailed analysis of aspect extraction accuracy for various models. LaBSE emerging as the top performer across both datasets can easily be noted. It is also evident how ALBERT+DistilRoBERTa and LaBSE+RoBERTa-STSb closely follow with accuracies verging on 90%. The outlying low accuracies of BGE [30] and Sentence-T5 [31] are also evident. Similarly, in 1(b), we look into the analysis of sentiment polarity identification percentages for the same models and datasets. Here, LaBSE+RoBERTa-STSb shows the highest accuracy for Res-14 while LaBSE+MpNet shows the highest accuracy for Lap-14. These results reaffirm the effectiveness of LaBSE across both tasks and datasets. Conversely, models such as CLIP-ViT-B-32-multilingual-v1 [35] and SPECTER demonstrate relatively lower performances.

The third section of Table II reports the results obtained from the implementation of the FAST LSA [22] model on PyABSA [42]. PyABSA is also fine-tuned end-to-end similar to SETFIT [27]. However, unlike SETFIT, PyABSA does not report Aspect Extraction and Sentiment Polarity accuracies separately. It only gives an overall value. This is the reason for it having only one value per dataset in Table II. Alternatively, it is not wrong to take the reported accuracies as the values for the Sentiment Polarity task as it is the task that we have at the tail end of the pipe. If regarded in that perspective, it can be claimed that FAST LSA on PyABSA has the best results for Sentiment Polarity among all the model combinations and configurations tested by this study. Hence we have opted to highlight those results in bold as we did for the best results in the second (SETFIT) section.