Dynamic Sentiment Analysis with Local Large Language Models using Majority Voting:
A Study on Factors Affecting Restaurant Evaluation

Previous studies on sentiment analysis mainly focused on electronic word-of-mouth (eWOM) and social network posts [15]. These methods can be broadly classified into four groups: rule-based models, machine learning, deep neural networks (DNNs), and LLMs.

With the rise of online AI chat services, running LLMs on local devices, such as laptop computers and smartphones, has been explored [60]. As mentioned previously, LLMs are assumed to be executed in abundant environments. In other words, the major challenge is the limited computing resources. One approach for addressing this issue is to apply quantization [61, 62, 63, 64, 65]. Quantization is referred to as "to map floating-point numbers into low-bit integers" [66], which discretizes the continuous values of the parameters on the LLMs to compress the model size and memory usage and accelerate its execution [67].

Quantization can be divided into two approaches: quantization-aware training (QAT) and post-training quantization (PTQ). QAT designs the learning process based on the assumption of quantization. The inference accuracy tends to increase since the model can be adapted for quantization during the early stages of training. However, the quantization is required to be considered during the training process, needing a large amount of resources and expertise compared to PTQ [68], which applies quantization after training. On the other hand, PTQ applies quantization after the training. In PTQ, a smaller amount of data is required for the calibration of the model, and its implementation is more effective than that of QAT [66]. In this study, we focused on PTQ—a widely used technique for quantization owing to its low training costs.

As previously mentioned, quantization generally limits the parameter $w\in\mathbb{R}$ to an integer. For example, $b$-bit quantization ($b\in\mathbb{N}$) uses the map function $\phi:(\mathbb{R},\mathbb{N})\to[0,2^{b})$ to obtain the quantized parameter $\hat{w}$ [67]. The actual PTQ algorithm has variants such as LLM.int8() [61], GPTQ [62], SpQR [69], AWQ [64], and GGUF [65]. Whichever technique is used, precision of quantization, specifically the number of bits in quantization, should be carefully considered because it generally degrades the performance of the quantized model. In other words, the performance of the model and the precision of the quantization have a tradeoff relationship [13, 70].

In summary, although methods to execute LLMs on resource-constrained devices have been explored, there is still a lack of verification of points such as the relationship between precision and accuracy. Additionally, it is difficult for industries to comprehensively evaluate an optimal model with a suitable balance between execution speed and accuracy in marketing analysis. Therefore, to understand this relationship, we construct AbSA models using pre-trained LLMs that have different numbers of parameters and are quantized with different precisions. Upon selecting the pretrained LLMs, we mainly focused on 4-bit quantization since several studies [68, 71] have shown that 4-bit quantization can perform close to the nonquantized model. Needless to say, it has also been pointed out that the performance of LLMs is highly dependent on the task [66].

In terms of pretrained models, we employ instruction-tuned models that require no additional training. In instruction tuning [72], the model is trained in advance using a combination of various instructions and their expected responses to adapt to a wide range of tasks. In this study, Llama provided by Meta [73] is adopted, which is an open-source LLM permitted for both commercial and academic use only if several conditions are fulfilled. According to the model card [74], latest Llama 3 is trained with more than 15 trillion tokens from publicly available data sources. Furthermore, the fine-tuning process of the model includes more than 10,000 manually annotated data in addition to instructional data, as well as reinforcement learning from human feedback (RLHF) [75]. Considering these learning processes, sufficient accuracy is expected for sentiment analysis of eWOM without additional training.

We focused on three factors: model scale (i.e., number of parameters), precision for quantization, and model architecture. First, in terms of model scale, the latest Llama 3 has two variants: 8 billion (8B) and 70 billion (70B). Second, in terms of precision, we primarily used 4-bit models with additional 3-bit and 5-bit models. Finally, the Llama 2 model was used to validate the impact of the model architecture and pretraining process. Table 1 lists the employed models. We compared the performance and duration of each model in practical marketing research tasks. For quantization, the GGUF format [65] is employed, which has been widely adopted within the LLM community because of its high practicality.

In addition, instead of implementing fine tuning, we employ one-shot learning. We included a single annotated example in the prompt and enabled the model to generate an accurate response. The samples used for the instructions were randomly extracted from the dataset, annotated by the authors, and excluded from the test data. Since Llama 3 has a large context window of 8192 tokens, it is sufficient to wrap up the entire text, including the prompt, one-shot example, and review texts. To extract the polarity value as structured data, the model is explicitly required to output text in the JSON format. Once obtained, the text is parsed into tabular data.

In the field of machine learning, ensemble learning is often employed to reduce the error rate and to generalize results. It is a combination of predictions by multiple models and has been shown to be robust to noise and outliers [76]. The utility of learning has been widely confirmed (cf. review articles and books [76, 77, 78]), particularly when models have different types of prediction errors. In other words, the proposed model is effective when the responses exhibit moderate diversity. In ensemble learning, majority voting is utilized particularly for multinomial classification, in which each model votes for one class, and the class with the most votes is considered as the final predictor [77]. While ordinary majority voting used the average evaluation value, LLMs may occasionally produce unexpected values such as outliers or missing values. Therefore, in this study, we adopt the median value for the metric of majority voting.

To implement this mechanism, the reproducibility parameters of the model were used. In most machine learning methods, by specifying the initial value of the random number generation (i.e., seed value), it is possible to ensure reproducibility in training and inference. In other words, by repeating annotations with different seed values, the model can generate the results obtained by multiple fictitious workers. Based on the responses of each virtual worker, we create two variables: 1) $m^{k}_{w}$, an indicator of whether dimension $k$ is mentioned in worker $w$, and 2) $v^{k}_{w}$, an indicator of how high the sentiment is if mentioned (where dimension $k=\{1,2,...,14\}$ and worker $w=\{1,2,...,5\}$).

First, for each worker $w$,

	$\displaystyle m^{k}_{w}$	$\displaystyle=\begin{cases}1&\text{(if~{}}k\text{~{}is mentioned)}\\ 0&\text{(otherwise)}\end{cases}$		(1)
	$\displaystyle v^{k}_{w}$	$\displaystyle=\begin{cases}v&\text{(if~{}}k\text{~{}is mentioned)}\\ 0&\text{(otherwise)}\end{cases}$		(2)

are obtained (sentiment $v=\{1,2,...,5\}$). Second, voting was conducted using the median as follows:

	$\displaystyle m^{k}$	$\displaystyle=median(m^{k}_{w}|~{}w=1,2,3,4,5)$		(3)
	$\displaystyle v^{k}$	$\displaystyle=\begin{cases}median(v^{k}_{w}\mid v^{k}_{w}\neq 0)&\text{(if }\exists~{}w\text{ s.t. }s^{k}_{w}\neq 0\text{)}\\ 0&\text{(otherwise)}\end{cases}$		(4)

This implies that $v^{k}$, the median of the sentiment values for the dimension $k$, is calculated only among nonzero $v^{k}_{i}$. Finally, the voted evaluation $s^{k}$ was obtained as follows:

$\displaystyle s^{k}=m^{k}v^{k}$

(5)

Thus, this mechanism employs two-stage majority voting to assess the level of sentiment on the dimension $k$: 1) determining whether the dimension is mentioned or not, and 2) evaluating the sentiment polarity, utilizing the median for both stages.

In addition, the randomness of the response of LLMs is determined using a temperature parameter. One study [9] suggests 0.2 and above for temperature while another study [79] pointed out that the result becomes highly random and not consistent with higher temperature. Therefore, we use a temperature value of 0.2 for a moderate randomness.

In this study, we use the Yelp Open Dataset [80], an open dataset publicly available for academic use. Yelp is an online platform on which users post evaluations and reviews about various facilities, including restaurants, stores, and public institutions. Some studies [54, 81, 82, 83] have utilized it to analyze user reviews. The data contain the ratings $nStars_{ij}$ and review texts $review_{ij}$ that user $i$ posts on facility $j$ (where $i=1,2,\dots,n$; $j=1,2,\dots,m$). One post was randomly extracted from each user. If a user posted a review for the same restaurant multiple times, only the most recent review was considered for sampling. Therefore, the sample size matches the number of target users extracted.

In addition, each establishment has category tags, such as Restaurant, Coffee & Tea, and Bar; thus, we can extract the target instances by choosing the tags. In this study, only restaurants holding a physical store in a fixed address were included in the analysis, and therefore we extracted only those tagged with Restaurant and excluded those of Fast Food, Food Trucks, Nightlife, and Bar. We sample 1000 instances from the data, and Table 2 lists the summary statistics. The numbers of characters and tokens are counted using the Tiktoken tokenizer [84] which is also used in Llama 3.

First, we predict the overall rating (i.e., the number of stars given) of restaurants in Study 1. By comparing the accuracy and processing speed of predictions among multiple pre-training and reference models using machine learning methods, we explore the best-balanced model that optimizes both prediction accuracy and processing speed. In the analysis, annotation was performed using a 5-point Likert scale, in accordance with the fact that the overall rating by actual users ranged from one to five stars. In addition to overall sentiment, 14 dimensions were simultaneously predicted for Study 2 (cf. Table 3, in Study 2).

Eight reference models were established to evaluate the model performance. First, we construct three DNN models: feed-forward neural network (FFNN), bidirectional LSTM (Bi-LSTM), and convolutional neural network (CNN). The FFNN model uses BERT (pre-trained model: bert-base-uncased [45]) for text vectorization, which has the 712-dimensional pooled output of the [CLS] token from BERT and is passed through three fully connected layers and classified using a softmax function. The Bi-LSTM model is constructed in accordance with the previous study [85]. We used word2vec for the vectorization and combined Bi-LSTM and multi-head attention, which addresses the long-term dependencies of the context and add weighted importance to the relevant information. ³³3The actual previous study [85] grouped the target variable into positive, neutral, and negative. The CNN model is established with the previous study [37] which used FastText for obtaining word embedding. The model alternates between convolutional layers and max-pooling layers applied to the 2D feature representations, followed by fully connected layers with ReLU activation, and finally, classification with a softmax function. Second, we used three rule-based methods, including VADER, SO-CAL, and TextBlob. These methods are based on pre-determined dictionaries and rules, so there are no adjustable parameters. Our predictions are generated directly from these polarity scores without any additional training⁴⁴4The obtained continuous values were adjusted to a range in $[1,5]$.. Third, for machine learning methods, we use Linear SVM. Since machine learning methods cannot directly handle the textual modality, we vectorize the text using TF-IDF in accordance with the previous study [34]. To create sufficiently sized features for prediction with TF-IDF, we set the dimensionality to 4000. Finally, in addition to these well-known models, a benchmark at chance level is created. Instead of generating completely random predictions, we calculated the proportion of sentiment labels from the training data and used it as weights to generate predictions for the test data. Regarding the additional dataset for training and validation, we randomly sampled 5000 observations for training and 1000 for validation set, ensuring no duplication among the datasets.

The results are shown in Table 4. The proposed models (Model 1–4) except Llama-2 (Model 5) outperformed the reference models. Notably, Llama 3 with 70B parameters (Model 1) demonstrated the best performance across all metrics. In terms of the number of parameters, the model with 70B in 4-bit (Model 1) is superior to that with 8B in 4-bit (Model 3), indicating that a larger-scale model achieves better performance within the same precision of 4-bit. In other words, the scale of the model generally contributes on the prediction accuracy as well as previous studies have shown [86, 87]. Second, in terms of precision in quantization for 8B models (Models 2–4), in contrast to expectations, the accuracy improved with lower precision. Third, as all Llama 3 models (Models 1–4) outperformed the Llama 2 model (Model 5), it can be concluded that the improved model structure and pre-training process contributed to the predictions.

Regarding the reference models, performance was in the following order: DNN models (Models 6–8), rule-based models (Models 9–11), machine learning models (Model 12), and chance level. Although the model using DNN achieved high accuracy as expected, it is noteworthy that most of them did not surpass that of Llama 2. Additionally, among the DNN models, the highest accuracy was shown when BERT was used for acquiring word embeddings (Model 6). Although Model 7 employs more complex architectures of LSTM and Multihead Attention, Model 6 with FFNN was superior in prediction. This confirms the significant improvement of BERT over word2vec and FastText. Second, all rule-based models (Models 9–11) showed better performance than the machine-learning model (Model 12). This result can be attributed to the relatively short length of the review texts and low text complexity. Alternatively, this could be due to the poor generalization performance of the machine-learning methods to imbalanced data, as restaurant ratings tend to gather at extreme values, such as 1 or 5. However, all the models scored above the chance level.

Finally, a comparison of the processing times among the LLMs showed that only Model 1, with 70B parameters, required a significantly longer processing time. Compared with Model 2, Even when analyzing only 1,000 samples, a total durations differ in more than 16 hours for a 2.4% improvement in the prediction error. From a practical standpoint, this increase in the processing time for improvement of the accuracy cannot be considered a reasonable trade-off. It is true that Model 2 has a lower accuracy than Model 1; however, its RMSE of 0.551 indicates that the model still effectively predicts sentiment. It accurately estimates higher values as high and lower values as low, with the fastest processing speed among the proposed LLMs. Therefore, this study employs Llama 3 with 8B parameters and quantized in 3-bit (Model 2) for Study 2 and 3.

Thus far, we found the best model for sentiment analysis to assess overall ratings in Study 1. However, since evaluating the detailed aspects is significantly more difficult than evaluating the overall rating, ambiguity of the evaluation occurs during the annotation. The fact that such fluctuations were resolved by voting in the actual annotation is also an important clue for this study. Therefore, in Study 2, we incorporated majority voting by virtually generating five annotators in one LLM and examined whether the robustness of the results increased depending on the introduction of voting. As explained, the mechanism operates in two stages with $m^{k}$ (the presence or absence of the mention of dimension $k$), $v^{k}$ (the ratings of the dimension $k$), and $s^{k}$ (the final evaluation for dimension $k$), based on the median of the five workers (see Section 3.2 for details).

Table 5 presents an example of the aggregated evaluations of the scores of five virtual workers for one review. First, as shown in the table, consistency among the workers was confirmed for most dimensions. Second, some of the evaluations were divided. For example, in $menu$, one worker rated it as 2, while the remaining four workers rated it as 3; hence, the voted sentiment was 3. In $congestion$, most of the workers evaluated that the dimension was not mentioned, whereas worker 4 evaluated it as 1. Thus, the final sentiment was set to 0, indicating that this aspect was not mentioned.

Based on the above, Table 6 shows a lift in the performance by incorporating the mechanism. The results indicate that that incorporating our proposed mechanism improved performance across all indices, even exceeding the average. This tendency is similar to ensemble learning which uses multiple machine-learning models to perform inference tasks and aggregates the results through statistics such as mean and median.

Furthermore, the difference of accuracy improvements between Study 1 and 2 is noteworthy. In Study 1, a fourteen-fold increase in the processing time resulted in improvement of 2.4%. In contrast, in Study 2, a five-fold increase in the processing time led to that of 3.9% using majority voting. This indicates that a medium-sized model with iterative inferences using the majority voting mechanism is significantly more efficient in terms of both training time and prediction accuracy than the larger model with a single inference.

Thus far, we have validated the sentiment analysis using LLMs (Study 1) and the robustness of the results using the majority voting mechanism (Study 2). In other words, we can freely extract any aspect of sentiment from review texts accurately with medium-scale LLMs using the majority voting mechanism. Finally, in Study 3, to demonstrate that further analyses are possible with the annotated data, we use regression analysis to examine how each aspect affects the overall evaluation. Since the dataset did not contain the sentiment values of each aspect, we assess the generalizability of the results from the marketing literature. For a similar analysis, a previous study [22] which examined the relationship between the emotion of the posts on the social network and the total amount of monthly donation to the university constructed linear regression models based on predicted sentiments.

We use a generalized linear model (GLM) to predict the overall rating of restaurants based on aspect-based sentiments (Table 3) as explanatory variables. In this process, we construct two GLMs with different target variables: one using the actual user rating and the other using the rating predicted by the LLM. By comparing the estimated parameters and other values of the two models, we verify that they have similar structures. Since there is a risk of multicollinearity if all the dimensions considered in this study are simultaneously used in the model, explanatory variables are selected based on Bayesian information criterion (BIC) to explore the best model.

With variable selection, seven dimensions were adopted in addition to the intercept. Table 7 shows the parameter estimations for the actual user ratings ($Y1$, left) and LLM-predicted ratings ($Y2$, right). First, for Y1, the results confirmed statistically significant differences at the 1% level for all explanatory variables. In particular, the $z$ values show that $dishes$ and $enjoyment$ have a strong positive effect and $congestion$ has a strong negative effect. These are assessed as valid results, as several studies reported a similar tendency for the effects, such as "food quality" [88, 89, 90], "entertainment" [91], and "waiting time for a meal" [90, 92], on outcomes such as perceived value, customer satisfaction, and behavioral intention. In addition to Y1, the statistical significances at the 1% level are also confirmed for all the variables in Y2. The signs of the estimated coefficients and $z$ values are similar to those in Y1 and the $t$ tests for all explanatory variables using the coefficients and standard errors reveal no significant differences between the models (cf. Diff in Table 7).

In summary, in Study 3, there is no significant discrepancy between the predicted and actual values of the overall evaluation, and that further analysis can be implemented using aspect-based sentiments. This indicates that the proposed model can accurately assess the impact of any dimensions on the overall evaluation. In other words, points that have not been fully evaluated in previous studies can be further analyzed by extracting more detailed perspectives from a large amount of review data. For example, in the previous study [91], the effect of the entertainment factor on the evaluation was investigated as the free provision of board games at a café. By using the proposed model, we can freely define and quantify the entertainment factor and then analyze its relationship to the overall rating.