GPT Assisted Annotation of Rhetorical and Linguistic Features for Interpretable Propaganda Technique Detection in News Text.

The study of rhetoric has a rich history spanning thousands of years, traditionally falling within the purview of the humanities and relying on manual analysis methods. Given the limitations in the scalability of manual analysis, we naturally turn to exploiting machine learning approaches. In turn, the challenge for the machine learning community is a shortage of labelled data. Hence, most machine learning-based approaches have been executed on small data sets annotated with specific rhetorical figures, such as, for example, “chiasmus” (Dubremetz and Nivre, 2018), “litotes” (Mitrović et al., 2020), or “doubt” (Ward et al., 2022), to name a few.

Both propagandistic techniques and rhetorical devices can be considered tools of persuasion and have certain overlapping characteristics. Indeed, “the major techniques of propaganda are long known rhetorical techniques gone wrong” (Bryant, 1953). To our knowledge, the most relevant data set related to rhetoric in news text is the Propaganda Technique Classification (PTC) corpus developed by Da San Martino et al. (2019). The PTC corpus is annotated with 18 propaganda techniques divided into “logical fallacies” and “emotion and sentiment mechanisms”. These can be considered logos and pathos in the nomenclature of rhetoric.¹¹1Notably, “appeals to authority” is categorized under “logical fallacies”, whereas, in the rhetoric tradition, this would belong to a third category, namely “ethos”. Furthermore, some propaganda techniques, such as “doubt”, have equivalent counterparts in rhetorical devices. In contrast, others tend to have a broader scope, such as “flag-waving,” which includes appeals to nationalism as a means of persuasion.

We propose that propaganda techniques and rhetorical devices can be further deconstructed into grammatical and word choice components as developed by Fahnestock (2011) in Rhetorical Style: The Uses of Language in Persuasion. If true, these components, or “features” as we will refer to them from here on in, should correlate with propaganda techniques. However, to test this assumption, we need to be able to extract the features from natural language in order to use them for model training and inference. No such extraction procedure exists for these specific features. Thus, the focus of this study is on constructing such a procedure.

We were further constrained by a very limited budget. This is not an unusual position to be in - cost-saving techniques can be critical to a researcher’s ability to conduct their work. We propose an annotation strategy which takes advantage of GPT-4’s explanations to aid in the human annotation phase, where a small subset of the data is annotated, and utilizes an iterative prompt engineering approach to annotate the remaining data using GPT-3.5. We show that even noisy human-labeled data can be utilized to fine-tune GPT-3.5 to achieve the performance of its state-of-the-art but 10x more expensive counterpart, GPT-4. Our process is illustrated in Figure 1.

Even though our long-term goal is specific to propaganda detection, we believe that the features and annotation procedure we propose can be used in any NLP task that aims to leverage the characteristics of the language of persuasion. In addition, this effort could contribute to developing a linguistic rhetorical representation akin to existing representations such as Abstract Meaning Representation (AMR) or Rhetorical Structure Theory (RST).

Interestingly, both intra-GPT-3.5 and intra-GPT-4 consistency is higher than inter-annotator agreement, illustrated in Figure 5. Consistency alone, however, does not tell us anything about accuracy. To measure accuracy, we sample 30 examples in each feature where all three annotators agree on the same list of properties and manually produce ground truth labels for those examples. Human accuracy, excluding a few exceptions, ranges from 70-100%, whereas GPT-3.5 performs very poorly. GPT-4 performs five times better than GPT-3.5 but at ten times the cost. The results are shown in Table 4 in the V1 columns. In addition, reducing the temperature parameter²²2https://platform.openai.com/docs/api-reference/chat/create#chat-create-temperature significantly increases the consistency of GPT-3.5. This is as expected. Notably, consistency in the GPT responses is not always correlated with accuracy. Thus, simply taking a majority vote is not a useful strategy in our case. To improve accuracy, we perform the following experiments.

We aim to make the best of the noisy data obtained through human expert annotations by utilizing the state-of-the-art large language models, which have recently dominated NLP research. Identifying linguistic features and rhetorical devices in natural language is not trivial for humans, as is borne out of the low inter-annotator agreement on our task. However, it seems this task would be well suited to an LLM for the following reasons: it does not require reasoning, it does not require any highly specialized domain knowledge that wouldn’t have been available in the data the LLM was trained on, nor does it require access to facts or events which would have occurred post the training date. The one thing that LLMs are particularly good at, pattern recognition and general ‘knowledge’ (arising from the vast amounts of publicly available data they were trained on) would suggest that they are ideally suited to our task. This aligns with the generally accepted premise that people are good at logical reasoning while machines are good at finding patterns in large volumes of data.

Our research highlights another interesting, perhaps less conspicuous, difference between human annotators and GPT. What we found was that expert human inter-annotator agreement is low when it comes to non-trivial linguistic feature identification tasks. Conversely, GPT, when called with a low-temperature setting, achieves consistency levels around 80% (see Figure 5). However, when humans do agree, they are correct, overall, 83% of the time, while the maximum GPT accuracy we were able to achieve is only 56% (see Table 4). This suggests that we can confidently rely on the established majority voting mechanism, or ‘wisdom of the crowd’ approach, when it comes to traditional human annotations, but not when it comes to GPT.

This brings us to our next point, which is that every setting appears to demand its own exploration, and the techniques that work in one scenario do not always translate to another. In particular, we found that providing a small set of carefully crafted high-quality data points for fine-tuning does not necessarily increase the accuracy of GPT-3.5, as discussed in the previous section. In addition to different settings demanding different approaches, on a more granular level, not all features respond similarly to the same techniques. For example, ‘Mood’ and ‘Emphasis’ accuracy was highest using the original prompts, which performed poorly for all the other features. The ‘Aspect’, ‘New words’, and ‘Subject choices’ features are relatively easy for humans but particularly difficult for GPT. And fine-tuned GPT outperforms humans on ‘Phrases built on nouns’. These insights enable us to make cost/benefit decisions about utilizing human annotators vs GPT, prompting technique, or fine-tuning.

We have run our experiments on GPT-3.5 and GPT-4 for comparison purposes, and no doubt the next version of GPT will be even better. However, our goal is to find ways to optimize not only for accuracy but also for cost. Our fine-tuned (FT) models perform on par with GPT-4 at 10x less the price. Since we intend to annotate the PTC corpus, which comprises approx. 20,000 sentences, we can estimate that using our FT models would cost approx. $2,000⁴⁴4This is a rough estimate based on our spending to date. A more precise estimate can be calculated by estimating the number of tokens input and output to and from GPT., GPT-4 would cost $20,000, and human annotators would cost $90,000 (at $1.50 per sentence, per expert, using three experts). These orders of magnitude differences can be critical to a researcher’s budget and ability to proceed.

Since the release of ChatGPT in November 2022, researchers (Gilardi et al., 2023; He et al., 2023; Yu et al., 2023; Pangakis et al., 2023; Törnberg, 2023; Kuzman et al., 2023; Thapa et al., 2023; Ding et al., 2023; Bansal and Sharma, 2023; Feng and Narayanan, 2023; Alizadeh et al., 2023; Savelka, 2023; Huang et al., 2023; Tekumalla and Banda, 2023; Zhang et al., 2023) have explored using it as a text annotation tool to reduce reliance on expensive human annotated data for downstream NLP tasks. Various prompt engineering strategies have been proposed including zero-shot, few-shot, chain-of-thought (CoT), and active learning.

Wang et al. (2021) first explore using GPT-3 (the model upon which ChatGPT was built) for text annotation and show that data labelled by GPT-3 achieves the same performance on several NLP and NLU tasks as human labelled data at a fraction of the cost. Furthermore, models trained using the GPT-3 labelled data outperform GPT-3 on their respective tasks. However, the authors suggest that labeling by GPT-3 should only be used in low stakes scenarios due to its now well known tendencies to “hallucinate” (Ji et al., 2023), and/or perpetuate undesirable biases. Reiss (2023) also cautions against using ChatGPT-3.5 for zero-shot annotation tasks, pointing out the intra-coder inconsistencies (e.g. the inconsistencies among ChatGPT-3.5 responses given identical prompts). To improve the accuracy of ChatGPT annotations, Reiss (2023) recommends using majority voting based on three, or ideally more, repetitions of each single input. We show that this does not improve results in our setting. Taking this a step further, Davani et al. (2022) show that taking a majority vote before training is not as effective as training a multi-task model using all annotators (ChatGPT repetitions) and subsequently predicting the final label as an aggregation of each task output. Similar to Reiss (2023), Pangakis et al. (2023) point out that while “LLM performance for text annotation is promising, it is highly contingent on both the data set and the type of annotation task, which reinforces the necessity to validate on a task-by-task basis.” The authors propose an iterative strategy that evaluates the LLM annotations on a small subset of human expert annotated data until acceptable performance is reached. We follow a similar strategy.

Despite these potential pitfalls, others have shown promising results. Kuzman et al. (2023) compare a fine-tuned XLM-RoBERTa model against ChatGPT-3 for genre identification, a task deemed difficult for human annotators, where the inter-annotator agreement is low. While this is not strictly an annotation comparison, we can think of the GPT zero-shot classifier as an annotator where GPT achieved an accuracy score of 72% against the human-annotated test set. GPT outperformed the fine-tuned model by 5 points on an out-of-distribution data set given a similar task. In contrast, He et al. (2023) propose a method which surpasses human annotators on some tasks and reaches near parity on others. Unlike the zero-shot scenario, the proposed method involves few-shot chain-of-thought (CoT) (Wei et al., 2023) prompting. Uniquely, GPT-3.5 is used to generate the explanations, which are then used as few-shot examples in the final prompt. Having GPT generate the explanations is justified by attributing an emergent reasoning capability to GPT. In contrast, we provide expertly written definitions and real-world examples in our prompts. However, when we examine the GPT-3.5 explanations in our setting, we cannot identify any emergent reasoning capabilities - see section 3.1 for details.

Alizadeh et al. (2023) extend the research into LLM annotation beyond ChatGPT to open-source (OS) models. They show that while ChatGPT generally outperforms human annotators and OS models on most tasks, OS models outperform ChatGPT on some tasks and approach ChatGPT performance on others. OS models also outperform human annotators at a lower cost, with all the open-source benefits such as transparency, reproducibility, and data privacy.

Bansal and Sharma (2023) take an active learning approach, where the LLM is utilized to annotate the most informative inputs based on a novel conditional informativeness sampling strategy, thereby reducing the cost of annotation even further while improving the performance of downstream NLP models trained on the LLM annotated data.

It’s clear from the literature that the ideal prompt engineering methods and hyper-parameters vary across tasks, especially when those tasks are complex or particularly difficult for human annotators. Our work focuses on the scenario where the task is unique and complex, and human annotator agreement is low, and therefore merits its own study.

In this study, we have identified linguistic features based on the premise that propaganda techniques can be decomposed into rhetorical devices and linguistic characteristics, as seen in the use of language for persuasion. Given our limited budget, we employed three expert annotators to label 350 sentences from the PTC corpus, to the best of our knowledge, the most granular propaganda technique corpus available. With the help of our small human-annotated data set, we investigated whether a large language model like GPT can be utilized to annotate the remaining 2̃0,000 sentences. The impetus for our experiments and solutions is driven by an acutely limited budget, thus the solutions we propose are necessarily a trade-off between accuracy and cost.

In addition, we find certain unexpected differences between human annotators and GPT. In particular, we discovered that majority voting of GPT responses does not improve accuracy, and fine-tuning on a small but high-quality data-set is also not an effective strategy. Both of these findings are contrary to usual practices in machine learning, and we must emphasize again that each setting demands its own exploration.

Finally, we fine-tune GPT-3.5 using the human-annotated data set by generating examples for each feature/property combination by taking a majority vote for the positive examples and the absence of a given label from all annotators as negative examples. Even with this noisy data, we achieve an accuracy on par with GPT-4 at 10x less the cost. We make all our code, including our web-based annotation application, the properties and their definitions and examples, and the GPT prompts available in GitHub.⁵⁵5https://github.com/kyleiwaniec/rhetoric-annotation

Our biggest limitation is our budget. However, even without this constraint, the task is not trivial, even for expert annotators. Majority voting by itself still leads to a lot of noise, and we intend to perform several more iterations on the 350 sentences with our human annotators to reach a higher agreement. We performed our experiments on GPT as it is the subject of considerable attention. In the future, we plan to test other models. In particular, LLaMA (Touvron et al., 2023) and Mistral (Jiang et al., 2023) are open source and available in different sizes, making them especially suitable for analyzing the trade-off between cost and accuracy.

Furthermore, the feature set we created is based on literature, but this, too, could benefit from additional research. We have yet to see if these features improve the detection of propaganda techniques. While this is outside the scope of this paper, it is the next natural step. By making these available, we hope to aid other researchers in pursuing the goal of interpretable propaganda detection.