Is Sarcasm Detection A Step-by-Step Reasoning Process in Large Language Models?

Inspired by the step-by-step thinking ability of humans, CoT prompting was proposed to “prompt” language models to produce intermediate reasoning steps that lead to the final answer. Wei et al. Wei et al. (2022) made a formal definition of CoT prompting in LLMs and proved its effectiveness by presenting empirical evaluations on arithmetic reasoning benchmarks. This work pioneered the use of CoT prompting in NLP. However, its performance hinged on the quality of manually crafted prompts, which was a costly and unstable process. To fill this gap, Auto-CoT was proposed to automatically construct demonstrations with questions and reasoning chains Zhang et al. (2022). Different from Auto-CoT, Diao et al. Diao et al. (2023) presented an Active-Prompt approach to determine which questions were the most important and helpful to annotate from a pool of task-specific queries, for reducing the human engineering workload. The impressive results of CoT prompting have sparked a surge of exploration into designing CoT prompting strategies across various tasks Fei et al. (2023); Li et al. (2023); Zheng et al. (2023). For instance, Wang et al. Wang et al. (2024) used formal grammars as the intermediate reasoning steps for domain-specific language generation.

Furthermore, Yao et al. Yao et al. (2024) introduced a non-chain prompting framework, namely ToT, which made LLMs consider multiple different reasoning paths and self-evaluated choices to decide the next course of action. They proved the effectiveness of the ToT approach on the tasks requiring non-trivial planning or search. Beyond CoT and ToT approaches, Besta et al. Besta et al. (2024) modeled the information generated by an LLM as an arbitrary graph (i.e., GoT), where units of information were considered as vertices and the dependencies between these vertices were edges. Although the above-mentioned approaches have shown exceptional performance on various arithmetic and logical reasoning tasks, all of them adopt the sequential decoding paradigm of “let LLMs think step by step”. Contrarily, it is argued that sarcasm judgment does not conform to step-by-step logical reasoning, and there is a need to develop non-sequential prompting approaches.

Sarcasm detection is habitually treated as a text classification task, where the target is to identify whether the given text is sarcastic or not Zhang et al. (2024). It has evolved from early rule based and statistical learning based approaches to traditional neural methods, such as CNN, RNN, and further advanced to modern neural methods epitomized by Transformer models. In early stage, the rule based approaches infer the overall sarcasm polarity based on the refined sarcasm rules, such as the occurrence of the interjection word Zhang et al. (2023a). Statistical learning based approaches mainly employ statistical learning techniques, e.g., SVM, RF, NB, etc., to extract patterns and relationships within the data Zhou et al. (2023).

As deep learning based architectures have shown the superiority over statistical learning, numerous base neural networks, e.g., such as CNN Jain et al. (2020), LSTM Ghosh et al. (2018), GCN Liang et al. (2022), etc., have been predominantly utilized during the middle stage of sarcasm detection research, aiming to learn and extract complex features in an end-to-end fashion. As the field of deep learning continues to evolve, sarcasm detection research has stepped into the era of pre-trained language models (PLMs). An increasing number of researchers are designing sophisticated PLM architectures to serve as encoders for obtaining effective text representations. For example, Liu et al. Liu et al. (2022) proposed a dual-channel framework by modeling both literal and implied sentiments separately. They also constructed two conflict prompts to elicit PLMs to generate the sarcasm polarity Liu et al. (2023b). Qiao et al. Qiao et al. (2023) presented a mutual-enhanced incongruity learning network to take advantage of the underlying consistency between the two modules to boost the performance. Tian et al. Tian et al. (2023) proposed a dynamic routing Transformer network to activate different routing transformer modules for modeling the dynamic mechanism in sarcasm detection.

However, the above-mentioned works still focus on how to utilize PLMs to extract effective features, without leveraging the extraodinary context learning capabilities of LLMs. In contrast, this paper makes the first attempt to explore the potential of prompting LLMs in sarcasm detection.

Consider a sarcasm detection task. Given the data set $\mathcal{D}=\left\{\left(\mathcal{X},\mathcal{Y}\right)\right\}$, where $\mathcal{X}=\{x_{1},x_{2},\ldots,x_{n}\}$ denotes the input text sequence and $\mathcal{Y}=\{y_{1},y_{2},\ldots,y_{n}\}$ denotes the output label sequence. We use $\mathcal{L}_{\theta}$ to represent a large language model with parameter $\theta$. Our task is to leverage a collection of cues $\mathcal{C}=\{c_{1},c_{2},...,c_{k}\}$ to bridge the input $\mathcal{X}$ and the output $\mathcal{Y}$, where each cue $c_{i}$ is a coherent language sequence related to linguistics, context, or emotion that serves as an intermediate indicator toward identifying sarcasm.

We capture the inherent paradoxical nature of sarcasm, which is the incongruity between the surface sentiment and the true intention, and introduce chain of contradiction, a CoT-style paradigm that allows LLMs to decompose the problem of sarcasm detection into intermediate steps and solve each before making decision (Fig. 2 (a)). Each cue $c_{k}\sim\mathcal{L}_{\theta}^{CoC}\left(c_{k}|\mathcal{X},c_{1},c_{2},...,c_{k-1}\right)$ is sampled sequentially, then the output $\mathcal{Y}\sim\mathcal{L}_{\theta}^{CoC}\left(\mathcal{Y}|\mathcal{X},c_{1},...,c_{k}\right)$. A specific instantiation of CoC involves three steps:

Step 1. We first ask LLM to detect the surface sentiment via the following prompt $p_{1}$:

Step 2. We then ask LLM to carefully discover the true intention via the following prompt $p_{2}$:

Step 3. We finally ask LLM to examine the consistency between surface sentiment and true intention and make the final prediction:

Notably, CoC is built based on the presumption that all the cues are linearly correlated, and detects human sarcasm through step-by-step reasoning. Different from the original CoT, however, the steps are explicitly designed for the sarcasm detection context. Further details are presented in Algorithm 1 in App. A.

The linear structure of CoC restricts it to a single path of reasoning. To fill this gap, we introduce graph of cues, a GoT-style paradigm that allows LLMs to flexibly choose and weigh multiple cues, unconstrained by the need for unique predecessor nodes (Fig. 2 (b)). GoC frames the problem of sarcasm detection as a search over a graph, and is formulated as a tuple $\left(\mathcal{M},\mathcal{G},\mathcal{E}\right)$, where $\mathcal{M}$ is the cue maker used to define what are the common cues, $\mathcal{G}$ is a graph of “sarcasm detection process”, $\mathcal{E}$ is cue evaluator used to determine which cues to keep selecting and in which order. Unlike ToT and GoT, GoC does not involve the modules of “thought generator” and “thought aggregation”.

1. Cue maker. Human sarcasm judgment often relies on the combination and analysis of one or more cues to achieve an accurate understanding. Such cues can be broadly categorized into three types: linguistic cues, contextual cues and emotional cues. Linguistic cues refer to the linguistic features inherent in the text, including keywords, rhetorical devices, punctuation and language style. Contextual cues refer to the environment and background of the text, including topic, cultural background, common knowledge. Emotional cues denote the emotional stance conveyed by the text, including emotional words, special symbols (such as emojis) and emotional contrasts. A total number of 4+3+3=10 cues are adopted.

2. Graph construction. In $\mathcal{G}=\left(V,E\right)$, the cues are regarded as vertices constituting the vertex set $V$, while the relations across cues form the edge set $E$. If there is an edge between cues $c_{k}$ and $c_{j}$, it is considered that $c_{k}$ and $c_{j}$ are closely related. Given the cue $c_{k}$, the cue evaluator $\mathcal{E}$ considers cue $c_{j}$ to provide the most complementary information to $c_{k}$, which would combine with $c_{k}$ to facilitate a deep understanding of sarcasm.

3. Cue evaluator. We involve $\mathcal{G}$ in the LLM detecting sarcasm process. To advance this process, the cue evaluator $\mathcal{E}$ assesses the current progress towards judging sarcasm by means of determining whether the cumulative cues obtained so far are sufficient to yield an accurate judgment. If so, the search goes to an end. Otherwise, it serves as a heuristic for the search algorithm, determining which additional cues to select and in what order, to further the detection process. Similar to ToT, an LLM is used as the cue evaluator $\mathcal{E}$.

We employ a voting strategy to determine the most valuable cue for selection, by explicitly comparing multiple potential cue candidates in a voting prompt, such as:

Although the GoC enables the exploration of many possible paths across the cue graph, its nature remains grounded in a step-by-step reasoning paradigm (see Algorithm 2 in App. A).

We further relax the assumption that the cues for sarcasm detection are inter-related. We introduce bagging of cues, an ensemble learning based paradigm that allows LLMs to independently consider varied combinations of cues without assuming a fixed order or dependency among them (Fig. 2 (c)).

BoC constructs a pool of the pre-defined $k=10$ cues $\mathcal{C}$. From this pool, $\mathcal{T}$ subsets are random sampled, each consisting of $q~{}\left(i.e.,1\leq q\leq k\right)$ cues. BoC thus leverages LLMs to generate $\mathcal{T}$ independent sarcasm predictions $\hat{y}_{t}$ based on the cues of each subset. Finally, such predictions are aggregated using a majority voting mechanism to produce the final sarcasm detection result. This approach embraces randomness in cue selection, enhancing the LLM’s ability to explore numerous potential paths, thus improving the robustness and accuracy of sarcasm detection. BoC consists of the following key steps:

Step 1. Cue subsets construction. A total of $\mathcal{T}$ cue subsets $\mathcal{S}_{t\in[1,2,...,\mathcal{T}]}=\left\{(c_{t_{1}},c_{t_{2}},...,c_{t_{q}}),t\in[1,2,...,\mathcal{T}]\right\}$ are created by randomly sampling without replacement from the complete pool of cues $\mathcal{C}$. Each sampling is independent.

Step 2. LLM prediction. For each subset $\mathcal{S}_{t}$, an LLM $\mathcal{L}_{\theta}^{BoC}$ is used to independently make sarcasm prediction through the comprehensive analysis of the cues in the subset and the input text. This can be conceptually encapsulated as $\hat{y}_{t}\sim\mathcal{L}_{\theta}^{BoC}\left(\mathcal{Y}|\mathcal{S}_{t},\mathcal{X}\right)$.

Step 3. Prediction aggregation. These individual predictions are then combined using an aggregation function, i.e., majority voting, to yield the final prediction: $Y\sim Vote\left(\left\{\hat{y}_{1},\hat{y}_{2},...,\hat{y}_{\mathcal{T}}\right\}\right)$.

BoC treats all cues as independent and does not follow the step-by-step reasoning paradigm for sarcasm detection (see Algorithm 3 in App. A).

CoC and GoC methods mainly handle low-order interactions between cues, while BoC assumes cues are independent. To capture high-order interactions among cues, we introduce tensor of cues, a novel paradigm that allows LLMs to amalgamate three types of cues ($viz.$ liguistic, contextual and emotional cues) into a high-dimensional representation (Fig. 2 (d)).

ToC treats each type of cues as an independent, orthogonal view for sarcasm understanding, and constructs a multi-view representation through the tensor product of such three types of cues. We first ask the LLM to extract linguistic, contextual, and emotional cues respectively via a simple prompt. Taking linguistic cue extraction as an example:

We take the outputs of the LLM’s final hidden layer as the embeddings of the linguistic, contextual and emotional cues, and apply a tensor fusion mechanism to fuse the cues as additional inputs to the sarcasm detection prompt. Inspired by the success of tensor fusion network (TFN) for multi-modal sentiment analysis Zadeh et al. (2017), we apply token-wise tensor fusion to aggregate the cues. In particular, the embeddings are projected on a low-dimensional space, i.e., $\vec{Lin}=\left(e_{1}^{l},e_{2}^{l},...,e_{L}^{l}\right)^{T}$, $\vec{Con}=\left(e_{1}^{c},e_{2}^{c},...,e_{L}^{c}\right)^{T}$, $\vec{Emo}=\left(e_{1}^{e},e_{2}^{e},...,e_{L}^{e}\right)^{T}$. Suppose the LLM has a hidden dimensionality of $d$, fully-connected layers $f_{lin},f_{con},f_{emo}$ are constructed to map the embeddings to dimensionality of $\{d_{l},d_{c},d_{e}\}$, respectively for linguistic, contextual and emotional cues. Then, a tensor product is computed to combine the cues into a high-dimensional representation $\mathcal{Z}=\left(e_{1},e_{2},...,e_{L}\right)^{T}$, where

The additional value of 1 facilitates an explicit rendering of single-cue features and bi-cue interactions, leading to a comprehensive fusion of different cues encapsulated in each fused token $e_{i}\in\mathcal{R}^{(d_{l}+1)\times(d_{c}+1)\times(d_{e}+1)}$. The values of $d_{l}$, $d_{c}$ and $d_{e}$ are delicately chosen such that the dimensionality of fused token is precisely $d$¹¹1Otherwise the fused tokens are truncated to d-dim vectors. That enables an integration of the aggregated cues to the main prompt via:

ToC facilitates deep interactions among these cues, providing a powerful and flexible framework for processing complex linguistic phenomena (see Algorithm 4 in App. A). Notably, as ToC manipulates cues on the vector level via neural structures, it requires access to the LLM structure and calls for supervised training on a collection of labeled samples. During training, the weights of the LLM are frozen, and the linear weights in $f_{lin},f_{con},f_{emo}$ are updated as an adaptation of LLM to the task context.

Datasets. Four benchmarking datasets are selected as the experimental beds, $viz.$ IAC-V1 Lukin and Walker (2013), IAC-V2 Oraby et al. (2016), SemEval 2018 Task 3 Van Hee et al. (2018) and MUStARD Castro et al. (2019).

IAC-V1 and IAC-V2 are from the Internet Argument Corpus (IAC) Lukin and Walker (2013), specifically designed for the task of identifying and analyzing sarcastic remarks within online debates and discussions. It encompasses a balanced mixture of sarcastic and non-sarcastic comments.

SemEval 2018 Task 3 is collected using irony-related hashtags (i.e. #irony, #sarcasm, #not) and are subsequently manually annotated to minimise the amount of noise in the corpuses. It emphasize the challenges inherent in identifying sarcasm within the constraints of MUStARD’s concise format, and highlight the importance of context and linguistic subtleties in recognizing sarcasm.

MUStARD is compiled from popular TV shows including Friends, The Golden Girls, The Big Bang Theory, etc. It consists of 690 samples total of 3,000 utterances. Each sample is a conversation consisting of several utterances. In this work, we only use the textual information. The statistics for each dataset are shown in Table 2.

Baselines. A wide range of SOTA baselines are included for comparison. They are:

• •

  PLMs.  (1) RoBERTa Liu et al. (2019), (2) BNS-Net Zhou et al. (2023), (3) DC-Net Liu et al. (2022), (4) QUIET Liu et al. (2023a) and (5) SarcPrompt Liu et al. (2023b) are five SOTA PLMs based approaches for sarcasm detection via pre-trained language modeling and refined representations.

• •

  Prompt tuning.  (6) IO, (7) CoT Wei et al. (2022) and (8) ToT Yao et al. (2024) are four SOTA prompting approaches by leveraging advanced prompt approaches to enhance LLM’s performance.

• •

  LLMs.  (9) GPT-4o²²2https://openai.com/index/hello-gpt-4o/ and (10) LLAMA 3-8B-Instruct³³3https://llama.meta.com/llama3/ are the strongest general LLMs.
  

Implementation. We have implemented the prompting methods for GPT-4o and LLaMA3-8B-Instruct, and reported the performance of PLMs in their original papers. The GPT-4o methods are implemented with the official openAI Python API library⁴⁴4https://github.com/openai/openai-python, while the LLaMA methods are implemented based on the Hugging Face Transformers library⁵⁵5https://huggingface.co/docs/transformers. All prompting strategies are implemented for GPT-4o and LLaMA3-8B-Instruct except for ToC, which can solely be deployed on open-sourced LLMs. Following previous works in this field, LangChain⁶⁶6https://github.com/langchain-ai/langchain is employed for the implementation of ToT and GoC. For the training of ToC, cross-entropy loss between the output logit and the true label token is computed to update the weights of the fully-connected layers.

We report both Accuracy and Macro-F1 results for SarcasmCue and baselines in a zero-shot setting in Table 3, except for ToC which requires supervised training for context adaption.