A Survey of Prompt Engineering Methods in Large Language Models for Different NLP Tasks

Basic prompting refers to the method of directly throwing a query at the LLM without any engineering around it to improve the LLM’s performance which is the core goal behind most of the prompting strategies. Basic prompting also goes by the name of Standard or Vanilla prompting in different research papers.

In this prompting strategy Wei et al. (2022), the authors build up on the idea of how human beings break a complex problem into smaller easier sub-problems before arriving at the final solution of the complex problem. Along similar lines, the authors investigate how capabilities of LLMs to do complicated reasoning is inherently enhanced by producing a chain of thought, or a sequence of intermediate reasoning steps. The results show a considerable improvement from Basic prompting with the maximum difference between CoT and Basic prompting results being as big as around 39% for Mathematical Problem Solving task and around 26% for Commonsense Reasoning task. This work opened a new direction of research for the field of prompt engineering.

Self-Consistency Wang et al. (2022) prompting technique is based on the intuition that complex reasoning problems can be solved in multiple ways and hence the correct answer can be reached via different reasoning paths. Self-Consistency uses a novel decoding strategy unlike the greedy one being used by CoT and consists of three important steps. The first step requires prompting the LLM using CoT, the second step samples diverse reasoning paths from LLM’s decoder and the final step involves choosing the most consistent answer across multiple reasoning paths. Self-Consistency on an average achieves 11% gain on Mathematical Problem Solving task, 3% gain on Commonsense Reasoning task and 6% gain on Multi-Hop Reasoning task when compared to CoT.

This prompting method has been discussed in Singhal et al. (2023). It builds on top of CoT and Self-Consistency. ER consists of two stages. First, given a few-shot CoT prompt and a query, LLM is made to produce multiple generations by adjusting it’s temperature. Each generation contains a reasoning and an answer for the query. Next, the LLM is conditioned on the original prompt, query and the concatenated generations from the previous stage to generate a better explanation and an answer. This second stage is done multiple times followed by a majority voting over these second stage generated answers just as it is done in case of Self-Consistency to select the final answer. ER is seen to perform better than CoT and Self-Consistency across many datasets belonging to the Context-Free Question-Answering task.

In this work Zhang et al. (2022), the authors address the problem faced by few-shot CoT or manual CoT which is the need of curation of good quality training datapoints. Auto-CoT consists of two primary steps. The first one requires dividing queries of a given dataset into a few clusters. The second one involves choosing a representative query from each cluster and then generating its corresponding reasoning chain using zero-shot CoT. The authors claim that Auto-CoT either outperforms or matches the performance of few-shot CoT across Mathematical Problem Solving, Multi-Hop Reasoning and Commonsense Reasoning task. This indicates that the step of curation of training datapoints for few-shot or manual CoT can be ruled out.

Fu et al. (2022) introduces a new prompting strategy which aims at choosing complex datapoint prompts over simpler ones. The complexity of a datapoint is defined here by the number of reasoning steps involved with it. The authors hypothesize that the LLMs’ reasoning performance can increase if complex datapoints are used as in-context training examples as they already subsume simpler datapoints. Another important aspect of Complex CoT apart from using complex datapoints as training examples is that during decoding, just like Self-Consistency, out of N sampled reasoning chains the majority answer over the top K most complex chains is chosen as the final answer. There is one other baseline prompting method which has been introduced in this paper called Random CoT. In Random CoT, the datapoints are randomly sampled without adhering to their complexity. Complex CoT achieves on an average a gain of 5.3% accuracy and up to 18% accuracy improvement across various datasets of Mathematical Problem Solving, Commonsense Reasoning, Table-Based Mathematical Problem Solving and Multi-Hop Reasoning tasks.

The authors of Chen et al. (2022a) build up on CoT but in contrast to CoT which uses LLMs to perform both reasoning and computation, PoT generates Python programs and thus relegates computation part to a Python interpreter. This work argues that reduced LLM responsibilities make it more accurate especially for numerical reasoning. PoT gets an average performance gain over CoT of around 12% across Mathematical Problem Solving, Table-Based Mathematical Problem Solving, Contextual Question-Answering and Conversational Contextual Question-Answering tasks.

Least-to-Most Zhou et al. (2022) prompting technique tries to address the problem of CoT where CoT fails to accurately solve problems harder than the exemplars shown in the prompts. It consists of two stages. First, the LLM is prompted to decompose a given problem into sub-problems. Next, the LLM is prompted to solve the sub-problems in a sequential manner. The answer to any sub-problem depends on the answer of the previous sub-problem. The authors show that Least-to-Most prompting is able to significantly outperform CoT and Basic prompting methods on Commonsense Reasoning, Language-Based Task Completion, Mathematical Problem Solving and Contextual Question-Answering tasks.

CoS Hu et al. (2023) builds up on the idea of CoT. In conventional CoT, the intermediate chain of reasoning steps are represented in natural language. While this approach has shown remarkable results in many cases, it can include incorrect or redundant information as well. The authors of this work present their hypothesis that spatial descriptions are hard to express in natural language thus making it difficult for LLMs to understand. Instead, expressing these relationships using symbols in word sequences can be a better form of representation for LLMs. CoS achieves an improvement of up to 60.8% accuracy for Spatial Question-Answering task.

The intuition behind SCoT Li et al. (2023b) is that structuring intermediate reasoning steps using program structures like sequencing, branching and looping helps in more accurate code generation than having intermediate reasoning steps in natural language as we see in conventional CoT. The authors claim that the former approach more closely mimics a human developer’s thought process than the latter one and the same has been confirmed by the the final results as SCoT outperforms CoT by up to 13.79% for the Code Generation task.

Wang et al. (2023) discusses and tries to address three shortcomings of CoT which are calculation errors, missing-step errors and semantic misunderstanding errors. PS contains two components where the first one requires devising a plan to divide the entire problem into smaller sub-problems and the second one needs to carry out these sub-problems according to the plan. A better version of PS called PS+ adds more detailed instructions which helps in improving the quality of reasoning steps. PS prompting method improves the accuracy over CoT by at least 5% for almost all the datasets in Mathematical Problem Solving task in zero-shot setting. Similarly, for the Commonsense Reasoning task, it consistently outperforms CoT by at least 5% in zero-shot setting whereas for the Multi-Hop Reasoning task it gets around 2% better accuracy score.

Imani et al. (2023) tries to address two key problems of CoT for Mathematical Problem Solving task: (1) lack of validity of steps followed by CoT for solving a problem; (2) how confident is an LLM in it’s predictions. MathPrompter prompting strategy consists of 4 steps in total. (I) Given a query, the first step requires to generate an algebraic expression for the query which replaces the numerical values by variables. (II) Next, LLM is prompted to solve the query analytically either by deriving the algebraic expression or writing a Python function. (III) Third, the query in step (I) is solved by assigning different values to the variables. (IV) If the solutions in (III) are correct over N iterations, the variables are finally replaced with original query values and the answer is computed. If not, then the steps (II), (III) and (IV) are repeated. MathPrompter is able to improve the performance on a dataset belonging to Mathematical Problem Solving task from 78.7% to 92.5%.

The authors of Chia et al. (2023) claim that Contrastive CoT or Contrastive Self Consistency is a general enhancement of CoT or Self-Consistency. The inspiration for this prompting approach is based on how humans can learn from both positive as well as negative examples. Along similar lines, in this prompting technique, both positive and negative demonstrations are provided to enhance the reasoning capabilities of the LLM. Contrastive CoT on an average is able to gain an average of 10% improvement over conventional CoT for Mathematical Problem Solving task across multiple datasets. Similarly, Contrastive Self-Consistency is able to outperform conventional Self-Consistency by over 15% for Mathematical Problem Solving task across multiple datasets. For Multi-Hop Reasoning task, both Contrastive CoT and Contrastive Self-Consistency have more than 10% gains over their conventional counterparts.

Introduced in Liu et al. (2023), this prompting method is based on the core idea of improving the reasoning capabilities of LLMs by using synonymous crowd-sourced queries. There are two slightly different variations of this prompting method. The first one is Fed-SP-SC where the crowd-sourced queries are paraphrased versions of the original query but with same parameters. Parameters here can refer to the numeric values in Mathematical Problem Solving task datapoints. For Fed-SP-SC, the answers are directly generated first and then Self-Consistency is applied on top of it. The other one is Fed-DP-CoT. In Fed-DP-CoT, LLMs are used to first generate answers to different queries and then they are federated by forming CoT to provide hints to the LLMs. The results for these methods on Mathematical Problem Solving task show that they are able to do better than conventional CoT by at least 10% and up to 20%.

The authors of this work Yasunaga et al. (2023) draw their inspiration from a psychological notion, analogical reasoning, where people use pertinent prior experiences to solve new problems. In the realm of LLMs, the authors first prompt them to generate examples similar to that of the original problem followed by solving them and then proceed to answer the original problem. The results show that Analogical Reasoning is able to achieve an average accuracy gain of 4% when compared to CoT across Mathematical Problem Solving, Code Generation, Logical Reasoning and Commonsense Reasoning tasks.

The authors of Shao et al. (2023) come up with Synthetic prompting using LLMs to generate synthetic examples which are augmented to the existing hand-crafted examples as seen in a conventional few-shot setting. This prompting method involves two steps: (1) the backward step, where the LLM synthesizes a query based on a self-generated reasoning chain; and (2) the forward step, where the LLM generates a reasoning chain for the synthesized query, making the reasoning chain to be more accurate. Finally, to choose the best examples, this work uses an in-cluster complexity and the most complex examples with the longest reasoning chains are used during inference. The results show Synthetic prompting achieving up to 15.6% absolute gains when experimented with different Mathematical Problem Solving, Commonsense Reasoning and Logical Reasoning task datasets.

ToT Yao et al. (2024) prompting technique has been drawn from the idea that any kind of problem solving requires searching through a combinatorial space represented as a tree where each node represents a partial solution and each branch corresponds to an operator that modifies it. Now, the decision about which branch to choose is determined by heuristics that help to navigate the problem-space and guide the problem-solver towards a solution. Based on this idea, the authors propose ToT which actively maintains a tree of thoughts where each thought is a coherent language sequence that serves as an intermediate reasoning step toward problem solving. This framework allows LLMs to evaluate the progress generated by thoughts while trying to solve the problem. ToT further incorporates search techniques such as breadth-first or depth-first search with the model’s ability to generate and evaluate thoughts. ToT achieves 65% better success rate than CoT on Mathematical Problem Solving task and around 40% better success rate on different Logical Reasoning task datasets. ToT further achieves coherency score of 7.56 where CoT gets only 6.93 on an average on Free Response task.

In this work Zhao et al. (2023b), the authors investigate the usage of logical equivalence in order to improve the zero-shot reasoning abilities of an LLM. In addition to allowing the LLM to reason step-by-step, LoT also allows the LLM to verify step-by-step in accordance with the guidelines provided by the Reductio ad Absurdum principle and, if needed, amend the reasoning chain to ensure a valid inference. LoT is able to surpass CoT in Mathematical Problem Solving task by a maximum of 3.7%, Commonsense Reasoning task by a maximum of 16.2%, Logical Reasoning task by a maximum of 2.5%, Causal Reasoning task by a maximum of 15.8% and Social Reasoning task by a maximum of 10% accuracy.

By using deep recursive reasoning to elicit abductive explanations for a variety of hypotheses, Maieutic prompting Jung et al. (2022) encourages the LLM to produce consistent responses by collaboratively eliminating alternatives that contradict one another. The generation process of Maieutic prompting derives a tree structure of generated propositions, where one proposition establishes a logical ground for the correctness of one another. Finally, to infer the answer to the original query, the degree to which the LLM believes each proposition and the logical connections between propositions in the maieutic tree is measured. The results for Maieutic prompting on Commonsense Reasoning task shows that it is able to achieve up to 20% better accuracy when compared to Basic prompting, CoT, Self-Consistency and GKP Liu et al. (2021) while performing competitively with supervised models.

Zhao et al. (2023a) focuses on developing a technique which can post-edit the reasoning chains generated by CoT for more factually aligned outputs. This method consists of three stages: (1) the deciding when to edit stage where the authors use Self-Consistency to find uncertain outputs; (2) the how to edit rationales stage where the authors edit CoT reasoning chains of uncertain outputs by searching for supporting facts from external knowledge sources and (3) the reasoning stage where the edited rationales from previous stage are used to come up with final answers. VE is able to outperform CoT, Self-Consistency and Basic prompting by up to 10% on Multi-Hop Reasoning task and by up to 2% on Truthfulness task.

Yao et al. (2022b) presents ReAct, which combines reasoning and acting with LLMs to solve diverse language reasoning and decision making tasks. In order to enable the model to perform dynamic reasoning to build and modify high-level plans for acting (reason to act), ReAct prompts LLMs to generate verbal reasoning traces and actions related to a task in an interleaved manner. Another prompting method similar to ReAct discussed in Yao et al. (2022b) is Act which basically removes thoughts or reasoning in ReAct trajectories but performs suboptimal to ReAct in all the discussed tasks. For Multi-Hop Reasoning and Truthfulness tasks, ReAct is able perform better than Basic prompting while being competitive with CoT. When ReAct is combined with CoT or Self-Consistency, it is able to get better results than CoT. For Language-Based Task Completion task, ReAct outperforms reinforcement learning methods with an absolute improvement of more than 10% in success rates individually on different datasets.

Diao et al. (2023) proposes Active-Prompt to help LLMs adapt to different tasks with task-specific examples by identifying the most relevant datapoints to be used as examples while prompting the LLM in a few-shot setting. Active-Prompt is a four-step technique. In the first step, the LLM is prompted k times for each query in the training set to generate k possible answers with their corresponding reasoning chains. The next step requires calculating the uncertainty metric based on the answers generated in step one. In the third step, the top n most uncertain queries are selected and annotated by humans. In the final step, the new annotated examples are used to do few-shot prompting for the test data. The authors also introduce a different version of Active-Prompt called Random CoT where in step 3, top n queries are selected randomly than based on the uncertainty metric. The results show that Active-Prompt is able to get better results than Self-Consistency, CoT, Auto-CoT and Random CoT across multiple datasets for Mathematical Problem Solving, Commonsense Reasoning, Multi-Hop Reasoning, Commonsense Reasoning tasks.

Zhou et al. (2023) proposes a prompting method focusing on handling long chaotic contexts. It is based on the idea that there is an unbroken flow of thought that people retain when going through a large amount of information, enabling the selective extraction of pertinent data and the rejection of irrelevant ones. This balance of attention across a document’s sections is important for accurate interpretation and response to the information supplied. ThoT consists of two steps. The first one requires the LLM to analyze and summarize the different sections of the context. In the second step, the LLM is prompted to answer the asked query based on the output of first step. ThoT is able to outperform CoT and Basic promoting techniques by achieving a score of around 0.56 exact match in Context-Free Question-Answering task. For Dialogue System task, ThoT is able to get the highest average score of 3.8 again surpassing other discussed prompting techniques.

Contrary to the conventional RAG Lewis et al. (2020), Implicit RAG Vatsal & Singh (2024); Vatsal et al. (2024) asks the LLM itself to retrieve important chunks or sections from the given context and then proceed to answer the asked query. This technique requires tuning of two hyper-parameters. The first one is the number of sections to extract whereas the second one is the number of words in each section. Implicit RAG achieves SoTA result on Contextual Question-Answering task in Vatsal et al. (2024) on Patient Case Reports dataset whereas achieved either SoTA or close to SoTA results on biomedical Contextual Question-Answering task datasets in Vatsal & Singh (2024).

LLMs can often end up making erroneous judgments when presented with irrelevant context. Weston & Sukhbaatar (2023) tries to address this issue with two-step prompting strategy. The first step instructs the LLM to regenerate a given context such that the regenerated version does not contain any irrelevant parts that could adversely affect the output. The second step then instructs the LLM to produce the final response using the regenerated context from step 1. The results show that S2A is able to outperform Basic, CoT as well Instructed prompting Shi et al. (2023) over different Truthfulness task datasets.

Intructed prompting Shi et al. (2023) again revolves around the same idea as that of S2A which tries to address the issue of LLMs getting distracted by irrelevant context. It consists of only one step of explicitly instructing the language model to ignore irrelevant information in the problem description. Instructed prompting is able to achieve 88.2 normalized micro accuracy for Truthfulness task and is able to surpass all it’s counterparts including CoT, Least-To-Most, Program prompting and Self-Consistency. Program prompting Chowdhery et al. (2023) strategy here tries to solve a problem by writing a Python program for it. Later, the correctness of the written program is verified by running the Python code using an external Python interpreter to obtain the final answer.

LLMs are prone to generating factually incorrect information called hallucination. The authors of Dhuliawala et al. (2023) try to address this problem of hallucination and improve performance via CoVe. CoVe performs four core steps. First, the LLM generates a baseline response for a given query. Second, using both the original query and the baseline response from step one, generate a list of verification queries that are capable of checking if there are any errors in the baseline response. Third, generate answers to all the verification queries from step three. Fourth, correct all the mistakes in the baseline response detected after step three and produce a revised response. The results show that CoVe is able to outperform CoT and Basic prompting by around at least 10% on Context-Free Question-Answering, Contextual Question-Answering and Free Response tasks.

Similar to CoVe, CoK Li et al. (2023c) tries to address the issue of hallucination to get more accurate results. It’s a three-stage prompting technique. The first stage is reasoning preparation where given a query, CoK prepares several preliminary rationales and answers while identifying the relevant knowledge domains. The second stage is dynamic knowledge adaptation where if there is no majority consensus among the answers, CoK corrects the rationales step by step by adapting knowledge from the identified domains in stage one. The third stage is answer consolidation which uses these corrected rationales from stage two to serve as a better foundation for the final answer consolidation. CoVe surpasses CoT, Self-Consistency, VE and Basic prompting across Context-Free Question-Answering, Table-Based Question-Answering, Multi-Hop Reasoning and Truthfulness tasks and shows an improvement of at least 3%, 3%, 1% and 1% respectively.

In this work Li et al. (2023a), the authors propose an extension to make LLM’s code-oriented reasoning better. Here, the LLM not only writes a code for a program but also selectively simulates the interpreter by producing the expected outputs of certain lines of code which cannot be actually executed by an interpreter. The main idea is to motivate LLMs to format semantic sub-tasks in a program as flexible pseudocode that may be explicitly caught and passed off to an LLM for emulation at runtime which the authors call an LMulator. Experiments demonstrate CoC surpassing CoT and other baselines across a variety of tasks including Recommender System, Causal Reasoning, Commonsense Reasoning, Spatial Question-Answering, Emotion/Sentiment Understanding, Machine Translation, Logical Reasoning, Table-Based Mathematical Problem Solving and Mathematical Problem Solving.

Gao et al. (2023) proposes a prompting strategy that uses an LLM to read natural language problems and generate interleaved natural language and programming language statements as reasoning steps. Finally, a Python interpreter is used to execute programming statements to get the answer. The results show that PAL easily performs better than it’s counterparts like CoT and Basic prompting across multiple NLP tasks including Mathematical Problem Solving, Table-Based Mathematical Problem Solving, Commonsense Reasoning and Logical Reasoning.

The authors claim Binder Cheng et al. (2022) to be a training-free neural-symbolic technique that maps an input to a program which (I) enables binding of a single API of LLM functionalities to a programming language such as Python or SQL in order to increase it’s coverage of grammar and to address a wider range of queries; (II) uses an LLM as the underlying model as well as the program parser during execution; (III) needs only a few in-context sample annotations. The binder pipeline has two stages. First, in the parsing stage, the LLM maps the input to a program given the query and knowledge sources. Second, in the execution stage, the LLM returns values in the chosen programming language and finally the program is run using an interpreter. Binder is able to get better accuracy when compared to previous methodologies which required explicit training or fine-tuning for Table-Based Truthfulness and Table-Based Question-Answering tasks.

Ye et al. (2023) explores the idea of few-shot learning with LLMs to decompose evidence and queries for efficient table-based reasoning. This prompting strategy involves three important steps. It starts with decomposing a huge table into relevant smaller sub-tables given the query. Next, SQL programming language is used to decompose the complex natural language query into logical and numerical computations. Finally, the sub-tables and sub-queries from previous two steps are used to arrive at the final answer in a few-shot setting. The results show that Dater is able to surpass previous methodologies which required explicit fine-tuning by at least 2% in Table-Based Truthfulness task. Similarly, for Table-Based Question-Answering task, it is able to outperform such methods by at least 1%. Dater is also able to do better than Binder for both the above-mentioned tasks.

In Wang et al. (2024), the authors build up on the famous prompting technique of CoT and bring it to the tabular setting. This multi-step tabular prompting approach leads to more accurate table understanding. Chain-of-Table is a three-step prompting technique. The first step instructs the LLM to dynamically plan the next table operation by in-context learning. An operation here could be anything from addition of columns to sorting of rows. The second step generates arguments for the selected table operation. The first two steps help in transforming the table and creating various intermediate table representations with the goal of answering the original query. In the final step, the last table representation from the first two steps is used to finally answer the query. Chain-of-Table achieves SoTA performance on Table-Based Question-Answering and Table-Based Truthfulness tasks. For Table-Based Question-Answering task, it gets around 3% of average better performance whereas for Table-Based Truthfulness task it is able to get around 1.5% of average better performance when compared to the prior SoTA results.

Khot et al. (2022) comes up with DecomP technique which decomposes a complex problem into simpler sub-problems and then delegates these to sub-problem specific LLMs, which have their own prompts and decomposers to further decompose the sub-problems. The decomposers can either resort to hierarchical decomposition, recursive decomposition or make external API calls to solve the sub-problem. DecomP is able to outperform CoT and Least-to-Most on an average by 25% in terms of exact match for Commonsense Reasoning task. For Multi-Hop Reasoning task, DecomP is comfortably able to do better than CoT on four different datasets.

The authors of Fei et al. (2023) come up with THOR to mimic human-like reasoning process for Emotion/Sentiment Understanding task. THOR consists of three steps. In the first step, the LLM is asked to identify the aspect mentioned in the given query. Next, based on previous step output and the original query, the LLM is asked to answer in detail about the underlying opinion embedded in the query. Finally, all of the above information is combined and the LLM is asked to infer the sentiment polarity associated with the given query. THOR is able to significantly surpass prior SoTA supervised as well as zero-shot models on multiple Emotion/Sentiment Understanding task datasets.

MP Wang & Zhao (2023) is based on the concept of meta-cognition which is derived from cognitive psychology and relates to an individual’s awareness and self-reflection on their cognitive processes. It consists of five stages. 1) understanding the input text, 2) making a preliminary judgment, 3) critically evaluating this preliminary analysis, 4) reaching a final decision accompanied by an explanation of the reasoning, and 5) evaluating the confidence level in the entire process. The results show that MP consistently excels CoT and PS across numerous NLP tasks including Paraphrasing, Natural Language Inference, Contextual Question-Answering, Word Sense Disambiguation, Named Entity Recognition, Relation Extraction and Multilabel Text Classification.

Bao et al. (2024) proposes CoE for the Summarization task. CoE has four sequential steps. The first one focuses on specific event extraction. Next, the events extracted in step one are analyzed and generalized into more concise and refined form. Third, the events generalized in the previous step are filtered and only those are selected which cover most of the text. In the last step, the events selected in step three are integrated based on their chronological order of importance. The results show that CoE is able to perform better than CoT across two Summarization datasets in terns of rouge score while also being more concise.

This is one of prompting methods discussed in Vatsal et al. (2024). In this method, basic prompt instructions get enhanced by addition of medical term definitions based on the hypothesis that adding these definitions would help the LLM in gaining more context while answering the asked query. But the results show that these term definitions do not really help possibly because of their narrow knowledge scope which may be conflicting with the bigger knowledge base of the LLM.

Hu et al. (2024) tests LLM capabilities in clinical Named Entity Recognition task. This prompting strategy has three different components. The basic component tells the LLM about the rudimentary information regarding the task and in what format the LLM should output the results. The annotation guideline component contains entity definitions and linguistic rules derived from the annotation guidelines. The error analysis component incorporates additional instructions following error analysis of LLM outputs using the training data. Different versions of this prompting method have been also experimented by the authors by creating different combination of above-mentioned components. This prompting method is able to get on an average 0.57 exact match F1 score on multiple datasets belonging to Named Entity Recognition task.

This task measures a model’s ability to perform any kind of mathematical computation in a non tabular setting. The different datasets which we came across while reading up on different prompting methods for this task are GSM8K Cobbe et al. (2021), MATH Hendrycks et al. (2021), SVAMP Patel et al. (2021), ASDiv Miao et al. (2021), AQuA Ling et al. (2017), MAWPS Koncel-Kedziorski et al. (2016), MultiArith Koncel-Kedziorski et al. (2016), AddSub Koncel-Kedziorski et al. (2016), SingleEq Koncel-Kedziorski et al. (2016), Game of 24 Yao et al. (2024), Multi-Step Arithmetic Srivastava et al. (2022), GSM-HARD Gao et al. (2023), SingleOp Koncel-Kedziorski et al. (2016) and MathQA Amini et al. (2019). Table LABEL:tab:mps lists above-mentioned datasets and different prompting methods that have been experimented on them along with the best performing prompting strategy.

Logical Reasoning task checks a model’s natural language understanding to follow a set of commands with inputs and solve a given problem. The different datasets which we covered while reading up on different prompting strategies for this task are Word Sorting Srivastava et al. (2022), Temporal Sequences Srivastava et al. (2022), Formal Fallacies Srivastava et al. (2022), Mini Crosswords Yao et al. (2024), Object Counting Srivastava et al. (2022), Logical Deduction Srivastava et al. (2022), Boolean Expressions Srivastava et al. (2022), Tracking Shuffled Objects Srivastava et al. (2022), Web of Lies Srivastava et al. (2022), Dyck Languages Srivastava et al. (2022), Geometric Shapes Srivastava et al. (2022), Repeat Copy Logic Srivastava et al. (2022). Table LABEL:tab:logicalr contains above-mentioned datasets and different prompting techniques that have been experimented on them along with the best performing prompting method.

Contrary to Logical Reasoning task, Commonsense Reasoning task measures a model’s ability in terms of common practical knowledge often referred to as commonsense by humans to make any kind of judgement. It does not involve solving a problem to arrive at an answer. Rather, it is more of a form of inherent general knowledge. The various datasets that we discovered while surveying different prompting methods for this task include Reasoning about Colored Objects Srivastava et al. (2022), CSQA Talmor et al. (2018), Date Understanding Srivastava et al. (2022), Sports Understanding Srivastava et al. (2022), Last Letter Concatenation Wei et al. (2022), Coin Flip Wei et al. (2022), Odd One Out Srivastava et al. (2022), Disambiguation QA Srivastava et al. (2022), Hyperbaton Srivastava et al. (2022), Com2Sense Singh et al. (2021), CSQA 2.0 Talmor et al. (2022), Creak Onoe et al. (2021) and List Reversal Khot et al. (2022). Table LABEL:tab:commonr shows above-mentioned datasets and different prompting strategies that have been experimented on them along with the best performing prompting method.

Multi-Hop Reasoning task assess how good a model is at connecting pieces of evidence from different parts of a context to answer a given query. The different datasets which we covered while reading up on different prompting strategies for this task are StrategyQA Geva et al. (2021), HotpotQA Yang et al. (2018), Bamboogle Press et al. (2022), CommaQA-E Khot et al. (2021), MuSiQue Trivedi et al. (2022), 2WikiMultihopQA and Ho et al. (2020). Table LABEL:tab:mhr lists above-mentioned datasets and different prompting methods that have been experimented on them along with the best performing prompting strategy.

Causal Reasoning task checks a model’s ability to deal with cause and effect. We came across two datasets while reading up on different prompting techniques for this task which are Cause And Effect Srivastava et al. (2022) and Causal Judgement Srivastava et al. (2022). Table LABEL:tab:causalr shows above-mentioned datasets and different prompting techniques that have been experimented on them along with the best performing prompting method.

This task tests a model’s ability to reason about human social interactions. We discovered only one dataset while surveying different prompting techniques for this task which is SocialQA Srivastava et al. (2022). Table LABEL:tab:socialr contains above-mentioned datasets and different prompting methods that have been experimented on them along with the best performing prompting strategy.

This task measures a model’s ability to answer a query solely by relying on a given context. The different datasets which we covered while reading up on different prompting methods for this task are ProcessBank Berant et al. (2014), BioMRC Pappas et al. (2020), MASH-QA Zhu et al. (2020), CliCR Šuster & Daelemans (2018), MultiSpanQA Li et al. (2022), FinQA Chen et al. (2021b), TAT-QA Zhu et al. (2021), Patient Case Reports Vatsal & Singh (2024), Drop Dua et al. (2019) and BoolQ Clark et al. (2019). Table LABEL:tab:cqa lists above-mentioned datasets and different prompting methods that have been experimented on them along with the best performing prompting technique.

In contrast to the Contextual Question-Answering task, the Context-Free Question-Answering task relies on model’s embedded knowledge base or any open-source knowledge base, such as Wikipedia, to answer a query instead of using only the context provided. The various datasets that we discovered while surveying different prompting techniques for this task are PopQA Mallen et al. (2022), EntityQ Sciavolino et al. (2021), Wikidata Dhuliawala et al. (2023), Wiki-Catoegory List Dhuliawala et al. (2023), MedMCQA Pal et al. (2022), MMLU Physics Hendrycks et al. (2020), MMLU Biology Hendrycks et al. (2020), USMLE Sample Exam Nori et al. (2023), USMLE Self Assessments Nori et al. (2023), MedQA Jin et al. (2021), PubMedQA Jin et al. (2019), MMLU Hendrycks et al. (2020) and AI2 Reasoning Challenge Clark et al. (2018). Table LABEL:tab:cfqa lists above-mentioned datasets and different prompting strategies that have been experimented on them along with the best performing prompting strategy.

Spatial Question-Answering task measures a model’s ability to deal with spatial reasoning which is a cognitive process based on the construction of mental representations for spatial objects, relations, and transformations. The various datasets which we came across while reading up on different prompting techniques for this task include Brick World Hu et al. (2023), NLVR-Based Manipulation Hu et al. (2023), Natural Language Navigation Hu et al. (2023), Spartun Mirzaee & Kordjamshidi (2022) and Navigate Srivastava et al. (2022). Table LABEL:tab:sqa contains above-mentioned datasets and different prompting methods that have been experimented on them along with the best performing prompting strategy.

In this task, the model is assessed based on it’s understanding of a given text extract and how it is able to answer a series of interconnected queries that appear in a conversational format. A key thing to note here is that each query may depend on previous queries’ answers. We covered only one dataset while reading up on different prompting methods for this task which includes ConvFinQA Chen et al. (2022b). Table LABEL:tab:ccqa shows above-mentioned datasets and different prompting methods that have been experimented on them along with the best performing prompting strategy.

Dialogue System task checks model’s ability to perform language generation in a user-to-machine conversation setting or answer queries given an already generated conversation. It is possible that when the text extract in case of Conversational Contextual Question-Answering becomes a conversation, there will be a strong overlap between these two tasks but based on the datasets and prompting techniques encountered during our survey, we decided to keep these two as separate tasks. We discovered only one dataset while surveying different prompting methods for this task which includes Multi-Turn Conversation Response (MTCR) Zhou et al. (2023). Table LABEL:tab:dias lists above-mentioned datasets and different prompting strategies that have been experimented on them along with the best performing prompting technique.

This task involves all the cases where the input or the final output to the model is a programming language code. The different datasets which we came across while reading up on different prompting strategies for this task are Codeforce Scraping Yasunaga et al. (2023), HumanEval Chen et al. (2021a), MBPP Austin et al. (2021) and MBCPP Athiwaratkun et al. (2022). Table LABEL:tab:codeg contains above-mentioned datasets and different prompting techniques that have been experimented on them along with the best performing prompting strategy.

This task assess a model’s ability in generating unconstrained textual response. The various datasets which we covered while reading up on different prompting methods for this task include Creative Writing Yao et al. (2024) and Longform Generation of Biographies Min et al. (2023). Table LABEL:tab:freer lists above-mentioned datasets and different prompting strategies that have been experimented on them along with the best technique.

This task assess a model’s ability to communicate factually and not spread any kind of misinformation. This task does not represent a model’s capability in understanding a given context, rather it is more focused on them not making false statements based on their understanding. The various datasets that we discovered while surveying different prompting strategies for this task are SycophancyEval, https://github.com/meg-tong/sycophancy-eval ¹¹1https://github.com/meg-tong/sycophancy-eval, GSM-IC Shi et al. (2023) and Fever Thorne et al. (2018). Table LABEL:tab:truth shows above-mentioned datasets and different prompting techniques that have been experimented on them along with the best performing prompting technique.

This task is an extension of Truthfulness task and measures a model’s ability to communicate factually and not spread any kind of misinformation in a tabular setting. The only dataset we came across while reading up on different prompting methods for this task is TabFact Chen et al. (2019). Table LABEL:tab:tbtruth contains above-mentioned datasets and different prompting strategies that have been experimented on them along with the best performing prompting strategy.

This task involves any kind of question-answering in a tabular setting. It can be considered as a super set of other kinds of table-based tasks like Table-Based Truthfulness or Table-Based Mathematical Problem Solving. But in this work, in order to avoid any confusion, we have captured all the datasets under this task which do not fall under more specific table-based tasks like Table-Based Truthfulness or Table-Based Mathematical Problem Solving. We came across only two datasets while reading up on different prompting strategies for this task which are FeTaQA Nan et al. (2022) and WikiTQPasupat & Liang (2015). Table LABEL:tab:tbqa shows above-mentioned datasets and different prompting methods that have been experimented on them along with the best performing prompting strategy.

This task is an extension of Mathematical Problem Solving task and measures a model’s ability to perform any kind of mathematical computation in a tabular setting. The different datasets which we covered while reading up on different prompting techniques for this task include TabMWP Lu et al. (2022) and Penguins in a Table Srivastava et al. (2022). Table LABEL:tab:tbmps lists above-mentioned datasets and different prompting methods that have been experimented on them along with the best performing prompting strategy.

This task measures a model’s ability to process a given input and suggest a set of items which are most relevant from a list of possible items as output. We discovered only one dataset while surveying different prompting techniques for this task which is Movie Recommendation Srivastava et al. (2022). Table LABEL:tab:rec lists above-mentioned datasets and different prompting methods that have been experimented on them along with the best performing prompting technique.

This task checks how good a model is at understanding human emotions or sentiments. The various datasets which we came across while reading up on different prompting methods for this task include Ruin Names Srivastava et al. (2022), SemEval14 Laptop and Restaurant Pontiki et al. (2016) and Forex Fatouros et al. (2023). Table LABEL:tab:emotion contains above-mentioned datasets and different prompting techniques that have been experimented on them along with the best performing prompting strategy.

In this task, a model is tested on it’s ability in terms of translation between two languages. The different datasets which we came across while reading up on different prompting techniques for this task include Salient Translation Error Detection Srivastava et al. (2022), FLORES Costa-jussà et al. (2022), WMT21 Farhad et al. (2021), Multi-Domain Aharoni & Goldberg (2020) and PDC Sun et al. (2020). Table LABEL:tab:mtrans lists above-mentioned datasets and different prompting methods that have been experimented on them along with the best performing prompting strategy.

Named Entity Recognition task aims at identifying predefined classes or categories of objects in a given input text. The different datasets that we discovered while surveying different prompting techniques for this task are MTSamples Uzuner et al. (2011), VAERS Du et al. (2021), Research Papers Tang et al. (2024) and BC5CDR-chem Li et al. (2016). Table LABEL:tab:ner shows above-mentioned datasets and different prompting strategies that have been experimented on them along with the best performing prompting strategy.

Word Sense Disambiguation task checks how good a model is at deciphering different meanings of a word in different contextual surroundings. We came across only one dataset while reading up on different prompting methods for this task which includes WiC Pilehvar & Camacho-Collados (2018). Table LABEL:tab:wsd shows above-mentioned datasets and different prompting techniques that have been experimented on them along with the best performing prompting method.

This task tests a model’s ability in breaking down a lengthy piece of input text into smaller chunks while ensuring retention of vital information in these smaller chunks. We covered only one dataset while reading up on different prompting methods for this task which is CCTC Bao et al. (2024). Table LABEL:tab:summa contains above-mentioned datasets and different prompting techniques that have been experimented on them along with the best performing prompting strategy.

Paraphrasing task aims at rewriting a given piece of input text by using different words while keeping the true semantics of the original input text same. A key difference between Summarization task and Paraphrasing task is that the main goal of Summarization task is to shorten the length of output text with respect to that of input text whereas Paraphasing task just focuses on using different words during it’s rewriting process. We discovered only one dataset while surveying different prompting methods for this task which includes QQP ²²2https://quoradata.quora.com/First-Quora-Dataset-Release-Question-Pairs. Table LABEL:tab:parap lists above-mentioned datasets and different prompting methods that have been experimented on them along with the best performing prompting technique.

This task evaluates a model’s ability in determining from text whether the author of the text is in favor or against a topic or target or an object of evaluation. The different datasets which we came across while reading up on different prompting techniques for this task are SemEval-2016 Mohammad et al. (2016), VAST Allaway & McKeown (2020) and P-Stance Li et al. (2021). Table LABEL:tab:stanced shows above-mentioned datasets and different prompting methods that have been experimented on them along with the best performing prompting technique.

The main objective of this task is to determine whether a hypothesis is true (entailment), false (contradiction), or undetermined (neutral) given a premise. The different datasets which we covered while reading up on different prompting methods for this task are QNLI Rajpurkar et al. (2016) and MedNLI Romanov & Shivade (2018). Table LABEL:tab:nli contains above-mentioned datasets and different prompting strategies that have been experimented on them along with the best performing prompting method.

The main objective of this task to check how good is a model in following a sequence of language-based navigational commands to make decisions about it’s actions required to complete a task.The different datasets that we discovered while surveying different prompting strategies for this task are ALFWorld Shridhar et al. (2020), WebShop Yao et al. (2022a), SayCan Ahn et al. (2022) and Scan Lake & Baroni (2018). Table LABEL:tab:langbased lists above-mentioned datasets and different prompting methods that have been experimented on them along with the best performing prompting method.

This task measures a model’s ability to assign each input to a set of predefined target labels. This task can encapsulate a lot of above-mentioned tasks like Stance Detection, Named Entity Recognition etc but again in order to keep these task definitions as disjoint as possible for a better survey of prompting methods, we have included only those datasets under this task which could not be suitably categorized under any of the above-discussed tasks. The different datasets which we covered while reading up on different prompting strategies for this task include EUR-LEX Chalkidis et al. (2021), UNFAIR-ToS Lippi et al. (2019) and LEDGAR Tuggener et al. (2020). Table LABEL:tab:mlc contains above-mentioned datasets and different prompting strategies that have been experimented on them along with the best performing prompting method.