Question-Analysis Prompting Improves LLM Performance in Reasoning Tasks

Large language models (LLMs) have recently shown rapid improvement across a host of standard natural language processing (NLP) tasks, including arithmetic, commonsense and symbolic reasoning. (Brown et al., 2020) Although these models show improved ability to understand and generate text (OpenAI, 2023), their performance can still be further improved. One solution is to encourage the model to think step-by-step. Using chain-of-thought prompting (Wei et al., 2022), LLMs are given Q&A exemplars which are designed to elicit a structured step-by-step response from the model. Many newly developed strategies meant to improve LLM performance have been focused on sophisticating the model’s step-by-step calculation (Gu et al., 2023). Despite SoTA prompts’ remarkable success across various tasks, their accuracies can still be further improved. In this work, we explore ways to improve the model reasoning not only in the answer steps, but also how the model interprets the question itself. By making the model to explicitly interpret the question, we maximize its understanding of the question and minimize missed key information. This paper introduces Question-Analysis Prompting (QAP), a simple zero-shot prompting strategy that induces the model to first explain the question before solving. This method is adaptable to various problem difficulties and shows promising results in math and commonsense reasoning across different model sizes.

The key principle behind QAP is that the model should reiterate the problem before solving. Another principle is that we should be able to control how much the model explains so that we can adapt the prompt to different model sizes and problem complexities. The specific prompt used is as follows:

"Explain this problem to me in at least $n$ words. Then solve for the answer."

In this work, we experiment with n = 25, 50, 100, 150, 200. The versions of these prompts are named QAP$n$. Although the model is not constrained to generating fewer than n tokens in its summary, we find that the number of tokens in the response correlates strongly with the choice of $n$. We show specific examples of the impacts of $n$ in the Appendix.

In Figure 1, we highlight the structure of a standard QAP output. First, the model breaks down the question in its own words and provides in-depth analysis on each event. We notice a direct relationship between the explanation and the answer steps. Each calculation is previously mentioned in the explanation portion, and this proves that the explanation has allowed the model to plan its approach even before solving. As a result, there is a significant increase in step-by-step calculation and a decreased chance of missed steps.

Question Difficulties Based On Baseline Performance: Within a given dataset, the difficulty of the individual question may vary. We propose a method to measure question difficulty based on performance with the baseline prompt. If the model can answer the problem correctly with the baseline prompt, then we consider the question to be easy; otherwise the question is hard. We analyze the performance of different prompts across “easy” and “hard” questions. Table 3 and Table 4 shows that QAP consistently outperforms other prompts in the “hard” category.

Impact Of Word Counts On Question Difficulties: QAP generates higher word counts for both “easy" and “hard" questions ( Table 5 and  Table 6 ), despite performing lower on “easy” questions. Although more step-by-step thought processes are encouraged to avoid mistakes during reasoning, this suggests that over-explanation can negatively impact the mode (also shown in Figure  Figure 5). Thus, the most suitable word count to solve a problem will vary from task to task; longer explanations are best suited to more complicated questions for which baseline prompting fails.

Downsides Of Smaller QAPs: Despite high performance on StrategyQA, QAP25 performs poorly on arithmetic datasets (mostly SAT and AQuA) using GPT-3.5 Turbo. Due to a small value of n, the model outputs are unfinished responses (i.e. the model stops midway through its reasoning steps) (shown in Figure 8) On SAT math, 51% of responses were incomplete for QAP25. On AQuA, 19% of responses were incomplete for QAP25.

Placement of the prompt: In this evaluation, we studied the impact of prompt placement on performance using GSM8K dataset. Two options for prompt placement were considered, Q_Begin - adding the prompt before the question and Q_End - adding the prompt after the question. Both placements provided similar results on GPT-3.5 and GPT-4. Results shown in the rest of the paper are based on Q_End.

Two-stage QAP: In this approach, we performed the prompting in two stages. In the first stage the model is prompted with, “Explain this problem to me in at least 50 words WITHOUT SOLVING.” In the second stage, the model is prompted again with the question and the explanation from the first stage. On GSM8K and AQuA, the model not only explained the problem, but also outlined steps needed to solve it. However, the accuracy was almost 50% worse than single stage prompting.

In one-shot and few-shot prompting, the model is given one or more input/output examples which will serve as a demonstration for it to solve the problem using in-context learning (Mahabadi et al., 2022). QAP is a zero-shot prompt. In zero-shot prompting the model does not receive exemplars, but is given a specially crafted instruction on how to approach the task (Kojima et al., 2022).

Chain of Thought: Chain-of-thought reasoning is a notable few-shot (zero-shot also exists (Yang et al., 2023) example in which the model is shown how to express its reasoning steps (Wei et al., 2022). This approach was highly effective as the model would replicate these exemplars, and their accuracies improved drastically. CoT encouraged the model to think step-by-step, and this concept would be repeating theme among other zero-shot counterparts.

TADB: Among different variants of Zero-Shot CoT, the TADB prompt (Yang et al., 2023) was derived using an optimization objective to find instructions that would maximize task accuracy. The eventual prompt was "Take a deep breath, and work on this problem step by step". TADB is an example of how the wording of a prompt can drastically impact responses.

Plan and Solve Prompting Plus: Another zero-shot prompt is Plan-and-Solve Prompting (Wang et al., 2022). There were two versions to this prompt. The first simply asked the model devise a plan and solve step-by-step. The second version (PS+) extended the prompt by specifically asking the prompt to extract relevant variables and their corresponding numerals and to calculate intermediate results. We used PS+ on our experiments. One difference between PS+ and QAP is that PS+ prompt is more specific to math datasets - as it instructs to extract variables, intermediate results etc, whereas QAP is more general. Also, PS+ prompts the model to understand the problem, but it is not clear if model should output anything specific to the question itself, but QAP explicitly instructs the model to explain the problem in n words.

Question Decomposition: Question Decomposition (Radhakrishnan et al., 2023) strategy causes the model to break down the question by creating sub-questions. The model answers each of these sub-questions and it ties together all the sub-answers into a final answer. It considers two methods for decomposition, Factored Decomposition and CoT Decomposition. In factored decomposition each sub-question is answered in a separate context. CoT decomposition is an intermediate between factored decomposition and CoT. It enforces one context for sub-question, sub-answer and the answer to the original question. The analysis of question decomposition shows reduced bias and ignored reasoning, improves the faithfulness of a model-generated reasoning over CoT while retaining the performance gains of CoT.

In this paper, we explored the approach of question analysis prompting to improve LLM accuracy across math and commonsense reasoning. The ability of this prompting method to perform well in diverse model types and tasks difficulty and type of tasks seems promising. To our best understanding, QAP is the first zero-shot prompt to introduce adaptability with a configurable parameter. We plan to extend this work further by combining QAP with other prompt strategies,  applying decoding strategies and evaluating multi-modal tasks.

There are a few limitations of QAP. First, LLMs are sensitive to the prompt’s word choice, particularly for zero-shot prompts. As a result so small changes to the prompt wording can impact the model’s performance. For example, the current QAP prompt asks the model to "solve" for the answer. While this works well for math tasks, it may not be optimal for commonsense tasks. Secondly, the results in this paper are based on four datasets and a single class of aligned models; further results should evaluate on more diverse and multi-modal datasets, as well as a greater variety of models. Finally, more robust methods (e.g., based on a classifier) to determine the choice of the parameter $n$ should be investigated to go beyond manual selection.

We experimented on three arithmetic datasets: GSM8K (Cobbe et al., 2021), AQuA (Ling et al., 2017), and AGIEval SAT Math (Zhong et al., 2023). For commonsense reasoning, used StrategyQA (Geva et al., 2021). GSM8K use the MIT License code, while AQUA and StrategyQA use the Apache-2.0 code. QAP and the prompts used in this work do not jeopardize the safety of others. They do not include any wording which may deem offensive to any individual or group.