Fine-Tuning and Prompt Optimization:
Two Great Steps that Work Better Together

While the capabilities of language models (LMs) continue to grow, recent work has shown the potential of building more powerful Natural Language Processing (NLP) systems by composing multiple skills of LMs into pipelines. Examples of this include systems for retrieval-augmented generation Guu et al. (2020); Lewis et al. (2020), multi-hop reasoning Qi et al. (2021); Khattab et al. (2021), information extraction Pourreza and Rafiei (2023); D’Oosterlinck et al. (2024), and other sophisticated pipelines Dohan et al. (2022); Khattab et al. (2022); Beurer-Kellner et al. (2023); Schlag et al. (2023).

Such LM Programs offer much more control for designing NLP systems, as they break down problems into modular, more manageable sub-tasks that can be assigned to LMs. If we could teach these LMs to accurately conduct their easier sub-tasks and to communicate effectively within multi-stage pipelines, this could greatly expand the scope of reliable NLP systems we can build.

To this end, Khattab et al. (2023) recently introduced the DSPy framework for defining and automatically optimizing LM Programs. In it, a program is defined as a function $\Phi$ that composes a set of stages, which we will refer to as language modules $M=\langle M_{1},\ldots,M_{|M|}\rangle$, into a pipeline. Each module $M_{i}$ specifies a fuzzy natural-language transformation (e.g., generating a summary of a supplied document) that needs to be learned. To do so, each module learns a particular prompt (template) $\pi$ to make a call to a particular LM with weights $\theta$. The optimization problem is then defined as maximizing the expected performance (per a downstream metric $\mu$) of the program $\Phi$ over a set of inputs by updating each module’s $\pi$ and $\theta$.

Existing work Khattab et al. (2023); Opsahl-Ong et al. (2024) has studied optimizing the discrete string prompt of each module and has considered simple approaches for fine-tuning each module’s LM weights. In this empirical study, we investigate updating each module’s prompt and LM weights together to maximize a downstream metric on the final output of the program. Doing this is challenging as $\Phi$ is not generally differentiable and its modules $M_{i}$ generally lack labeled outputs and exhibit sophisticated dependencies. Moreover, in realistic settings, the training set is usually very small and only a small number of LM calls are possible for training and inference.

To address this challenge, we propose to alternate between optimizing prompts and fine-tuning LM weights and evaluate approximate optimization strategies in which we bootstrap training labels for all pipeline modules. In experiments with multi-hop QA (HotPotQA), mathematical reasoning (GSM8K), and feature-based classification (Iris), we show that these tandem strategies are highly effective across three different LMs, leading to 5–78% gains for HotPotQA, 2.5–10% gains for GSM8K, and -5.9–136% gains for Iris against prompts only and weights only strategies, averaged across mistral-7b-instruct-v0.2, llama-2-7b-chat, and llama-3-8b-instruct.

We are given an LM program $\Phi$, which operates like a blackbox function $\Phi:\mathcal{X}\to\mathcal{Y}$, in which $\mathcal{X}$ and $\mathcal{Y}$ are typically in natural language (e.g., questions and their program generated answers, respectively). For example, we may have a program $\Phi$ for answering complex questions with short factoid answers. In the course of its execution, $\Phi$ makes one or more calls to each of $|M|\geq 1$ language modules, $M=\langle M_{1},\ldots,M_{|M|}\rangle$.

For example, the program may implement a multi-hop, retrieval-augmented pipeline for question answering. This common pipeline Qi et al. (2021); Khattab et al. (2021); Press et al. (2023); Khattab et al. (2022) breaks down the input into sub-questions that are used to iteratively find relevant passages (e.g., from a corpus like Wikipedia) until the question can be faithfully answered. In general terms, each module $M_{i}:\mathcal{X}_{i}\to\mathcal{Y}_{i}$ is a declarative LM invocation that defines, in inherently fuzzy natural-language terms, an input $\mathcal{X}_{i}$ domain (like a user-supplied question and a set of retrieved passages) and an output $\mathcal{Y}_{i}$ co-domain (like a search query to find additional relevant passages).

We seek to implement each language module as some specific, well-tuned strategy for invoking an underlying language model $\mathbf{LM}$. Concretely, we assume that a module $M_{i}$ will be fully implemented by specifying (1) the string prompt $\pi_{i}$ in which the module inputs $\mathcal{X}_{i}$ are plugged in to decode the module outputs $\mathcal{Y}_{i}$ and (2) the floating-point weights $\theta_{i}$ assigned to the parameters of $\mathbf{LM}$ in the course of this module. We refer to the version of $\Phi$ in which the prompts and LM weights are assigned explicitly to $\Pi$ and $\Theta$, respectively, as $\Phi_{\langle\Theta,\Pi\rangle}$.

Given nothing but a small training set $X=\{(x_{1},m_{1}),\ldots,(x_{|X|},m_{|X|)})\}$ of inputs $x_{i}\in\mathcal{X}$ and optional metadata like output labels or other hints $m_{i}\in\mathcal{M}$ that can be used for determining the correctness of a given program run, and a metric $\mu:\mathcal{Y}\times\mathcal{M}\to\mathbb{R}$, our goal is to optimize $\Phi$, that is, configure its modules’ prompts and LM weights to maximize the following objective:

Researchers tuning LM pipelines are in effect seeking to achieve this objective. It is also a very large subspace of the optimization problem in the DSPy framework¹¹1http://dspy.ai for LM programs. Unfortunately, this problem is intractable: we don’t have gradients or intermediate output labels to optimize each module, so we seek approximate strategies for such optimization.

We now introduce the BetterTogether algorithm, which simply alternates prompt and weight optimization steps for LM programs. We hypothesize that, when an LM is used to teach itself how to tackle the task defined by an LM program, optimizing prompts and fine-tuning LM weights are both essential to achieve the highest quality. In particular, we expect that (1) prompt optimization before fine-tuning can lead to more successful datapoints for fine-tuning and (2) prompt optimization after fine-tuning can make adjustments to the behavior of the LM program that lead to higher quality. Considering that fine-tuning is often perceived as a more powerful tool, this can be surprising, especially when both forms of optimization are ultimately applied over the same set of training inputs $X$.

Accordingly, the general optimization framework for our algorithm is defined in Algorithm 1. Given a program $\Phi$, the algorithm begins by optimizing $\Phi$’s prompts, then fine-tuning its set of LM weights, and finally optimizing its prompts again. In principle, each of these steps could be treated as optional. This will define the different possible combinations that we will seek to evaluate in Section 4. Specifically, we are interested in the quality of (1) the vanilla program $\Phi$ with simple user-supplied instructions as the prompts and no fine-tuning of $\mathbf{LM}$, (2) optimizing the prompts only, (3) optimizing the weights only, (4) optimizing the prompts twice, i.e. using the prompt-optimized $\Phi$ as a starting point for a second round of prompt optimization, (5) optimizing the weights twice, (6) optimizing the prompts then the weights, (7) vice versa, and (8) optimizing the prompts, weights, then prompts. Overall, we expect the final three to consistently outperform the first five.

For our algorithm in Algorithm 1 to be complete, we need to instantiate Lines 1–3 with specific approaches for prompt optimization and LM fine-tuning. For this, we choose the Bootstrap-$*$ family of algorithms from Khattab et al. (2023), which work by executing an initial version of the program on input examples $(x_{i},m_{i})\in X$ and recording the inputs/outputs observed at each module when the final output is “correct”, i.e., $\mu(\Phi(x_{i}),m_{i})\geq\lambda$ for some threshold $\lambda$ (e.g., $1.0$ for binary accuracy). This is important to note: in line with our formulation, our prompt and weight optimization regimes are not simply training on hand-labeled data but on self-generated program traces.

Algorithm 2 shows the instantiations for Lines 1–3 of our Algorithm 1. For prompt optimization, we use BootstrapFewshotRS (BFRS) of DSPy, which self-generates potential few-shot examples of every module and applies a form of random search (RS) to select the specific generated few-shot examples that are used for prompting. Overall, BFRS first divides $X$ into a training split $T$ and a validation split $V$ (Line 2). It then executes the provided $\Phi_{\langle\Theta,\Pi\rangle}$ on the training inputs, collecting input–output pairs for every module in $\Phi$ for each $x_{i}\in T$. This is called a trace $\tau$, and we keep only the traces assigned high scores by $\mu$ (Line 4). Given all of these traces, BFRS samples multiple different subsets of a few traces $\tau^{\prime}$ (Line 6), each of them containing a potential few-shot example for each module in $\Phi$, and ultimately selects the subset that, when used to construct few-shot prompts (Line 7) achieves the highest score (Line 8). This simple search strategy is known to consistently leads to large quality improvements in prompting LM programs Khattab et al. (2023); Opsahl-Ong et al. (2024), often outperforming manually or automatically optimizing prompt instructions or writing examples by hand.

For fine-tuning, we extend BootstrapFinetune (BFT) of DSPy, which self-generates a large number examples for every module and combines them into one dataset to finetune the LM weights with an implicit multi-task objective, where the sub-tasks are the modules’ roles. Existing work has only considered BFT in a very narrow setting for LM programs: on HotPotQA, Khattab et al. (2023) train a T5-Large model using traces from a few-shot Llama2-13b program, without considering getting an LM to teach itself via BFT nor considering a role for BFRS in the fine-tuned program. In this work, we focus on allowing models to teach themselves and self-improve. We propose for the first time combining the strategies of BFRS and BFT via alternation to get the same LM to teach itself far better than either prompt or weight optimization in isolation. (We could also test similar ideas in scenarios where a larger models does the bootstrapping for a smaller LM. This may lead to even higher results but is outside our scope.)

We now seek to evaluate our hypothesis on the importance of optimizing both prompts and LM weights of LM programs. We conduct our evaluation across three datasets that span different tasks (and thus LM programs) each. In particular, we use HotPotQA Yang et al. (2018) for multi-hop reasoning, GSM8K Cobbe et al. (2021) for arithmetic reasoning, and Iris Fisher (1988) for classification. Unless otherwise specified, we use $1000$ training set and $500$ development set examples for each dataset. We conduct our main experiments using the same model for prompt optimization, bootstrapping training traces, and fine-tuning. We experiment with three models: mistral-7b-instruct-v0.2 Jiang et al. (2023), llama-2-7b-chat Touvron et al. (2023), llama-3-8b-instruct MetaAI (2024).

We implement all of our programs and optimizers as extensions to the DSPy framework. All evaluation results are the average of three random seeds, which are used to shuffle our training sets before optimization. Full text for programs is shared in Appendix A. Appendices B and C report the license information for all LMs and datasets used as well as our implementation details (e.g., hyperparameters and software), respectively.

Table 1 reports how each of the strategies described in Section 3 perform on the held-out test sets of our datasets. Reported values are averaged across three runs with unique random seeds. Appendix D separately reports the results from each run.

In 7 out of the 9 dataset and LM pairs, we observe that the best-performing strategies are always strategies that utilize prompt ($\Pi$) and weight ($\Theta$) optimization steps together, although there is no clear winner among the three methods that optimize both. Overall, optimizing prompts is essential on all the tasks, but optimizing prompts and weights together leads to strong gains over the best setting that only optimizes one of the two.

In summary, we have proposed to alternate between prompt optimization and fine-tuning LM weights. In experiments with multi-hop QA (HotPotQA), mathematical reasoning (GSM8K), and feature-based classification (Iris), we show that our strategies are highly effective for getting an LM to teach itself to perform an LM program via bootstrapping, leading to 5–78% gains for HotPotQA, 2.5–10% gains for GSM8K, and -5.9–136% gains for Iris.

While this short paper presents strong evidence from nine case studies in total, spanning three tasks (and their corresponding LM programs) and three LMs, it is possible that other tasks, programs, or LMs will change the pattern in unforeseen ways. In particular, we have only experimented with weight optimization in the form of LoRA fine-tuning of pre-trained models. It is in principle possible that some other fine-tuning strategy would be so powerful and cost-effective as to remove the need for prompt optimization.

In addition, though we expect our findings to inform many researchers and practitioners interested in optimizing LM programs, and encourage them to explore optimizing prompts and fine-tuning LM weights together, we do not yet understand why both are important. The role of prompt optimization and the role of fine-tuning multi-stage LM programs are both new, and the relative lack of deep understanding of these roles in the emerging literature could pose risks in unanticipated interactions between these components, compared with standard gradient descent for neural networks, which has been studied for decades.

D.S. is supported by Ravi Family Graduate Fellowship. This work was partially supported by IBM as a founding member of the Stanford Institute for Human-Centered Artificial Intelligence (HAI), and by the HAI Hoffman–Yee Grant “Dendritic Computation for Knowledge Systems”.