Weak-to-Strong Reasoning

We outline common learning paradigms in large model training, primarily characterized by the need for ground truth answers and supervision from stronger models as shown in Tab. 1.

In this paper, we address reasoning tasks in the weak-to-strong setting, as illustrated in Tab. 2. First, we examine mathematical reasoning tasks, such as those in GSM8k and MATH. These tasks require each step of the reasoning process to demonstrate fundamental mathematical problem-solving skills, including problem comprehension and algebraic operations, and build upon the previous steps. It imposes higher demands on the model’s learning and generalization capabilities. Unlike classification tasks, where models can rely on superficial pattern extrapolation or recognition, reasoning tasks offer minimal benefit from guessing. Then, we use a weak model (e.g., Llama2-7b) with a certain degree of mathematical problem-solving ability,²²2Otherwise, the weak model can hardly provide useful supervision. denoted as $m$. This model acts analogously to human supervisors with limited expertise in the era of superintelligence. Besides, we only have a set of questions $\mathcal{Q}=\{q_{i}\}$ without ground truth answers and the goal is to improve the reasoning capability of a strong model $\mathcal{M}$ (e.g., Llama2-70b). To implement this, following Burns et al. (2023), we randomly divide the original training set into two equal parts, $\mathcal{D}_{\text{gold},1}$ and $\mathcal{D}_{\text{gold},2}$. The weak model is initially fine-tuned using $\mathcal{D}_{\text{gold},1}$ where the gold solutions are available, resulting in a weak model with some problem-solving capability, i.e. $m$. In contrast, the strong model can only access the questions from $\mathcal{D}_{\text{gold},2}$, without reasoning chains or final answers, i.e., $\mathcal{Q}$.

Given a weak model $m$ and a series of math problems $\mathcal{Q}$ without ground truth, $m$ generates weak data $\mathcal{D}_{\text{weak}}=\{q_{i},c_{\text{weak},i},a_{\text{weak},i}\}$, where $q_{i}\in\mathcal{Q}$, $c_{\text{weak},i}$ represents a reasoning chain, and $a_{\text{weak},i}$ represents the final answer. The correctness of $a_{\text{weak},i}$ is unknown. The central challenge is: how can we maximize the use of $m$ and $\mathcal{D}_{\text{weak}}$ to fully enhance and recover the mathematical reasoning capabilities of a stronger model $\mathcal{M}$?

We denote the final iteration of $\mathcal{M}_{\text{hybrid-ft}}$ from Stage I as $\mathcal{M}_{\text{plus}}$, which has learned dual mathematical solutions and holds potential for further enhancement. Next, we apply preference optimization techniques to strategically utilize the potentially erroneous subset of the original weak dataset $\mathcal{D}_{\text{weak}}=\{q_{i},c_{\text{weak},i},a_{\text{weak},i}\}$ generated by $m$, which allows the strong model to identify and avoid similar errors in future reasoning processes. The key factor lies in how to construct contrastive samples for learning.

Without access to ground truth, the current strong model with enhanced reasoning capabilities identifies the most likely correct answers based on its confidence. Specifically, for each question $q_{i}\in\mathcal{Q}$, we sample $n$ responses from $\mathcal{M}_{\text{plus}}$, and define the probability of the answer that appears most frequently among these responses as confidence. When the confidence falls below a specified threshold $\tau$, we consider the model’s judgment on this question unreliable and therefore discard it. Conversely, if the confidence is no less than $\tau$, we regard the model as capable of solving the question and proceed to construct contrastive samples as follows.

• •

  For a question $q_{i}$ where $\mathcal{M}_{\text{plus}}$ is confident, we denote the most confident answer as $a_{\text{strong},i}^{+}$ and $P(a_{\text{strong},i}^{+})\geq\tau$. It can be considered as the “correct” answer according to $\mathcal{M}_{\text{plus}}$. For instance, if we set $\tau=0.6$ and 8 out of 10 sampled responses have the same final answer “4.2”, we say that $\mathcal{M}_{\text{plus}}$ considers “4.2” to be the correct answer to this question, i.e. $a_{\text{strong},i}^{+}=4.2$.

• •

  Then we divide the sampled $n$ responses of $\mathcal{M}_{\text{plus}}$ to $q_{i}$ into two sets: $A_{\text{strong},i}^{+}=\{c_{\text{strong},i}^{j},a_{\text{strong},i}^{j}\}$ where $a_{\text{strong},i}^{j}=a_{\text{strong},i}^{+}$; $A_{\text{strong},i}^{-}=\{c_{\text{strong},i}^{k},a_{\text{strong},i}^{k}\}$ where $a_{\text{strong},i}^{k}\neq a_{\text{strong},i}^{+}$. In the above example, $|A_{\text{strong},i}^{+}|=8$ and $|A_{\text{strong},i}^{-}|=2$.

• •

  If the weak model holds an answer that the enhanced model considers “correct”, that is, $a_{\text{weak},i}=a_{\text{strong},i}^{+}$, we treat the weak model’s response $\{c_{\text{weak},i},a_{\text{weak},i}\}$ as chosen response and randomly select a rejected response from $A_{\text{strong},i}^{-}$. Otherwise, if $a_{\text{weak},i}\neq a_{\text{strong},i}^{+}$, we treat $\{c_{\text{weak},i},a_{\text{weak},i}\}$ as rejected response and randomly select a chosen response from $A_{\text{strong},i}^{+}$. Examples are shown in Tab. 3.

Further training $\mathcal{M}_{\text{plus}}$ on these samples enables it to distinguish between correct and incorrect solutions, leading to a stronger model $\mathcal{M}_{\text{pro}}$.

GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021) are two widely used datasets for mathematical reasoning, and MATH comprises more challenging competition problems. The data statistics we use are presented in Tab. 4. Particularly, to ensure a sufficient amount of training data for developing preliminary mathematical skills in the weak model, we augment the GSM8K training set with the data constructed by Chern et al. (2023). Further details are available in §A.1.

We use Llama2-70b as the strong model and employ three weak models from different families: Llama2-7b, Gemma-2b, and Mistral-7b. We apply full parameter fine-tuning to the weak models on $\mathcal{D}_{\text{gold},1}$, and consistently adopt LoRA (Hu et al., 2022) to fine-tune the strong model. In Stage I, we perform two rounds of iterations on GSM8K and one round on MATH according to the principles of iteration outlined in §3.1. In Stage II, we adopt two preference learning-based approaches, DPO (Rafailov et al., 2023) and its variant ORPO (Hong et al., 2024). Details are provided in §A.2.

We evaluate the accuracy on the test set. The performance of the weak model $m$ is defined as the “weak floor”. The performance of the strong model $\mathcal{M}$, fine-tuned with data containing gold solutions from $\mathcal{D}_{\text{gold},2}$, is termed the “strong ceiling”. It represents the upper limit of the capabilities that the strong model can achieve with $\mathcal{D}_{\text{gold},2}$.

The main results of Stage I on both GSM8K and MATH datasets are depicted in Fig. 4. Notably, in the MATH experiments, we randomly sample additional data that is not chosen based on the final answer consistency, due to the small amount available. Please refer to §A.4.1 for details. According to Fig. 4, we have the following observations.

Weak-ICL fine-tuning demonstrates a notable enhancement. Using our proposed method, the performance of the strong model, supervised only by the weak Gemma-2b with 25.17 accuracy on GSM8K (without any gold answers), can be improved up to 60.12, surpassing naive full weak fine-tuning by 31.08, and $\mathcal{M}_{\text{plus}}$ (i.e., $\mathcal{M}_{\text{hybrid-ft}}^{2}$) outperforms it by 26.99. This verifies the effectiveness of data refining before supervised fine-tuning. Also, experimental results show that the mathematical reasoning capabilities of the strong model are increasingly recovered as the weak model improves, a conclusion verified by Liu and Alahi (2024) on classification tasks. In detail, the performance on GSM8K gradually improves for Gemma-2b, Llama-7b, and Mistral-7b ($25.17\to 33.81\to 59.51$). Hence, the maximum performance of the strong model, trained with data generated by these models, also progressively enhances ($60.12\to 63.76\to 68.39$).

$\mathcal{M}_{\text{hybrid-ft}}$ achieves the highest pass@k scores. As expected, $\mathcal{M}_{\text{hybrid-ft}}$ achieves the highest pass@k scores in the weak-to-strong setting, benefiting from its training data that incorporates two types of solutions—one from the weak model, and another from the strong model. This diversity enhances the robustness of the model by reducing the likelihood of overfitting. Additionally, the performance of $\mathcal{M}_{\text{icl-ft}}$ generally surpasses that of $\mathcal{M}_{\text{weak-ft}}$, which can be attributed to variations in process-level accuracy and possibly the solution format. Detailed analyses are conducted in §A.3.

Naive fine-tuning is inadequate for weak-to-strong reasoning. When using Gemma-2b as the weak model on the MATH dataset, full weak fine-tuning underperforms compared to the weak floor (10.0 v.s. 11.6). This indicates that naive fine-tuning, though successfully applied to classification, chess, and reward modeling tasks (Burns et al., 2023), falls short for intricate reasoning tasks, particularly those of substantial difficulty like questions in MATH. In contrast, our weak-icl fine-tuning method effectively bridges the gap, offering an effective and scalable solution for the weak-to-strong reasoning challenge.

As discussed in §3.2, we employ the final iteration of $\mathcal{M}_{\text{hybrid-ft}}$ as $\mathcal{M}_{\text{plus}}$ for subsequent preference learning. The experimental results in §4.3 validate this checkpoint achieves higher pass@k and possesses significant potential for further refinement.

As shown in Tab. 5, our method for constructing positive and negative samples effectively enhances the strong model’s math reasoning capabilities. On GSM8K, both DPO and ORPO consistently achieve significant improvements using our constructed datasets, notably resulting in an increase of 8.49 points when supervised by Gemma-2b. Despite the inherently challenging nature of MATH problem, which compromises the strong model’s judgment and introduces inaccuracies in the training data, our method still achieves improvements on MATH through ORPO by at least 1 point.⁴⁴4Pang et al. (2024); Xu et al. (2024); Yuan et al. (2024) demonstrate that DPO can cause performance degradation on MATH due to the lack of regularization in its loss.

For further analysis, we examine the accuracy across different difficulty levels in the MATH test set (See §A.1.2 for data statistics).

As shown in Fig. 6, the strong model exhibits better generalization on easier problems. Specifically, even though Llama2-7b achieves only 6.98 points accuracy on level 1 problems, Llama2-70b can achieve an accuracy exceeding 30 points after training using this weak supervision. For more challenging problems (levels 4-5), $\mathcal{M}_{\text{pro}}$, enhanced with ORPO, even surpasses the strong ceiling obtained by supervised fine-tuning solely on gold solutions. This phenomenon serves to validate the effectiveness of learning from incorrect data.

In preliminary tests with Llama3-70b (AI@Meta, 2024), we observe that on GSM8K and MATH, Llama3-70b can largely unlock its potential through in-context learning, with marginal or even adverse impacts from parameter updates due to training instabilities. Consequently, we focus on a more challenging dataset developed after the release of Llama3-70b, OlympicArena (Huang et al., 2024), to simulate a more realistic future scenario.

We only consider English questions in OlympicArena, excluding the CODE (Code Generation) and OT (Others) problem types that require case-based or expert evaluation. This results in 6,020 training data without solutions and final answers, and 313 test data with final answers to assess the performance of different methods. We use Llama3-8b-instruct (without initial fine-tuning on a subset of training data) as the weak model and Llama3-70b as the strong model to be improved. The hyperparameters are consistent with those used for GSM8K. This configuration more closely resembles future real-world weak-to-strong scenarios.

Experimental results are displayed in Tab. 6. “Weak Floor” represents the zero-shot performance of Llama3-8b-instruct, “Full Weak FT” denotes the performance of Llama3-70b after supervised fine-tuning on the full set (i.e, 6,020) of weak solutions generated by Llama3-8b-instruct on the training set, and “Weak ICL” indicates the performance of Llama3-70b under 4-shot weak demonstrations generated by Llama3-8b-instruct. Despite having more parameters, Llama3-70b under in-context learning still performs lower than the zero-shot performance of Llama3-8b-instruct due to insufficient mining capabilities.

$\mathcal{M}_{\text{weak-ft}}^{1}$, obtained by our proposed weak-icl fine-tuning method, achieves higher performance than Full Weak FT with fewer training data (i.e., 746), outperforming it by 0.32 points. After the second stage of preference optimization, which further exploits the weak model and training questions without answers, the strong model’s performance is improved by an additional 3.19 points over Full Weak FT. This demonstrates the robustness and generalizability of our method in scenarios closer to future conditions.

LLMs can enhance their ability to solve tasks and better align with human instructions through a supervised fine-tuning (SFT) phase (Zhang et al., 2023; Dong et al., 2023a; Lv et al., 2023b, a). This phase heavily relies on the quality of training data, as previous studies (Zhou et al., 2023a; Wang et al., 2023b) demonstrate that higher data quality translates to substantial gains in model performance. In this paper, we investigate the potential of learning from weak supervisions.

To further align LLMs with human values and enable learning from both positive and negative feedback, additional training is required, such as reinforcement learning from human feedback (RLHF, Ouyang et al. (2022); Bai et al. (2022)) and direct preference optimization (DPO, Rafailov et al. (2023)). In particular, DPO reparameterizes reward functions in RLHF and has been widely used due to its simplicity. Several variants of DPO have then emerged to further enhance its stability and performance, such as ORPO (Hong et al., 2024) and SimPO (Meng et al., 2024), etc. This paper explores the capabilities of DPO and ORPO using our constructed contrastive samples in a weak-to-strong setting.

The exploration of mathematical reasoning within LLMs has been a focal point for evaluating their cognitive capabilities akin to human reasoning (Qiao et al., 2023; Lu et al., 2023). Researchers have developed various methods to enhance the mathematical reasoning capabilities of LLMs after pre-training, which can be broadly classified into two categories: (1) Prompting: Some works (Kojima et al., 2022; Wei et al., 2022; Zhou et al., 2023b; Liu et al., 2023) aims to elicit the intrinsic reasoning abilities of LLMs by specific prompting engineering, without updating the model parameters; (2) Fine-tuning: Another line of studies focuses on generating a more extensive and higher-quality collection of question-answer pairs (Yu et al., 2023; Wang et al., 2023c, a). Through supervised fine-tuning and preference optimization (Luo et al., 2023; Azerbayev et al., 2023; Mitra et al., 2024; Xu et al., 2024), the models can achieve significant improvements in their mathematical problem-solving capabilities.