MaxMin-RLHF:
Towards Equitable Alignment of Large Language Models with Diverse Human Preferences
Abstract
Reinforcement Learning from Human Feedback (RLHF) aligns language models to human preferences by employing a singular reward model derived from preference data. However, such an approach overlooks the rich diversity of human preferences inherent in data collected from multiple users. In this work, we first derive an impossibility result of alignment with single reward RLHF, thereby highlighting its insufficiency in representing diverse human preferences. To provide an equitable solution to the problem, we learn a mixture of preference distributions via an expectation-maximization algorithm and propose a MaxMin alignment objective for policy learning inspired by the Egalitarian principle in social choice theory to better represent diverse human preferences. We elucidate the connection of our proposed approach to distributionally robust optimization and general utility RL, thereby highlighting the generality and robustness of our proposed solution. We present comprehensive experimental results on small-scale (GPT-2) and large-scale language models (with Tulu2-7B) and show the efficacy of the proposed approach in the presence of diversity among human preferences. Our algorithm achieves an average improvement of more than 16% in win-rates over conventional RLHF algorithms and improves the win-rate (accuracy) for minority groups by over 33% without compromising the performance of majority groups, showcasing the robustness and fairness of our approach. We remark that our findings in this work are not only limited to language models but also extend to reinforcement learning in general.
1 Introduction
The alignment problem, central to developing and fine-tuning current large language models (LLMs), represents a crucial challenge in artificial intelligence, especially in ensuring these models operate in harmony with human values, preferences and social welfare (Wang et al., 2023; Christian, 2020). Reinforcement learning from human feedback (RLHF) has emerged as a pivotal approach to alignment problems, specifically aligning LLM (Wang et al., 2023; Ouyang et al., 2022b; Stiennon et al., 2022a; Ouyang et al., 2022a). RLHF operates in three steps (a) supervised fine-tuning, (2) reward learning, and (3) RL fine-tuning. Step 2 learns a reward function that is expected to represent the preference feedback of the human population. However, there has been minimal emphasis on accurately representing the diversity of human preferences and the broad spectrum of user populations. As highlighted by Aroyo & Welty (2015); Aroyo et al. (2023a, b), “the notion of ‘one truth’ in crowdsourcing responses is a myth” and we need to account for the diversity in opinions and preferences.
Despite the criticality, most of the latest RLHF approaches ignore the consideration of the diversity in human preference feedback by aligning the language model with a single reward function (Wang et al., 2023; Christian, 2020; Stiennon et al., 2022a; Ouyang et al., 2022a). This assumption of a single ground truth reward is restrictive and can potentially subdue the preferences or opinions of minority groups, leading to societal biases (Figure 1). To mitigate this issue, some of the recent research proposes to learn multiple reward functions, which can then be aggregated in arbitrary manners (Bakker et al., 2022; Jang et al., 2023). On the other hand, (Ovadya, 2023) adopts a consensus-based method for aggregating human representations by emphasizing specific principles (Bai et al., 2022b; Kovač et al., 2023), which might result in the under-representation of marginalized groups (Ramé et al., 2023). Another line of research focuses on the aspect of designing multi-policy strategies by fine-tuning personalized language models towards individual rewards (Jang et al., 2023; Ramé et al., 2023; Ji et al., 2023a).
Some recent literature has brought attention to the challenge of aligning single utility RLHF with diverse preferences (Bakker et al., 2022; Ramé et al., 2023; Jang et al., 2023). However, a thorough understanding of how the diversity within human sub-populations influences the overall alignment objective remains elusive. Consequently, this prompts us to pose the following question:
Is the current single reward RLHF pipeline sufficient to align with diverse human preferences?
In this work, we present negative results for the above question by demonstrating the impossibility of alignment using single reward RLHF (Theorem 3.3). We introduce a notion of diversity between human subpopulations due to the differences in preference distributions and establish lower bounds on the alignment performance of single reward RLHF. However, this impossibility result naturally raises another important question:
What strategies can we design to efficiently align with diverse human preferences?
In response to this question, we draw inspiration from the Egalitarian rule (Sen, 2017) to provide an equitable solution to the problem and aim to maximize the social utility objective for alignment. We propose learning a mixture of preference distributions via expectation-maximization algorithm in absence of human sub-population group information. Under the mixture distributions, we introduce the notion of MaxMin RLHF, for policy learning inspired by the Egalitarian principle in social choice theory to better represent human preferences under diversity. We elucidate the connection of our proposed approach to distributionally robust optimization and general utility RL, thereby highlighting the generality and robustness of our proposed solution. We summarize our contributions as follows.
(1) An impossibility result of alignment with single reward-based RLHF. We first introduce the notation of diversity as a divergence between the human sub-population preference distributions (Definition 2) and derive lower bounds on the reward model sub-optimality (Lemma 3.2) in terms of the diversity. Finally, we establish a lower bound (Theorem 3.3) on the alignment gap due to the diversity in the human preference feedback. True to our knowledge, ours is the first work to formally characterize the sub-optimality gap with conventional RLHF methods under diverse preferences.
(2) Max-Min RLHF alignment with diverse user preferences. To provide an equitable solution to the problem, we learn a mixture of preference distributions using the Expectation-Maximization (EM) algorithm (Algorithm 2) and propose a novel MaxMin-RLHF algorithm as an egalitarian strategy to align language models with social utility objectives under diverse preferences. (Algorithm 1).
(3) A comprehensive empirical study. We present a detailed empirical analysis of our proposed approach on both small (GPT-2) and large language models (Tulu-7B). Initially, we provide empirical evidence highlighting the impossibilities of alignment with single reward RLHF, followed by demonstrating the feasibility and effectiveness of MaxMin-RLHF in achieving social utility objectives. Our algorithm achieves an average improvement of more than 16% in win-rates over conventional RLHF algorithms and improves the win-rate (accuracy) for minority groups by over 33% without compromising the performance of majority groups, showcasing the robustness and fairness of our approach.
1.1 Related Works
The current RLHF approaches (Stiennon et al., 2022b; Ziegler et al., 2020; Zhu et al., 2023) involve training a reward model based on human preference feedback and then fine-tuning the language model using proximal policy optimization (PPO) (Schulman et al., 2017). Besides PPO, DPO (Direct Preference Optimization, Rafailov et al. (2023)) directly trains the large language model using human preferences without training the reward model. A self-play-based approach such as SPIN (Chen et al., 2024) is similar to DPO but has an iterative framework. However, most of the existing alignment approaches only consider the average preference by human annotators and ignore the inherent diversity among human preferences (Casper et al., 2023; Kaufmann et al., 2023). Diverse human preferences stem significantly from various factors related to social and cultural backgrounds (Aroyo et al., 2023b, a; Denton et al., 2021a). The key factors contributing to this diversity include (i) socio-demographic backgrounds (Vogels, 2021), (ii) personal bias and context subjectivity (Denton et al., 2021b; Sandri et al., 2023)), (iii) Imperfect preferences, (Sandri et al., 2023), and (iv) Linguistic ambiguity & missing context (Sandri et al., 2023; Denton et al., 2021b; Sap et al., 2022). These factors collectively underscore the complexity of aligning LLM outputs with the diverse preferences of human users, demonstrating the importance of recognizing and addressing the multifaceted nature of user feedback. Appendix B provides a detailed context of related works.
2 Preliminaries
Let us start by defining a language model mathematically. We denote a vocabulary set as and a language model by a mapping (parameterized by ). A language model takes a sequence of tokens (called prompt) as input denoted by , where each token . The prompt , where is the set of prompts, is fed as input to the language model, and it generates output response .
RLHF pipeline. We start by considering the RLHF pipeline in Ziegler et al. (2019), which has also been adopted in subsequent works (Stiennon et al., 2022c; Bai et al., 2022a; Ouyang et al., 2022b). It consists of three steps detailed as follows:
Step 1: Supervised Fine-tuning (SFT): In this phase, a generic pre-trained LM is fine-tuned with supervised learning on a high-quality dataset for the downstream task(s) of interest, such as dialogue, instruction following, summarization, etc., to obtain a model .
Step 2: Reward Modelling: In the second phase, the SFT model is queried with prompts to produce pairs of responses which are then presented to human labelers for preference evaluation, and , denotes the preferred and dispreferred response, respectively. The preference distribution under the Bradley-Terry (BT) preference model (Bradley & Terry, 1952) is written as
(1) |
where is the latent reward model. With a static dataset sampled from , we can learn a parameterized reward model via maximum likelihood estimation. Framing the problem as a binary classification, we have the negative log-likelihood loss:
(2) |
where is the logistic function.
Step 3: RL Fine-Tuning: In the final step, the optimal policy under the reward is obtained by solving the KL-regularized reward maximization problem given by
(3) |
where, controls the deviation from the base reference policy .
3 An Impossibility Result for Single Reward RLHF with Diverse Preferences
In this section, we mathematically prove the impossibility of aligning language models with diverse human preferences with the single reward RLHF framework. We start by discussing the motivation and mathematical definition of diversity in human preferences in Section 3.1, then connect the reward learning step of the RLHF pipeline with diversity in Section 3.2, and then finally prove the impossibility of language model alignment in Section 3.3 by connecting Step 3 of RLHF pipeline with human preference diversity.
3.1 Diversity in Human Preferences
The main shortcoming of state-of-the-art alignment approaches arises from the underlying assumption that human preferences are derived from a single latent reward model (cf. (2)), which fails to account for the inherent diversity among the human sub-populations (see Figure 2). As discussed in Section 1.1, one of the key reasons for the diverse human preferences is the varied socio-demographic and socio-cultural backgrounds of human sub-populations (Aroyo et al., 2023b, a). For example, population groups with diverse demographic markers such as race, ethnicity, age groups, genders, etc., have highly varied preferences as highlighted in (Aroyo et al., 2023b, a; Denton et al., 2021a). Such diversity inevitably leads to natural sub-groups of populations among humans. Modeling this diversity in preferences for the fine-tuning of language models in RLHF is crucial, which, to the best of our knowledge, is currently missing from the literature.
Sub-population Preference Distributions: Let us consider the human population providing the preference feedback represented by . We can write the preference distribution (Stiennon et al., 2022a; Ouyang et al., 2022a) as
(4) | ||||
where is the probability of preferring over for any given pair corresponding to prompt . In (4), the expectation is over a finite set of humans . We next introduce the concept of human subpopulations as a hidden random variable, denoted as with distribution , to account for the inherent diversity within the population. Specifically, represents the human subpopulation defined over a finite discrete set , such that . The cardinality of the set represents the number of sub-populations/groups present in the total human population . Therefore, similar to (4), we can define a human-subpopulation or group-specific preference distribution for a given pair of responses and prompt as
(5) | ||||
for all groups in . Next, we define the preference diversity among the human population in Definition 2 as follows.
[Diversity in Human Preferences] Consider a human population , composed of sub-population groups where , and a sub-population-specific preference as defined in (5), we define the diversity of sub-population group with respect to other group as
(6) |
where TV denotes the total variation distance between two preference distributions. By utilizing the definition of sub-population groups in , we can express the preference in (4) as
(7) |
where is a shorthand notation and denotes the distribution over the humans . Here, is the sub-population specific preference distribution (cf. (5)) and represents the marginal probability distribution of sub-population and quantifies the probability of occurrence of sub-population to provide feedback for pair z. We can think of as a weighting function that quantifies the relative importance of each sub-population (say ) within the full population reflecting their contributions to the aggregate preference distribution . Thus, from the expansion in (3.1), it is evident that the preference distribution under consideration is a weighted sum of sub-population specific preference distribution, weighted by . We remark that distributions and are crucial to rigorously characterize the alignment performance of different approaches, which is not considered in the existing literature (Christian, 2020; Bai et al., 2022a).
3.2 Reward Mismatch Due to Diversity
From equations (1) and (2), we note that the existing RLHF approach focuses on learning the ground-truth single reward parameter to represent the preference distribution by minimizing the cross-entropy loss (cf. (2)) given by
(8) |
The assumption of single ground-truth reward (corresponding to ) which is violated due to the existence of diverse sub-populations with separate preference distributions, as discussed in Section 3.1. This would lead to an implicit aggregation as shown in (3.1) and the equivalent MLE objective in (3.2) can be re-written as :
(9) |
Now, expanding upon the cross-entropy objective, we note (see Lemma C for details) that the objective in (3.2) essentially reduces to minimizing the Kullback-Leibler (KL) divergence and the objective is minimized at . This implies that by minimizing the loss function in (3.2), when we try to learn a single to recover , an implicit averaging happens over the preferences of human subpopulation groups they belong to, which plays a critical role in the sub-optimality in reward learning summarized in Lemma 3.2. {lem}[] Let denotes the reward parameter, which models (cf. 1) and models the human sub-population group specific , it holds that
where denotes the weights distribution across human sub-population groups, denotes the upper bound on the feature representation for all , and diversity as defined in Definition 2. Proof Sketch. Here we describe the proof sketch of Lemma 3.2 with a detailed proof provided in Appendix D. We begin with the definition of sub-optimality in the learned reward for a subpopulation group as where which is the approximation to the true parameter which minimizes the loss function in (2). However, we know in the limit of infinite data, converges to and hence we focus on the sub-optimality gap due to diversity as . Using the Lipschitzness of the preference probability distribution under the Bradley-Teryy preference model (derived in Lemma C in Appendix) we lower-bound the sub-optimality gap by and finally expanding upon the definition of as shown in (3.1), we get the final result.
Remark. Lemma 3.2 indicates that the current RLHF-based reward learning paradigm (Christian, 2020; Bai et al., 2022a; Rafailov et al., 2023) will suffer sub-optimality due to diversity amongst the humans, which is highly likely in practice (Aroyo et al., 2023b). Lemma 1 implies that the degree to which the learned reward parameter diverges from optimality for a given subgroup is influenced by two key factors: the distinctiveness of that subgroup’s preferences compared to all the other subgroups, and the relative weight assigned to the subgroup in the overall preference model.
3.3 An Impossibility Result of Alignment
To mathematically characterize the impossibility of aligning the language model with diverse sub-population groups, let us reconsider the RL fine-tuning optimization problem, which is given by (step 3 in RLHF in Section 2)
(10) |
where we define . Let us define where is the optimal aligned policy with single reward RLHF. On the other hand, we define a human sub-population specific optimal policy as , where is the optimal aligned policy with individual subpopulation group . We define the alignment gap of RLHF model to a specific user group by
(11) |
We note that the alignment gap defined in (11) measures the discrepancy between the reward returns by the single reward RLHF model and the optimal model tailored for subpopulation evaluated under true reward function . Next, we present our impossibility result in Theorem 3.3. {thm}[An Impossibility Result] Let denotes the reward parameter, which models (cf. 1), denotes the human sub-population group specific reward function to model , and alignment gap is as defined in (11). Then, it holds that
(12) |
where denotes the weights distribution across human sub-population groups, denotes the upper bound on the feature representation for all , denotes the minimum eigenvalue of the feature matrix, is the regularization parameter of RLHF framework, and diversity as defined in Definition 2. A detailed proof of Theorem 3.3 is provided in Appendix E. We briefly describe the proof sketch of Theorem 3.3 as follows.
Proof Sketch. We begin by considering the KL-regularized alignment objective (cf. (3)). Utilizing the strong concavity of the objective under the KL regularization and the analytical mapping from reward functions to optimal policies (as used in DPO (Rafailov et al., 2023)), we first derive a lower bound on the alignment gap as . Under the linear parametrization in reward and utilizing the boundedness on the representation space, we can lower-bound the alignment gap with the reward sub-optimality and eventually the diversity coefficient.
Remark. Theorem 3.3 shows that high subpopulation diversity inevitably leads to a greater alignment gap for the model aligned with single reward-based RLHF framework. In summary, if a subgroup exhibits distinctive preferences or constitutes a minority with a smaller representation, the resulting model from single reward RLHF setting cannot accurately reflect the sub-population’s specific preferences. We provide empirical evidence of impossibility of alignment in Figure 6.
4 Proposed Approach: Alignment with Diverse Preferences
From the statement of Theorem 3.3, we note that it is impossible to align the language model with diverse human preferences with a single reward model based RLHF. Now the natural question arises is what can we do in practice to deal with the diverse human preferences alignment problem? This is exactly what we answer in this section. We start by proposing to learn a mixture of preference distributions as follows.
4.1 Learning a Mixture Distribution for Diverse Preferences
To deal with the diversity in preferences, we first propose learning a mixture distribution for preferences rather than a single preference distribution, which is prevalent in the current RLHF pipelines (Christian, 2020; Ouyang et al., 2022b). The intuition behind learning a mixture of preference models is rooted in the principles of statistical learning theory (Lindsay, 1995; Seidel, 2011) which states that a combination of multiple distributions can approximate a wide range of complex, real-world distributions more effectively than a single model. This concept is akin to the property for mixture models, which suggests that a mixture model can approximate any number of diverse user sub-populations to an arbitrary level of accuracy, given a sufficient number of components and appropriate parameterization (Goodfellow et al., 2016). Thus, the preference distribution as shown in (3.1) can be expressed as
(13) |
where the event , are the chosen and rejected response by the human sub-population-group . With the definition in (13), the log-likelihood maximization objective to learn the sub-population preference distributions parametrized by (under the Bradley Terry preference model in (1)) and the groups/clusters can be written as
(14) | ||||
We learn the parameter by maximizing the above log-likelihood objective. It is important to emphasize that mixture models do not require prior knowledge of user group classifications, i.e., which user belongs to which group, and can learn the distribution of preferences in an unsupervised manner. This capability underscores the generality and flexibility of mixture distributions in capturing the diversity of user preferences without explicit user group information.
Remark. If we have access to individual human sub-population information, we can directly learn the reward parameter via maximizing likelihood per user group. But often, in practice, they are hardly available. To address this challenge, we consider an expectation-maximization algorithm to learn a mixture of reward models summarized in Algorithm 2 which learns the ’s and the clusters. We summarize the EM algorithm for reward learning in Algorithm 2.
4.2 MaxMin RLHF
We begin by noting that while we circumvent the sub-optimality in reward learning (cf. Lemma 3.2) sub-population specific reward models for all subsets in using Algorithm 2, this does not address the ultimate goal of language model alignment. Since the eventual objective is to develop a single policy that accommodates diverse user preferences while avoiding bias towards specific groups such as minorities. To achieve that, we take motivation from the Egalitarian rule in social choice theory (Sen, 2017), which states that society should focus on maximizing the minimum utility of all individuals. Hence, we write our proposed alignment objective, which maximizes the social utility as follows {defi}(Alignment with Diverse Human Preferences) From the mixture of preference distributions in (14), we get where denotes reward model parameter for each human subpopulation in . We define the alignment objective in such diverse settings as
(15) |
where the expectation is with respect to .
Proof of Concept: An illustration of MaxMin RLHF. To illustrate the social utility of the Max-Min objective within the context of RLHF, in Figure 4, we present a simplified robotic scenario. In this scenario, the robotics agent’s goal is to distribute goods among two user groups located at green and orange boxes, respectively. Each user group would prefer that the agent visit them and distribute goods. Hence, we define the reward for each user group is defined as follows: the agent receives a reward if it reaches the goal state for the user and , otherwise. We demonstrate that in this basic scenario, our proposed max-min algorithm ensures that the agent serves all the user groups irrespective of the difference in utility between the user groups . Mathematically, from the Definition 4.2, we can write
(16) |
where we define for the compactness of notation and denotes the state in the gridworld and denotes the actions. From the definition of the value function as an inner product of the state-action occupancy and the reward (Bertsekas, 2019; Sutton & Barto, 2018), we can write (16) as
(17) |
where denote the occupancy measure corresponding to policy . Let us focus on the minimization objective , where signifies the goal state and action for a user , and is the corresponding reward. When the policy does not visit the goal for user , the value of is zero, making the minimum always zero. Therefore, the maximum objective aims to learn a policy that maximizes the reward with respect to the users until for all users. Even when the difference is significantly large, highlighting a disparity between majority and minority user groups, the max-min approach effectively learns a policy that respects the needs of both users, demonstrating its social utility as shown in Figures 4.
MaxMin RLHF Algorithm. Next, we introduce the algorithm to solve the MaxMin RLHF proposed in Definition 4.2 in Algorithm 1. After learning the reward in Step 1 for individual user sub-population using Algorithm 2, we next compute the minimum utility user group as . Once the minimum utility user subpopulation group is determined, we fine-tune the language model policy for that group by taking gradient steps at each iteration as
(18) |
where and denotes the step size. Note that for the implementation in practice, we leverage the PPO algorithm (Schulman et al., 2017) as commonly used in existing RLHF pipelines (Stiennon et al., 2022b; Christian, 2020; Ouyang et al., 2022b).
4.3 Connection with DRO and General Utility RL
In this subsection, we highlight the generality of our proposed formulation beyond social utility design to broader formulations of general utility reinforcement learning (Zhang et al., 2020) and distributionally robust optimization (Duchi et al., 2018). We discuss then in details follows.
1. Distributionally Robust Optimization: We begin by noting that once we have learned the mixture of reward models for all user-groups using Algorithm 2, we can formulate the problem of learning a robust policy as a distributionally robust optimization (DRO) (Ben-Tal et al., 2013; Duchi et al., 2018). In DRO, the objective is to maximize the worst-case expected utility over an uncertainty set of distributions. In our scenario, this boils down to constructing an uncertainty set over the distribution of rewards . Now, expanding the RL fine-tuning objective in the equation
(19) |
where as obtained from equation (3). The uncertainty set is typically obtained by constructing a distribution across the reward distributions , however, this may lead to an overtly pessimistic solution. Hence, recent literature (Sagawa et al., 2020) has shown leveraging the prior knowledge of user-group information, can mitigate this by constructing the uncertainty set in terms of these groups , also referred to as the group distributionally robust optimization (Group-DRO). In Group DRO, the uncertainty set over reward distribution is constructed by a mixture of groups as , where represents the dimensional probability simplex of preference distributions. However, since the inner problem is a linear program and the optimum is attained at the vertex, the resulting group DRO problem boils down to solving
(20) | ||||
where represents the reward distribution of the user sub-population group . It is important to note that the group-DRO formulation boils down to the Max-Min RLHF formulation under this selection of uncertainty set under the distributional reward setting. This highlights that our proposed MaxMin RLHF provides a socially fair and distributionally robust solution to alignment problem with diverse user groups.
2. General Utility Reinforcement Learning: Next, we highlight the connection of our proposed formulation to general utility reinforcement learning as proposed in (Zhang et al., 2020, 2021). Zhang et al. (2020) first proposed the notion of general utility reinforcement learning that goes beyond the standard setting of cumulative reward maximization to more general utilities where standard notions of Bellman’s equation, the value function, etc fail. Thus in our formulation with diverse user preferences, once we have learned the reward models for all user-groups using Algorithm 2, we can formulate the general utility objective as the minimizer over the value-functions of user groups as . where, as obtained from equation (3). With this, the general utility formulation boils down to
(21) |
Hence we can utilize the analysis developed in (Zhang et al., 2020, 2021) for the general utility RL to derive the theoretical convergence results for our proposed algorithm in this work.
5 Experimental Results
In this section, we present a comprehensive empirical evaluation of the alignment impossibilities and our proposed solutions for language models, structured into two distinct subsections: Small Scale experiments (Sec. 5.1) for initial proof of concept, and Large Scale experiments (Sec. 5.2) for broader validation. We first demonstrate the practical challenges of alignment (cf. Theorem 3.3), followed by showcasing the efficacy of our MaxMin-RLHF strategy. This approach illustrates that, with a focus on social welfare objectives, alignment across diverse human preferences is attainable.
5.1 Small Scale Experiments (with GPT-2): Sentiment and Conciseness Alignment
Dataset. For the experiment in this section on controlled sentiment generation, we categorized the humans into two groups: majority (Group 1) and minority (Group 2). In these sub-groups, Group 1 prefers responses with positive sentiment, and Group 2 prefers brevity (conciseness) in responses. We use the IMDb dataset as a basis for our inputs (Maas et al., 2011), the goal for the optimal policy is to produce responses y that exhibit positive sentiment (catering to Group 1) while remaining concise (catering to Group 2). We generated two sets of preference pairs for a controlled evaluation for each user group. For Group 1, we utilized a pre-trained sentiment classifier to ensure and similarly for Group 2 we preferred shorter responses over longer ones. To illustrate the majority and minority group dynamics, we control the proportion of the user groups in the preference data (Group 1: 80% and Group 2 - 20%). For the experiments in this subsection, we use GPT-2 (Radford et al., 2019) as the base model.
Impossibility Results. To demonstrate our impossibility results as stated in Theorem 3.3, we perform the three steps of RLHF (described in (Christian, 2020; Ouyang et al., 2022b)) as prevalent currently with a single utility reward function on the combined preference dataset. For SFT, we fine-tune GPT-2 until convergence on reviews from the train split of the IMDB dataset and use this GPT-2 backbone for both the reward model and PPO training. The generations are evaluated against the ground truth rewards for positive sentiment (majority group) and for conciseness (minority group). It is evident from Figure 3 that the generated responses are significantly biased toward the majority user group’s preference who preference positive sentiment (note high sentiment score (green curve, high score is better) after alignment) while the preferences (concise responses) of the minority user group were neglected (note high conciseness score (red curve, lower score is better) after alignment), resulting in more verbose generations than desired.
Proposed MaxMin RLHF. Our proposed algorithm can efficiently align to both group preferences as shown in Figure 5 thereby generating responses that are of positive sentiment and concise and thus cater to both the majority and minority user groups mitigating the social disparity. We further collectively present the average performance of MaxMin RLHF with the single reward RLHF and baseline model in Figure 6.
5.2 Large Scale Experments (with Tulu2-7B)
Datasets and Experimental Setup. We use the same dataset as Jang et al. (2023) and k data points from GPT4-Alpaca (Peng et al., 2023) are used as the instruction dataset to generate rollouts, collect pairwise feedback data, and PPO training. We utilize GPT-4 to simulate human annotators with preference prompts described in Table 4 in Appendix F. We divide the datasets into groups of human users. Each group has users, which are split into users in training data and users in testing data. For the experiments in this subsection, we use Tulu2-7B (Ivison et al., 2023) as the base model. For each dataset, P1, P2, and P3, we mix the training user groups to build the simulation dataset. We have 60 users in training data which are mixed from two different groups with diverse preferences. The original distribution is that users are evenly distributed in two clusters. Then, we use the EM algorithm to train reward models until we converge. Update by minimizing the negative log-likelihood loss (2). Then, trained model is used to assign clusters to users in testing data.
5.2.1 Main Results
Impossibility of Single Reward Model. When the user groups are biased in-terms of the representation of the user-groups (divided into majority and minority groups based on the preference dataset), the single reward model fails to capture the preferences of minority user groups. We test on preference dataset P1A/P1B representing two user groups and adjust the ratio of the number of users from group P1A and group P1B. Table 1 summarizes the accuracy for the majority group and minority group, as well as the accuracy of the total data. Here, low evaluation accuracy is an indication that the eventual alignment with the minority user group will be poor after the PPO step since the reward model itself is not accurate.
Ratio | Total | Majority | Minority |
---|---|---|---|
1:1 | 0.686 | 0.668 | 0.704 |
2:1 | 0.608 | 0.728 | 0.488 |
6:1 | 0.588 | 0.724 | 0.452 |
10:1 | 0.568 | 0.716 | 0.42 |
Reward Learning with EM (Algorithm 2). Following the procedures from Algorithm 2) in the experiment setup, we learn a mixture of reward models with EM algorithm on all three datasets until convergence, as shown in Figure 7. From the results in Figure 7, we note that after the fourth iteration, all users are clustered correctly, meaning the mixture preference model successfully converges we successfully learn diverse groups of users with diverse preferences.
MaxMin RLHF Alignment. We evaluate the performance of our MaxMin-RLHF alignment against conventional single-reward RLHF trained on preference datasets with diversity and representational bias. Our baselines include ratios of 1, 2, 6, and 10, the same setting as discussed in Table 1 for illustration of the representational bias in preference data. Following Jang et al. (2023), we use the same instances from Koala evaluation(Geng et al., 2023) and test the model’s ability to generate answers in different groups of users’ preferences. We run pairwise evaluations by GPT-4 using AlpacaFarm codebase(Dubois et al., 2023) and use the win rate to the base Tulu2-7B model as the metric. Our results in Table 2 and Table 3 show that MaxMin alignment keeps a high win-rate in comparison with conventional RLHF methods trained by PPO with a single reward model. Our MaxMin RLHF approach achieves an average improvement of more than 16% in win-rates over conventional RLHF algorithms and improves the win-rate (accuracy) for minority groups by over 33% without compromising the performance of majority groups. Table 2 illustrates that as the minority group’s representation in the preference data diminishes, the performance of conventional RLHF algorithms significantly deteriorates for them, whereas, our proposed Max-Min approach maintains a balance, respecting the performance of both groups by aligning with both.
Method | P3A | P3B | Average |
---|---|---|---|
MaxMin | 57.78 | 55.56 | 56.67 |
1:1 | 55.85 | 52.62 | 54.24 |
2:1 | 55.56 | 48.89 | 52.23 |
6:1 | 58.06 | 46.67 | 52.37 |
10:1 | 56.00 | 45.00 | 50.50 |
Method | P1A | P1B | Average |
---|---|---|---|
MaxMin | 57.50 | 60.00 | 58.75 |
1:1 | 56.00 | 51.97 | 53.99 |
2:1 | 57.78 | 44.00 | 50.89 |
6:1 | 54.81 | 48.00 | 51.41 |
10:1 | 55.11 | 45.08 | 50.10 |
Method | P2A | P2B | Average |
---|---|---|---|
MaxMin | 54.50 | 56.00 | 55.25 |
1:1 | 53.73 | 54.00 | 53.87 |
2:1 | 55.55 | 51.72 | 53.64 |
6:1 | 52.14 | 49.40 | 50.77 |
10:1 | 53.96 | 45.98 | 49.97 |
6 Conclusions
In this work, we critically examine the limitations of the conventional single-reward Reinforcement Learning from Human Feedback (RLHF) framework, particularly its insufficiency in addressing the diversity of human preferences, leading to an impossibility result for alignment with diverse preferences. To achieve a socially fair alignment in diverse human preference settings, we introduce a novel MaxMin-RLHF approach, which learns a max-min policy over mixture of reward functions to achieve a more equitable model alignment. Our experiments demonstrate the effectiveness of MaxMin-RLHF in producing socially fairer outcomes, highlighting the need for more inclusive strategies in RLHF methodologies.
References
- Aroyo & Welty (2015) Aroyo, L. and Welty, C. Truth is a lie: Crowd truth and the seven myths of human annotation. AI Magazine, 36(1):15–24, Mar. 2015. doi: 10.1609/aimag.v36i1.2564. URL https://ojs.aaai.org/aimagazine/index.php/aimagazine/article/view/2564.
- Aroyo et al. (2023a) Aroyo, L., Diaz, M., Homan, C., Prabhakaran, V., Taylor, A., and Wang, D. The reasonable effectiveness of diverse evaluation data, 2023a.
- Aroyo et al. (2023b) Aroyo, L., Taylor, A. S., Diaz, M., Homan, C. M., Parrish, A., Serapio-Garcia, G., Prabhakaran, V., and Wang, D. Dices dataset: Diversity in conversational ai evaluation for safety, 2023b.
- Bai et al. (2022a) Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022a.
- Bai et al. (2022b) Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., Chen, C., Olsson, C., Olah, C., Hernandez, D., Drain, D., Ganguli, D., Li, D., Tran-Johnson, E., Perez, E., Kerr, J., Mueller, J., Ladish, J., Landau, J., Ndousse, K., Lukosuite, K., Lovitt, L., Sellitto, M., Elhage, N., Schiefer, N., Mercado, N., DasSarma, N., Lasenby, R., Larson, R., Ringer, S., Johnston, S., Kravec, S., Showk, S. E., Fort, S., Lanham, T., Telleen-Lawton, T., Conerly, T., Henighan, T., Hume, T., Bowman, S. R., Hatfield-Dodds, Z., Mann, B., Amodei, D., Joseph, N., McCandlish, S., Brown, T., and Kaplan, J. Constitutional ai: Harmlessness from ai feedback, 2022b.
- Bakker et al. (2022) Bakker, M. A., Chadwick, M. J., Sheahan, H. R., Tessler, M. H., Campbell-Gillingham, L., Balaguer, J., McAleese, N., Glaese, A., Aslanides, J., Botvinick, M. M., and Summerfield, C. Fine-tuning language models to find agreement among humans with diverse preferences, 2022.
- Ben-Tal et al. (2013) Ben-Tal, A., den Hertog, D., Waegenaere, A. D., Melenberg, B., and Rennen, G. Robust solutions of optimization problems affected by uncertain probabilities. Management Science, 59(2):341–357, 2013. ISSN 00251909, 15265501. URL http://www.jstor.org/stable/23359484.
- Bertsekas (2019) Bertsekas, D. P. Reinforcement Learning and Optimal Control. Athena Scientific, Belmont, MA, 2019. ISBN 978-1-886529-39-7.
- Bradley & Terry (1952) Bradley, R. A. and Terry, M. E. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952.
- Casper et al. (2023) Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217, 2023.
- Chakraborty et al. (2024) Chakraborty, S., Bedi, A., Koppel, A., Wang, H., Manocha, D., Wang, M., and Huang, F. Parl: A unified framework for policy alignment in reinforcement learning. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
- Chen et al. (2024) Chen, Z., Deng, Y., Yuan, H., Ji, K., and Gu, Q. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335, 2024.
- Cho et al. (2018) Cho, W. S., Zhang, P., Zhang, Y., Li, X., Galley, M., Brockett, C., Wang, M., and Gao, J. Towards coherent and cohesive long-form text generation. arXiv preprint arXiv:1811.00511, 2018. URL https://ar5iv.org/abs/1811.00511.
- Christian (2020) Christian, B. The alignment problem: Machine learning and human values. WW Norton & Company, 2020.
- Denton et al. (2021a) Denton, E., Díaz, M., Kivlichan, I., Prabhakaran, V., and Rosen, R. Whose ground truth? accounting for individual and collective identities underlying dataset annotation, 2021a.
- Denton et al. (2021b) Denton, E., Díaz, M., Kivlichan, I., Prabhakaran, V., and Rosen, R. Whose ground truth? accounting for individual and collective identities underlying dataset annotation, 2021b.
- Dubois et al. (2023) Dubois, Y., Li, X., Taori, R., Zhang, T., Gulrajani, I., Ba, J., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpacafarm: A simulation framework for methods that learn from human feedback. arXiv preprint arXiv:2305.14387, 2023.
- Duchi et al. (2018) Duchi, J., Glynn, P., and Namkoong, H. Statistics of robust optimization: A generalized empirical likelihood approach, 2018.
- Geng et al. (2023) Geng, X., Gudibande, A., Liu, H., Wallace, E., Abbeel, P., Levine, S., and Song, D. Koala: A dialogue model for academic research. Blog post, April, 1, 2023.
- Goodfellow et al. (2016) Goodfellow, I., Bengio, Y., and Courville, A. Deep Learning. The MIT Press, 2016. ISBN 0262035618.
- Ivison et al. (2023) Ivison, H., Wang, Y., Pyatkin, V., Lambert, N., Peters, M., Dasigi, P., Jang, J., Wadden, D., Smith, N. A., Beltagy, I., et al. Camels in a changing climate: Enhancing lm adaptation with tulu 2. arXiv preprint arXiv:2311.10702, 2023.
- Jang et al. (2023) Jang, J., Kim, S., Lin, B. Y., Wang, Y., Hessel, J., Zettlemoyer, L., Hajishirzi, H., Choi, Y., and Ammanabrolu, P. Personalized soups: Personalized large language model alignment via post-hoc parameter merging. arXiv preprint arXiv:2310.11564, 2023.
- Ji et al. (2023a) Ji, J., Liu, M., Dai, J., Pan, X., Zhang, C., Bian, C., Zhang, C., Sun, R., Wang, Y., and Yang, Y. Beavertails: Towards improved safety alignment of llm via a human-preference dataset, 2023a.
- Ji et al. (2023b) Ji, X., Wang, H., Chen, M., Zhao, T., and Wang, M. Provable benefits of policy learning from human preferences in contextual bandit problems. arXiv preprint arXiv:2307.12975, 2023b. URL https://ar5iv.org/abs/2307.12975.
- Kaufmann et al. (2023) Kaufmann, T., Weng, P., Bengs, V., and Hüllermeier, E. A survey of reinforcement learning from human feedback, 2023.
- Kovač et al. (2023) Kovač, G., Sawayama, M., Portelas, R., Colas, C., Dominey, P. F., and Oudeyer, P.-Y. Large language models as superpositions of cultural perspectives, 2023.
- Li et al. (2023) Li, Z., Yang, Z., and Wang, M. Reinforcement learning with human feedback: Learning dynamic choices via pessimism. arXiv preprint arXiv:2305.18438, 2023. URL https://ar5iv.org/abs/2305.18438.
- Lindsay (1995) Lindsay, B. G. Mixture models: Theory, geometry and applications. NSF-CBMS Regional Conference Series in Probability and Statistics, 5:i–163, 1995. ISSN 19355920, 23290978. URL http://www.jstor.org/stable/4153184.
- Maas et al. (2011) Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., and Potts, C. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142–150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http://www.aclweb.org/anthology/P11-1015.
- Ouyang et al. (2022a) Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback, 2022a.
- Ouyang et al. (2022b) Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback, 2022b.
- Ovadya (2023) Ovadya, A. ’generative ci’ through collective response systems, 2023.
- Peng et al. (2023) Peng, B., Li, C., He, P., Galley, M., and Gao, J. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023.
- Radford et al. (2019) Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. 2019. URL https://api.semanticscholar.org/CorpusID:160025533.
- Rafailov et al. (2023) Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C. Direct preference optimization: Your language model is secretly a reward model, 2023.
- Ramé et al. (2023) Ramé, A., Couairon, G., Shukor, M., Dancette, C., Gaya, J.-B., Soulier, L., and Cord, M. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards, 2023.
- Sagawa et al. (2020) Sagawa, S., Koh, P. W., Hashimoto, T. B., and Liang, P. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization, 2020.
- Sandri et al. (2023) Sandri, M., Leonardelli, E., Tonelli, S., and Jezek, E. Why don’t you do it right? analysing annotators’ disagreement in subjective tasks. In Vlachos, A. and Augenstein, I. (eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 2428–2441, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.eacl-main.178. URL https://aclanthology.org/2023.eacl-main.178.
- Santurkar et al. (2023) Santurkar, S., Durmus, E., Ladhak, F., Lee, C., Liang, P., and Hashimoto, T. Whose opinions do language models reflect? arXiv preprint arXiv:2303.17548, 2023.
- Sap et al. (2022) Sap, M., Swayamdipta, S., Vianna, L., Zhou, X., Choi, Y., and Smith, N. A. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. In Carpuat, M., de Marneffe, M.-C., and Meza Ruiz, I. V. (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5884–5906, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.naacl-main.431. URL https://aclanthology.org/2022.naacl-main.431.
- Schulman et al. (2017) Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
- Seidel (2011) Seidel, W. Mixture Models, pp. 827–829. Springer Berlin Heidelberg, Berlin, Heidelberg, 2011. ISBN 978-3-642-04898-2. doi: 10.1007/978-3-642-04898-2˙368. URL https://doi.org/10.1007/978-3-642-04898-2_368.
- Sen (2017) Sen, A. Collective Choice and Social Welfare. Harvard University Press, Cambridge, MA and London, England, 2017. ISBN 9780674974616. doi: doi:10.4159/9780674974616. URL https://doi.org/10.4159/9780674974616.
- Stiennon et al. (2022a) Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. Learning to summarize from human feedback, 2022a.
- Stiennon et al. (2022b) Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. Learning to summarize from human feedback, 2022b.
- Stiennon et al. (2022c) Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. Learning to summarize from human feedback, 2022c.
- Sutton & Barto (2018) Sutton, R. S. and Barto, A. G. Reinforcement Learning: An Introduction. The MIT Press, Cambridge, MA, 2 edition, 2018. ISBN 978-0262039246.
- Vogels (2021) Vogels, E. A. The state of online harassment. Pew Research Center, 13:625, 2021.
- Wang et al. (2023) Wang, Y., Zhong, W., Li, L., Mi, F., Zeng, X., Huang, W., Shang, L., Jiang, X., and Liu, Q. Aligning large language models with human: A survey. arXiv preprint arXiv:2307.12966, 2023.
- Zhang et al. (2020) Zhang, J., Koppel, A., Bedi, A. S., Szepesvari, C., and Wang, M. Variational policy gradient method for reinforcement learning with general utilities, 2020.
- Zhang et al. (2021) Zhang, J., Ni, C., Yu, Z., Szepesvari, C., and Wang, M. On the convergence and sample efficiency of variance-reduced policy gradient method, 2021.
- Zhang et al. (2023) Zhang, Z., Su, Y., Yuan, H., Wu, Y., Balasubramanian, R., Wu, Q., Wang, H., and Wang, M. Unified off-policy learning to rank: a reinforcement learning perspective. arXiv preprint arXiv:2306.07528, 2023. URL https://ar5iv.org/abs/2306.07528.
- Zhu et al. (2023) Zhu, B., Jiao, J., and Jordan, M. I. Principled reinforcement learning with human feedback from pairwise or -wise comparisons, 2023.
- Ziegler et al. (2019) Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019.
- Ziegler et al. (2020) Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences, 2020.
Inhalt
- 1 Introduction
- 2 Preliminaries
- 3 An Impossibility Result for Single Reward RLHF with Diverse Preferences
- 4 Proposed Approach: Alignment with Diverse Preferences
- 5 Experimental Results
- 6 Conclusions
- A Notations
- B A Detailed Context of Related Works
- C Preliminary Results
- D Proof of Lemma
- E Proof of Theorem
- F Additional Details of the Experiments
Appendix A Notations
We define the various notations in this table first.
Notations | Description |
---|---|
x | prompt |
set of prompts | |
y | output text generated by the LLM |
direct supervised fine-tuning model, takes x as input and generates y as output | |
output pair generated by LLM | |
human | |
dataset which has the data of the form | |
reward model parameter | |
language model parameter | |
set of human population |
Appendix B A Detailed Context of Related Works
Reinforcement Learning from Human Feedback. RL methods, such as policy gradient, applied to train language models for long-form generation Cho et al. (2018). The current RLHF approaches (Stiennon et al., 2022b; Ziegler et al., 2020; Zhu et al., 2023) involve training a reward model based on human preference feedback and then fine-tuning the language model using proximal policy optimization (PPO) (Schulman et al., 2017). The PPO algorithm helps to learn a model that produces responses that maximize the reward (Ouyang et al., 2022b; Bai et al., 2022a). Besides PPO, DPO (Direct Preference Optimization, Rafailov et al. (2023)) directly trains the large language model using human preferences without training the reward model. A self-play-based approach such as SPIN (Chen et al., 2024) is similar to DPO but has an iterative framework. However, most of the existing alignment approaches only consider the average preference by human annotators and ignore the inherent diversity among human preferences (Casper et al., 2023; Kaufmann et al., 2023). A number of theoretical studies have analyzed the efficiency and benefits for reinforcement learning using preference data Ji et al. (2023b); Zhang et al. (2023); Li et al. (2023). Chakraborty et al. (2024) proposed a bilevel reinforcement learning framework for policy alignment. Recently (Santurkar et al., 2023) created a dataset for evaluating the alignment of language models with US demographic groups over a wide range of topics and found substantial misalignment between a selanguage models and those groups. It emphasizes the criticality of considering diversity while performing alignment.
Diversity in Human Preferences. Here, we briefly review the literature highlighting the reasons for diversity in the context of LLMs. Diverse human preferences stem significantly from various factors related to social and cultural backgrounds (Aroyo et al., 2023b, a; Denton et al., 2021a). The key factors contributing to this diversity include (i) socio-demographic backgrounds, including race, ethnicity, age, and gender shape preferences. Gender differences, for example, influence sensitivity to online content, with women facing more online harassment Vogels (2021). (ii) Personal bias and context subjectivity, which affects the human preferences for controversial topics in interpreting language and divisive themes (Denton et al., 2021b; Sandri et al., 2023)). (iii) Imperfect preferences, which arises due to variations in expertise, training, or quality control leading to diverse preferences, with certain content inaccurately considered offensive by some groups (Sandri et al., 2023). (iii) Linguistic ambiguity & missing context, could lead to diversity because of words or phrases with multiple possible interpretations and without clear context (Sandri et al., 2023; Denton et al., 2021b; Sap et al., 2022). These factors collectively underscore the complexity of aligning LLM outputs with the diverse preferences of human users, demonstrating the importance of recognizing and addressing the multifaceted nature of user feedback.
Appendix C Preliminary Results
{lem}The parametrized preference probability distribution
under the Bradley -Terry model (Bradley & Terry, 1952) is Lipschitz with respect to parameter . This implies that
(22) |
with where denotes the upper bound on the feature representation for all .
Proof.
Let us start from the definition of given by
(23) |
From the definition of the Bradley-Terry preference model from equation (1) with the linear parametrization of the reward function as , we can write the equality in (23) as
(24) |
where we define for the ease of notation. Next, differentiating both sides in (C) with respect to , we obtain
(25) |
Taking the norm on both sides and applying Cauchy-Schwartz inequality, we get
(26) |
From the definition of and the boundedness of the feature representations, we note that . Hence, we obtain the final bound
(27) |
Hence proved.
∎
The cross-entropy loss minimization for reward learning in step 2 in the RLHF pipeline (cf. (2)) leads to implicit weightage minimization among the user groups. Specifically, the loss function minimizes the distance to distribution , where is the implicit distribution among user groups.
Proof of Lemma C.
From the equality in (3.1), we note that we can write . With this notation, the loss function for reward learning in (3.2) can be written as
(28) |
where the equation incorporates the individual user group’s optimal (we denote the corresponding individual optimal reward parameter by ) in the likelihood objective. As a first step, let us decompose (28) as
(29) |
where, we add and subtract and to get the final expression. After rearranging the terms in (C), we get
(30) | ||||
Next, by utilizing the definition of KL-divergence and entropy to get the final expression as follows
(31) |
From the above objective in (31), we note that the objective is minimized for when To proceed further, let us focus on the term from equation (31) as
(32) |
From the definition of KL d in (3.1), it holds that
(33) |
Next, by adding and subtracting the term in the right hand side of (33), we get
(34) | ||||
Now, replacing this expression in the original implicit minimization objective in (31), we note that the minimization will be achieved when for all . Hence, the reward learning objective is implicitly learning a weighted combination, which would lead to a significant gap in individual utilities, as discussed in the subsequent section. ∎
Appendix D Proof of Lemma 3.2
Proof.
Let us reconsider the reward learning loss whose empirical version is minimized to obtain parameter which is the approximation to the true parameter
. As discussed in Sec. 3.2, due to human user groups, a user group specific will also exist. Our goal is to characterize the gap between and defined as
(35) |
where the optimal for the user group is given by
(36) |
Let us consider the idealistic setting of infinite data under which we know that MLE would converge to optimal (Zhu et al., 2023). Hence, to proceed further, let us add subtract in the right-hand side of (35), we get
(37) |
To derive the lower bound on the reward suboptimality , we begin with the definition of the total variation distance as
(38) |
From the Lipschitzness of the preference probability as derived in Lemme C, we can write
(39) |
From the lower bound in (38) and the expression in (37), we obtain
(40) |
Next, we expand on the term . From the statement of Lemma C, we note that , hence we can write
(41) |
where we use to denote a user group for ease of notation. Since , we can write
(42) |
After interchanging the order of summation, we get
(43) |
Using the equality in (D) into the right hand side of (40), we obtain
(44) |
From the Definition 2, we will get the final result. Hence proved.
∎
Appendix E Proof of Theorem 3.3
Proof.
We can define the alignment gap of RLHF model to a specific user group as
(45) |
We note that in this specific RLHF setting under the KL-based regularization, the objective satisfies strong convexity w.r.t with strong convexity parameter , hence it holds that
(46) |
Now utilizing that is Lipschitz continuous with parameter , under the condition that there exists some such that for all , we get
(47) |
From the results in (Rafailov et al., 2023), we can derive an analytical mapping from reward functions to optimal policies for the KL-constrained reward maximization objective as denied in (10) as :
(48) |
where is the optimal policy under the reward and is the partition function given as . Note that such an equivalence is specific to the RLHF problem under the Bradley Terry preference model as shown in (Rafailov et al., 2023). Next, replacing equation (48) in the equation (47), we get
(49) | ||||
As stated in (C), under the linearly parametrized reward function, we have , where the parameter and similarly . Let and we denote the feature matrix as , replacing in (49), we get the final expression. Next, expanding the norm on the right hand side, we obtain
(50) |
Next we lower-bound the matrix norm of with the minimum eigen value as
(51) |
where we obtain the lower bound in terms of the reward suboptimality. From the statement of Lemma 3.2, we can lower bound the right hand side in (51) as follows
(52) |
Hence proved.
∎
Appendix F Additional Details of the Experiments
In this section, we provide additional details of the experiments in Section 5.
User Group | Preference Prompt |
---|---|
P1A | Generate/Choose a response that can be easily understood by an elementary school student. |
P1B | Generate/Choose a response that only a PhD Student in that specific field could understand. |
P2A | Generate/Choose a response that is concise and to the point, without being verbose. |
P2B | Generate/Choose a response that is very informative, without missing any background information. |
P3A | Generate/Choose a response that is friendly, witty, funny, and humorous, like a close friend. |
P3B | Generate/Choose a response (that answers) in an unfriendly manner. |