Online Policy Learning from Offline Preferences

Reinforcement learning (RL) uses the Markov decision process $\langle\mathcal{S},\mathcal{A},P,r,\gamma,\mu\rangle$ (Sutton and Barto, 2018) to model sequential decision-making tasks. Here, $\mathcal{S}$ is the set of states, and $\mathcal{A}$ is the set of actions; they represent information and options available to an agent for decision-making, respectively. The transition probability $P:\mathcal{S}\times\mathcal{A}\to\mathit{\Delta}(\mathcal{S)}$ governs how states transit, where $\mathit{\Delta}(\cdot)$ is the set of distributions over a set. The reward function $r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ is a function that evaluates agents’ decisions, which is unknown and to be inferred in PbRL. The discount factor $\gamma\in(0,1)$ is later used to define the value function, and $\mu$ is the distribution for initial states. An MDP prescribes a protocol for sequential interaction between an agent and a hypothetical entity called the environment. Starting from an initial state sampled from $\mu$, the agent observes a state $s\in\mathcal{S}$ from the environment and selects an action $a$ according to a stochastic policy $\pi:\mathcal{S}\to\mathit{\Delta}(\mathcal{A})$ associated with the agent. It then receives the next state from the environment, which is sampled from $P(\cdot|s,a)$. A sequence of states and actions generated in interaction $(s_{1},a_{1},s_{2},a_{2},\dots)\overset{\mathrm{def}}{=}\tau$ is defined as a trajectory. To simplify notations, this work occasionally uses $x$ as a shorthand notation for a state-action pair $(s,a)$.

The return of a trajectory is the sum of rewards assigned to its state-action pairs. For state $s\in\mathcal{S}$, the value function $v^{\pi}(s)$ of a policy $\pi$ specifies the expectation of $\gamma$-discounted return starting from an initial state $s$ and following a policy $\pi$. It is defined as follows:

where the expectation is taken over states (following the transition probability) and actions (following the policy). The value of a policy $\pi$ is the expectation of $v^{\pi}(s)$ over initial-state distribution $\mu$: $v^{\pi}=\mathbb{E}_{s\sim\mu}[v^{\pi}(s)]$. The goal of RL is to find an optimal policy $\pi^{*}$ such that $v^{\pi^{*}}\geq v^{\pi}$ for any policy $\pi$ under a given reward function $r$.

A useful quantity is the discounted occupancy measure, which is defined as follows.

The discounted occupancy measure $\rho^{\pi}$ can be interpreted as an unnormalized measure for state-action pairs generated by $\pi$. It is straightforward to show that the normalizer of occupancy measures is $\frac{1}{1-\gamma}$.

For function $f:\mathcal{S}\times\mathcal{A}\to\mathbb{R}$, we write $\mathbb{E}_{x\sim\rho^{\pi}}[f(x)]$ as the sum of $f(x)$ over all state-action pairs weighted by $\rho^{\pi}(x)$ with slight abuse of notation. The values can be expressed using $\rho^{\pi}$ alternatively.

As a final remark, the following theorem relates policies with occupancy measures, which allows one to regard policies as samplers for state-action pairs.

Preferences are formally defined using Thurstone’s model (Thurstone, 1927). Suppose there is a general object space $\mathcal{O}$. For each object $o\in\mathcal{O}$, this model assumes a “true” utility score $G(o)\in\mathbb{R}$. Annotators compare objects using disturbed utility $G(o)+\epsilon$, where $\epsilon$ is sampled from some distribution for noise. Then, the probability of object $o$ being preferred over object $o^{\prime}$ follows $\Pr(o\succ o^{\prime})=\Pr(G(o)-G(o^{\prime})>\epsilon^{\prime}-\epsilon)$, where $\epsilon^{\prime}$ is the noise associated with $o^{\prime}$. Subsequently, we work with a special case of Thurstone’s model called Bradley–Terry model (BT model) (Bradley and Terry, 1952). Assuming $\epsilon\sim\mathrm{Gumbel}(0,1)$, we have $\Pr(o\succ o^{\prime})=\sigma(z-z^{\prime})$, where $\sigma(\cdot)$ is the sigmoid function. Under this model, the log-likelihood of a preference $o\succ o^{\prime}$ is denoted by $\ell(o,o^{\prime})=\log\sigma(G(o)-G(o^{\prime}))$.

We assume that the offline preferences are specified between trajectories, as they provide annotators with more information than states or state-action pairs. Under the BT model, a trajectory $\tau$ is preferred over another trajectory $\tau^{\prime}$, denoted as $\tau\succ\tau^{\prime}$, if the utility of $\tau$ is higher than that of $\tau^{\prime}$. This means that $\tau$ corresponds to a more desirable outcome from an annotator’s perspective. We will use preferences between state-action pairs to formulate the proposed virtual preferences. In this case, a state action pair $x$ is preferred over another state-action pair $x^{\prime}$ if the utility of $r(x)>r(x^{\prime})$.

Reward functions can be learned by maximizing the likelihood of preferences under the BT model. A canonical assumption for parameterizing trajectory utility is the sum of rewards in a trajectory: $\sum_{x\in\tau}r(x)$ (Christiano et al., 2017; Ibarz et al., 2018). We use the reward of a state-action pair as its utility. A sample for trajectory preferences is written as $(\tau_{1},\tau_{2},c)$, where $\tau_{1}$ and $\tau_{2}$ are the trajectories being compared. $c=1$ if $\tau_{1}\succ\tau_{2}$, and $c_{j}=0$ otherwise. Suppose the reward function is parameterized by $\theta_{r}$, and let the sum of estimated rewards for state-action pairs in a trajectory $\tau$ to be $G(\tau;\theta_{r})$. Given a set of $M$ preferences $\mathcal{Y}$, we can learn $\theta_{r}$ by minimizing:

Note that Equation 4 coincides with learning from state-action preferences $x\succ x^{\prime}$ such that $x\in\mathcal{D}$ and $x^{\prime}\in\mathcal{D}_{\text{RL}}$. By minimizing Equation 4, we are maximizing the log-likelihood that state-action pairs in offline data $\mathcal{D}$ are preferred over state-action pairs generated by the agent. Since such preferences are not collected from annotators, we consider them to be generated by a virtual annotator and call them virtual preferences. We use $\succ_{v}$ for virtual comparisons and $\mathcal{Y}_{v}$ for virtual preferences. The use of virtual state-action preferences is a key idea of PbAIL. As will be discussed in Section 4.3, it leads to a straightforward approach for handling non-optimal offline data.

As a final remark, it is straightforward to combine recent ideas of PbRL, such as the use of data augmentation (Park et al., 2022) or active query generation (Biyik et al., 2020), with PbAIL. We leave these extensions as future work and focus solely on showing the effectiveness of PbAIL.

The root cause for virtual preferences to be unreliable here is that, our assumption $b\succ\pi$ for any $\pi$ is no longer justified. Given preferences $\mathcal{Y}$, we expect the agent to outperform the behavior policy of $\mathcal{D}$. Our approach reduces the problem of handling imperfect behavior policy to inferring the reliability of state-action preferences in $\mathcal{Y}_{v}$.

In summary, when offline trajectories are not perfect, PbAIL uses a probabilistic model for virtual preferences that can simultaneously infer the reliability of virtual preferences and learn a reward function. Algorithm 1 summarizes the algorithm for this case.

We considered seven Mujoco tasks: Ant, Walker2d, Hopper, HalfCheetah (HC for short), HumanoidStandup (HS for short), Pusher, and Swimmer. To reveal algorithms’ performance in practice, we evaluated them using imperfect offline trajectories generated by multiple policies. For each task, we used two versions of offline data: novice and mixture. We first trained five SAC agents for five million steps using different random seeds and considered their final performance as the final performance for SAC. The novice version was generated using five model checkpoints reaching 20% of the final performance, while the mixture version was generated by them and five checkpoints reaching 50% of the final performance. For preferences, we considered two sizes of $\mathcal{Y}$: $|\mathcal{Y}|=1225$ and $|\mathcal{Y}|=300$. Following prior practice (Christiano et al., 2017), preferences were generated for trajectory clips of length 60 using the truth rewards.

We considered the method proposed by Christiano et al. (2017) (referred to as PbRL), which is an algorithm for online PbRL. We also included two other baselines, SACfD and CAIL (Zhang et al., 2021). SACfD is an extension to the method proposed by Ibarz et al. (2018) that combines BC with PbRL. CAIL is an imitation learning algorithm that encourages the ranking of offline trajectories induced by the inferred returns aligns with the given preferences. In addition, we present the results for an ablation of the reliability estimation by using Equation 4 instead of Equation 6, denoted as PbAIL^-. For reference, this paper also reports the results for SAC trained with the truth rewards (denoted as GT) and the return of offline trajectories (denoted as Data).

The algorithms were compared for test returns after being trained for one million steps. Returns were normalized to a 0–1 scale, such that 0 corresponds to a random policy and 1 corresponds to the final performance; a negative number indicates a method is worse than random policy. We report the mean and standard deviation of results for five random seeds.

We used SAC as the policy learner. The reward functions of PbRL, SACfD, and PbAIL, as well as the discriminator of CAIL, were parameterized with a FFN of 64 units with the spectral normalization (Miyato et al., 2018). We also applied the weight decay to the reward functions of PbRL, SACfD, and PbAIL, and the reliability network of PbAIL. All experiments were performed on six NVIDIA A6000 GPUs. Our code is available here¹¹1https://www.dropbox.com/s/g0m3qwng4cnltmt/PbAIL.zip?dl=0.

Table 1 shows the results on the novice datasets with $|\mathcal{Y}|=1225$. First, we compare the performance of PbRL, SACfD, and PbAIL. Except in the case of Walker2d and Swimmer, PbRL does not demonstrate competitive performance. SACfD outperforms PbRL in Ant, Walker2d, and HC, but it does not perform well for the rest. PbAIL surpasses PbRL in six of the seven tasks, achieving the best performance in five tasks and even outperforming GT in three tasks. In addition, PbAIL presents significant advantages over CAIL. These results corroborate PbAIL’s efficacy for learning from offline trajectories and offline preferences.

Table 1 also shows the individual effect of using virtual preferences and inferring their reliability. The advantage of PbAIL^- over PbRL, particularly in Ant, HS, and Pusher, confirms the efficacy of using virtual preferences. The efficacy of inferring preference reliability is supported by comparing data return (denoted as Data) and the performance of PbAIL^- accompanied with the comparison between PbAIL^- and PbAIL. Except in Walker2d, PbAIL^- only matches the return of data, which means that information in offline preferences is not properly utilized. PbAIL’s substantial improvement over data return confirms the necessity of modeling virtual preference reliability when dealing with imperfect offline data.

Table 2 shows the results for $|\mathcal{Y}|=300$ on the novice datasets. Compared to the results presented in Table 1, preference-based methods (PbRL, SACfD, and PbAIL) worsen significantly, which is expected as less information is available in preferences. Interestingly, with fewer preferences, CAIL tends to perform better—it improves in five tasks. This is probably due to the fact that it uses preferences to update the weights of samples in $\mathcal{D}$ for imitation learning, using a max-margin loss function for preferences. With fewer preferences, the weights of samples can be determined more easily, so its adversarial learning process is more stable.

Table 8 shows the results for the mixture datasets with $|\mathcal{Y}|=1225$. The quality of offline data increases by 0.1 to 0.2 when compared with the case in Table 1. PbRL and PbAIL cannot benefit from the improved quality, but PbAIL is the only method that attains the best performance in three tasks. SACfD improves in Ant and Walker2d but fails in HS. CAIL can best benefit from improved quality of offline trajectories. The results in Table 1 and Table 8 imply that, PbAIL is suitable for scenarios where preferences are accessible, and imitation-based methods (SACfD and CAIL) can be better when good demonstrations can be collected.

Finally, let us discuss our claim on the generalizability problem. We analyzed the generalizability of reward functions using the mixture datasets with 1225 preferences. For each method, we took 10 model checkpoints in the early stage of policy learning. We then rolled out trajectories using the actor of each checkpoint and inferred their returns using the corresponding reward function. The Kendall’s rank correlation coefficient reflects how well the ranks of trajectories computed using inferred returns match those computed using ground-truth returns. The higher it is, the better the two ranks match, which means that reward functions better generalize to the learning agent’s behaviors.

Figure 2 shows the results for Ant, HC, and Pusher. The reward function of PbRL fails to generalize for Ant and Pusher, which explains PbRL’s poor performance for the two tasks. Compared to PbRL, the reward function of SACfD generalizes better for Ant but fails for Pusher. Meanwhile, for all four tasks, PbAIL’s reward functions demonstrate good generalizability, which explains its performance.