SuperPADL: Scaling Language-Directed Physics-Based Control with Progressive Supervised Distillation

Recent work has assembled large datasets $D=\{(m_{i},C_{i})\}$ of motion capture sequences $m_{i}$ annotated with a set of one or more natural language labels $C_{i}$ (Punnakkal et al., 2021; Guo et al., 2022). However, these motion capture recordings are kinematic, preventing the direct application of supervised losses when training physically-simulated control policies. In order to address this incompatibility, in the first stage of SuperPADL we translate our motion capture dataset into the physical domain by training an expert tracking policy on every motion in our dataset.

Our approach is inspired by the first stage of MoCapAct (Wagener et al., 2023), leveraging DeepMimic as our tracking method (Peng et al., 2018). We train a DeepMimic expert policy $\pi^{e}_{i}({\mathbf{a}}_{t}|{\mathbf{o}}_{t},\phi)$ on every motion capture sequence in our dataset, conditioned on the current state of character ${\mathbf{o}}_{t}$ as well as a phase variable $\phi\in[0,1]$ that synchronizes the policy to the reference motion.

We train each policy for a maximum of 3000 epochs, corresponding to approximately 200M frames of experience. To reduce overall compute usage, we monitor the cartesian pose error throughout training and stop training early when the error falls below 3cm. If the epoch limit is reached and the pose error remains above 5cm, we discard the motion from our dataset. By leveraging a GPU-accelerated implementation of DeepMimic in NVIDIA Isaac Gym (Peng et al., 2018; Makoviychuk et al., 2021), the majority of tracking policies complete training in under an hour, while only 5% of motions are discarded (see Section 4.1). The discarded items in our dataset often correspond to physically implausible motions, such as those that involve third-party objects that are not recorded in the motion capture sequence (e.g. climbing up nonexistent stairs, sitting on nonexistent chairs, etc.). Additionally, our tracking environment resets whenever a character’s body part, excluding the feet, touches the ground. This causes motions like crawling and lying prone to be discarded.

After training each tracking policy, we record a dataset of trajectories containing 10000 observation-action frames from each expert:

In practice, in order to increase the diversity of states encountered during these rollouts, $T_{i}$ is created in chunks by initializing the character 100 different times using random frames from the reference motion and rolling out each initialization for 100 frames. Additionally, in 90% of rollouts, stochastic actions are sampled from the policy’s action distribution, while in the remaining 10% deterministic/greedy actions are taken from the mean of the action distribution. Note that while a combination of stochastic and deterministic actions are used to generate the rollouts, the actions recorded in the trajectory data always correspond to the deterministic action that the expert policy would have taken at each state. This is done to avoid adding noise to the trajectory action labels. We refer to this collection of trajectories $D_{T}=\{T_{i}\}$ as our trajectory dataset.

Our ultimate goal is to train a single model that can perform a wide range of behaviors in response to user text commands and seamlessly transition between different behaviors as the user’s command changes. However, at the end of the first stage of SuperPADL training, we are instead left with a collection of highly specialized expert tracking policies. Each expert can reproduce their corresponding motion capture clip with high quality, yet they cannot generate any other motion. Additionally, tracking policies lack the robustness to reliably recover from perturbations or initialization in out-of-distribution states, as might be necessary when transitioning between different motions. Therefore, the next step in our pipeline is to develop more general control policies that retain the motion quality of individual experts while also being much more robust.

Previous work has demonstrated that adversarial reinforcement learning can effectively train policies that exhibit these properties (Peng et al., 2021, 2022; Juravsky et al., 2022). However, as we experimentally demonstrate in Section 4.3, these RL techniques do not directly scale to thousands of motions. We address this limitation by employing a progressive distillation approach in the second and third stages of SuperPADL training. In the second stage of SuperPADL, we train controllers on small groups of motions using an objective combining adversarial RL and behaviour cloning, before performing a second, purely-supervised distillation stage to train a final global policy. In this first distillation stage, we randomly partition our dataset into groups of 20 motions:

and train a group controller $\pi^{g}_{i}({\mathbf{a}}_{t}|{\mathbf{o}}_{t},I)$ on each partition $P_{i}$, parameterized by the current character state ${\mathbf{o}}_{t}$ and the motion index $I\in\{20i+1,...,20i+20\}$. The index $I$ is encoded using a trainable, randomly-initialized embedding table.

Our goal is for each group controller to:

• •

  Imitate the motions in its partition when conditioned on the corresponding motion.

• •

  Naturally transition between these motions when the input index changes.

• •

  Generally avoid falling over.

We optimize each group controller using PADL+BC, a novel objective that combines the adversarial RL setup of PADL with behaviour cloning:

PADL introduces a motion-conditioned discriminator network:

which is trained to distinguish between state transitions $({\mathbf{s}},{\mathbf{s}}^{\prime})$ from a reference motion $m_{I}$ and state transitions generated by the policy when conditioned on $I$. The policy is optimized with PPO (Schulman et al., 2017), using a reward that encourages the policy to “fool” the discriminator:

In addition to this PPO loss, we introduce an additional behaviour cloning loss on the dataset of expert trajectories $D_{T}$. At each step of optimization, we sample (observation, action) pairs from the stored trajectories of our grouped motions and encourage the group controller to imitate the expert actions:

While the observations for the tracking experts included a phase variable to synchronize the policy to the target motion, this phase variable is omitted from the observations given to the group controller. This is crucial since determining the correct phase observation to give to the group controller during inference is difficult - for example, when transitioning from one motion to another, it is unclear what the phase should be set to.

The motivation behind the added behaviour cloning loss is twofold. First, the supervised training signal allows us to significantly cut down on the training cost of a group controller relative to a pure-PADL policy. We train each group controller using a 2000-epoch warmup phase where only the BC loss is applied. These warmup epochs complete significantly faster than normal PPO epochs where trajectories must be rolled out in the environment. Following this warmup, we only train group controllers with PPO+BC for an additional 1B samples of experience, compared to the 7B samples reported in Juravsky et al. (2022) for training a PADL policy from scratch. Overall, training a group controller with PADL+BC completes in around 12 hours on a single A40 GPU, compared to almost three days of training for a pure-PADL controller. Additionally, we demonstrate in Section 4.4 that PADL+BC controllers, leveraging the experts’ accurate motion reconstructions, can generate motions with higher quality than pure PADL policies.

Group controllers mark a significant improvement in generalization over the tracking expert policies, enabling a single controller to reproduce multiple motions, naturally transition between motions, and operate without a phase variable. However, the group controllers are still constrained by the relatively small set of motions that each controller is trained on. As we demonstrate in Section 4.3, PADL+BC alone does not scale to the thousands of motions available in open-source motion capture datasets (Mahmood et al., 2019). Moreover, our group controllers are conditioned using simple motion indices and cannot follow commands in natural language.

To train a global, language-conditioned policy $\pi^{G}({\mathbf{a}}_{t}|{\mathbf{o}}_{t},c)$ (where $c$ denotes a text caption), we perform a second round of distillation, now leveraging the group controllers as teacher policies. Unlike PADL+BC, which combines RL and supervised training objectives, the training of the global policy is purely supervised, allowing us to scale to much larger datasets.

Training of the global policy begins similarly to group controllers with an offline, behaviour cloning warmup phase using the static trajectory dataset $D_{T}$ collected from the tracking policies. This initializes the state distribution of the global policy to be similar enough to those of the group controllers that they can provide effective teacher feedback.

Following warmup, the global policy is trained to convergence using online imitation learning similar to DAGGER (Ross et al., 2011). Every epoch, trajectories are rolled using the current global controller. Each observation in those trajectories is then annotated using the appropriate group controller (based on the motion that the policy is trying to imitate). The global controller is optimized to match these annotations. Unlike with group controllers, when conditioning the global controller to imitate motion $i$, we provide a natural language caption for the motion instead of simply an index variable. At every rollout, we sample a motion $(m_{i},C_{i})$ from $D$ and sample a caption $c$ from $C_{i}$. The caption is encoded using the CLIP text encoder, with the pooled encoder embedding provided to the main policy network $\pi^{G}$.

In Figure 4 we visualize the distribution of training times for the expert tracking policies detailed in Section 3.1. All policies were trained on individual NVIDIA A40 GPUs. We see that ending training early based on the most recent tracking error significantly reduces the total cost of compute required to train all experts. A majority of experts finish training in under an hour, and over 30% complete in less than 30 minutes. However, since policies that do not reach the target error threshold are trained until the epoch limit of 3000 epochs, we see that the 5% of rejected policies have an oversized impact on cumulative training cost. This highlights the importance of strong dataset filters that can identify physically-implausible motions before training begins.

Measuring tracking error is difficult for policies lacking a phase variable input that synchronizes them to a reference motion. In order to evaluate the quality of motions produced by non-tracking policies (i.e. group and global controllers), we introduce the metrics of thresholded precision and recall. These metrics are inspired by the thresholded coverage metric used in (Juravsky et al., 2022), with our thresholded recall metric being almost identical to that work’s construction of thresholded coverage. Thresholded recall measures the fraction of a reference motion that is reproduced by a policy when conditioned to generate that motion. To calculate the recall of a policy $\pi$ on a motion sequence $\hat{{\mathbf{m}}}=(\hat{{\mathbf{s}}_{0}},\hat{{\mathbf{s}}_{1}},...,\hat{{\mathbf{s}}_{n}})$, we first roll out a (deterministic) trajectory $\tau=({\mathbf{s}}_{0},{\mathbf{s}}_{1},...,{\mathbf{s}}_{k})$ from $\pi$. We then consider all ten-frame-long sliding windows from $\hat{{\mathbf{m}}}$ and check whether any ten-frame window in $\tau$ is “sufficiently close”, as determined by some threshold ${\epsilon}$. Specifically, we define:

where ${\mathbf{s}}_{x:y}$ denotes the concatenation of frames $({\mathbf{s}}_{x},{\mathbf{s}}_{x+1},...,{\mathbf{s}}_{y})$, and $\mathcal{I}$ denotes an indicator variable. The key difference between this metric and thresholded coverage is that thresholded recall operates on windows of consecutive states, while thresholded coverage only considers individual frames. We choose to construct windows to better capture the temporal structure of the reference motion: for example, a hypothetical trajectory that perfectly imitated $\hat{{\mathbf{m}}}$ in reverse would always produce a perfect thresholded coverage score, but this is not true for thresholded recall.

Complementing this thresholded recall metric is a thresholded precision metric, which considers all the windows in $\tau$ and measures whether any window in $\hat{{\mathbf{m}}}$ is sufficiently close:

While thresholded recall measures the fraction of $\hat{{\mathbf{m}}}$ that the policy imitates, thresholded precision measures the fraction of $\tau$ that imitates a portion of $\hat{{\mathbf{m}}}$. For example, a trajectory that perfectly loops a subset of the reference motion would score very highly on precision, but low on recall. Conversely, a trajectory that perfectly imitated the entire reference clip, but then contained some bizarre additional motions, would have a very high recall score but a very low precision. In practice, we find that the two metrics are correlated, however policies that mostly ignore their conditioning and focus on staying upright (such as the global controller baselines in Section 4.3) will often have a higher precision score than recall.

To evaluate trained policies, we follow the procedure of Juravsky et al. (2022) and sweep over many values of ${\epsilon}$ when calculating thresholded precision and recall. We record trajectories and calculate metrics for all motions in the training dataset, and report the averaged results as a plot. Additionally, to summarize these plots with individual scalars, we report the area under each curve (AUC).

We use thresholded precision and recall metrics to compare our global SuperPADL controller against two baselines trained on the same dataset of 5587 motions. Our first baseline directly applies PADL on the full dataset. Additionally, we train a PADL+BC controller on the entire dataset, instead of on a group of 20 motions. We use the same language encoder architecture as SuperPADL for both baselines and focus on evaluating the motion quality of each method when training on thousands of motions. The policy network sizes are held constant across all three methods. For the PADL and PADL+BC runs, the critic the discriminator are also appropriately scaled up in size.

We report our thresholded precision and recall metrics in Figure 5 and Table 1, observing that both baselines achieve lower precision and recall scores than SuperPADL. Qualitatively, these baseline networks are unable to do much more than stay upright and stumble around, appearing to respond very little to the user’s text command. The PADL+BC network will occasionally reproduce short snippets of simple motions such as jogging. We emphasize that these baselines attempt to apply adversarial reinforcement learning objectives at scale, while SuperPADL only trains small-scale policies with RL. Instead, SuperPADL relies exclusively on supervised learning (through DAGGER) to train the global controller (Ross et al., 2011).

We assess the ability of the SuperPADL global controller to transition between skills in Appendix A. The global controller can successfully transition (i.e. not fall) over 90% of the time, even when transitioning between two skills from different motion groups. Additionally, we evaluate SuperPADL’s ability to respond to language commands in Appendix B, showing that human raters are able to match animations to the appropriate caption a majority of the time.

We also assess the motion quality of our PADL+BC group controllers when compared against a pure-PADL baseline. We randomly select four groups of 20 motions from our dataset and train a controller using both approaches on each group. Note that unlike the PADL models trained in Juravsky et al. (2022), we train pure-PADL group controllers without any language conditioning, instead using the same simple motion index embedding as our PADL+BC controllers. Additionally, we measure the training time required for each network when using a single A40 GPU.

We report our results in Figure 6 and Table 2. We observe that group controllers trained with PADL+BC are able to generate higher-quality motions than vanilla PADL controllers while simultaneously requiring much less GPU time. Qualitatively, we observe that PADL+BC models are less prone to imitating only a subsection of a reference motion than PADL policies. Additionally, we find that PADL+BC models seem to be more successful at looping their generations and avoiding getting stuck. Note that the GPU training time reported for PADL+BC models does not include the time required to train the 20 tracking policies that the group controller distills from. A proper accounting of end-to-end compute cost should include these prerequisite training steps as well. However, as shown in Figure 4, the mean tracking policy time is under one hour, therefore even when considering the cost of training tracking experts, the total cost of training one PADL+BC group controller is lower than that of one pure-PADL group controller. Additionally, the training of 20 separate tracking policies can be more easily distributed across multiple GPUs than the training of a single group controller.