Curiosity & Entropy Driven Unsupervised RL in Multiple Environments

Instead of just using the worst-performing trajectories to learn an exploration policy as done in [1], we construct a probability distribution over the entropies of all the trajectories. The distribution is designed such that the trajectories with the worst/lowest entropies have the highest probabilities and vice-versa. We achieve this by computing the probability of the $i^{th}$ trajectory as:

where $e_{i}$ is the state visitation entropy of the $i^{th}$ trajectory

Then, we simply sample a fixed number of trajectories from this distribution and use these sampled trajectories to learn the exploration policy. Thus, the policy will learn not only from the worst-performing trajectories (although they will constitute the majority of the sampled trajectories) but also from a few easier/good trajectories. The intuition behind this approach was that sampling from a probability distribution would introduce some stochasticity and allow the policy to generalize better by learning from both, easy and difficult trajectories.

First, we will define $\alpha$ in this scenario.

Consider a random variable $X$ with a cumulative density function (CDF) as follows:

We can define the Value-at-Risk (VaR) as follows:

where $\alpha$ is our confidence level and the VaR is essentially the $\alpha$-percentile.

In [1], the objective function is defined as the Conditional Value-at-Risk (CVaR), i.e., the mean of the $\alpha$-percentile entropies as below:

Thus, when $\alpha$ is lower, the lower the probability is of a bad exploration outcome after training for a given environment from a class of Controlled Markov Processes as we hedge more against such outcomes. In the original paper, [1], the confidence level ($\alpha$) was kept constant throughout training. We attempt to improve overall results by dynamically changing $\alpha$ during training. We tried several variations of this idea such as increasing $\alpha$ over time, using different scheduling techniques, and decreasing $\alpha$ over time. We obtained the best results when the percentile was decreased by 1 every $x$ epochs where:

That new percentile is then used for $\alpha$, where $\alpha$ is $percentile/num\_batches$.

This makes sense because as training progresses, we should keep decreasing the $\alpha$ value so that our model improves more over time by lowering its tolerance for ”bad outcomes”. Essentially, as our model learns, it will do better for the easier trajectories and we want it to continue learning to perform better for harder and harder trajectories.

Another approach we tried to improve performance was to increase the KL Divergence threshold. In the original paper, they use a threshold on the KL Divergence between the policy from the previous epoch (behavioral policy) and the latest/updated policy (target policy) as a stopping condition for the number of off-policy optimization steps taken within an epoch of training. The behavioral policy is used to collect trajectories at the beginning of each epoch and the off-policy optimizations are done using this collected data within the epoch. Thus, by increasing the threshold, the target policy is allowed to diverge more from the behavioral policy during the off-policy optimization stage.

The original KL Divergence threshold may have been too restrictive, constraining the policy to not explore the environment freely. Typically, the KL divergence is used in Trust Region Policy Optimization (TRPO) to make learning more conservative and to prevent overshooting over the local minima during optimization. This is achieved by preventing the learnt policy from diverging too much from a fixed starting policy. The lower the threshold for the KL Divergence is, the more conservative the learning will be. However, such a conservative approach may be excessive for the pretraining phase where there is no extrinsic reward from the environment and only an intrinsic reward. Our intuition is that during this phase it is better to learn a more robust policy that can explore adequately and thus it is better to allow a more free learning process.

When someone is curious, they have a desire to know something that they don’t currently know. In RL, [4] proposed an analogous definition of curiosity for an agent as the error in the agent’s ability to predict the consequences of its own actions. The more uncertain an agent is about what will occur after taking some action, the more curious it is said to be. If an agent can accurately predict what will happen given an action, the agent is less curious. We use curiosity to try to improve the performance of the method from [1]. The idea behind this approach is that if the agent is more unsure of the outcome of taking an action in a given state, then the agent has not explored the current space enough and needs to do so.

Curiosity-driven exploration is implemented as described in [4]. For the sake of brevity, we will simply summarize the method here, please see [4] for more details. There are two components to this approach. The first involves a forward dynamics model that tries to predict the next state given the current state and the action taken in the current state as input. The error in this model’s prediction is defined as the curiosity term. The policy is trained to maximize this curiosity term while the forward dynamics model is optimized to minimize the error in its predictions. The 2nd component termed the inverse dynamics model tries to predict the action taken to make the transition between two given consecutive states. The purpose of this component is to learn a mapping from a higher dimensional state space to a lower dimensional feature space that encodes all the essential state information. For our experiments, due to the constraints of limited compute and time, we were only able to train and test on relatively simpler environments with very low dimensional observation spaces. Thus, for the curiosity experiment, we only implemented and used the forward dynamics model. We do not learn a state encoder or an inverse dynamics model due to the low dimensional observation space.

Expanding upon the preceding discussion, our approach here involves leveraging the $\alpha$-percentile most curious trajectories to facilitate the learning process of the policy. This strategic choice is driven by the intention to compel the policy to further explore the trajectories it is most uncertain about as these trajectories probably cover states that have not been explored sufficiently by the policy. By selecting trajectories within the $\alpha$-percentile highest curiosity scores, we encourage the model to focus on and learn from instances that may initially seem less explored. This deliberate optimization of the policy on only the most uncertain trajectories is motivated by the belief that understanding and effectively navigating challenging and unknown scenarios is vital in enhancing the overall robustness and adaptability of the agent.

We observe that introducing curiosity drastically improved performance for a more complex environment like Ant. It did not show its full potential in simpler environments like Grid World as the model can not be curious enough in such environments where it can easily learn and know the consequences of its actions. Just increasing the KL Divergence threshold also improved the performance more visibly in the Ant environment. Finally, decreasing $\alpha$ over time allows us to learn from worse and worse trajectories as the model gets better over time and has shown positive effects to a certain extent.