Visually Robust Adversarial Imitation Learning from Videos with Contrastive Learning
Abstract
We propose C-LAIfO, a computationally efficient algorithm designed for imitation learning from videos, even in the presence of visual mismatch between agent and expert domains. We analyze the problem of imitation from expert videos with visual discrepancies, and introduce a solution for robust latent space estimation using contrastive learning and data augmentation. Provided a visually robust latent space, our algorithm performs imitation entirely within this space using off-policy adversarial imitation learning. We conduct a thorough ablation study to justify our design choices and test C-LAIfO on high-dimensional continuous robotic tasks. Additionally, we demonstrate how C-LAIfO can be combined with other reward signals to facilitate learning on a set of challenging hand manipulation tasks with sparse rewards. Our experiments show improved performance compared to baseline methods, highlighting the effectiveness and versatility of C-LAIfO. To ensure reproducibility, we provide open access to our code.
1 Introduction
In recent years, there has been a significant surge in research on imitation learning from expert videos, commonly referred to as the Visual Imitation from Observations (V-IfO) problem. The approach of mimicking experts from videos holds great promise for the future, as it offers a cost-effective way to teach autonomous agents new skills and behaviors. To achieve this goal, prior research has developed methods capable of concurrently addressing two primary challenges of the V-IfO framework: the partial observability of the decision-making process and the absence of expert actions [1]. Despite these advancements, end-to-end state-of-the-art algorithms still face significant barriers in real-world applications due to the assumption that both the expert and learning agent operate within the same environment [1, 2]. For instance, consider the scenario described in Fig. 1, where expert videos are collected under the conditions in Fig. 1(a) and an autonomous agent is deployed in Fig. 1(b) or Fig. 1(c). Current methods are not designed to handle such variations in lighting and background, leading to failures in these contexts. Our goal, in this paper, is to enhance the imitation capabilities of autonomous agents in the presence of visual mismatches.
![Refer to caption](extracted/5676764/Figures/Door.png)
![Refer to caption](extracted/5676764/Figures/door_mismatch.png)
![Refer to caption](extracted/5676764/Figures/Door_color.png)
In this context, our paper introduces a novel end-to-end pipeline for imitation from expert videos with visual mismatch. We begin by analyzing the V-IfO problem with visual mismatch and propose a novel, simple, and computationally efficient algorithm called Contrastive Latent Adversarial Imitation from Observations (C-LAIfO). Notably, C-LAIfO builds upon the recent LAIfO algorithm [1] and achieves visually robust latent state estimation through data augmentation and contrastive learning techniques [3, 4]. We justify each design choice for our algorithm, including the types of data augmentation and contrastive loss, through a comprehensive ablation study. Furthermore, we evaluate the performance of C-LAIfO on the DeepMind Control Suite [5] and compare it against two V-IfO baselines—LAIfO [1] and PatchAIL [2]—as well as DisentanGAIL [6], which serves as a baseline for V-IfO with domain mismatch. Additionally, we show how the reward signal learned from expert data using C-LAIfO can be easily integrated with other signals to enhance efficiency and enable learning in scenarios with sparse rewards. In this context, we extend our algorithm’s evaluation to the Adroit platform for dynamic dexterous manipulation [7]. These additional experiments highlight the versatility of our approach, showcasing its efficacy in handling complex robotic tasks.
2 Related Work
Our work is closely related to the Adversarial Imitation Learning (AIL) framework [8], which has emerged as a promising approach to tackle the Imitation Learning (IL) problem. This framework builds upon a substantial body of research in inverse Reinforcement Learning (RL), where the primary objective is to identify a reward function that enables expert demonstrations (state-action pairs) to be optimal [9, 10, 11, 12, 13, 14]. The derived reward function is then used to train RL agents capable of achieving expert-level performance.
AIL was initially formulated for the fully observable setting with access to expert actions [8, 15] and subsequently extended to observation-only scenarios in follow-up studies [16, 17, 18]. Furthermore, the stability and efficiency of this framework have been further investigated in [19, 20, 21, 22, 23]. In the context of partial observability, AIL has been explored for scenarios involving missing information [24], and in Visual IL where agents learn from video frames as state observations [25]. Relaxing the assumption on the availability of expert actions gives rise to the V-IfO problem, which is the primary focus of our work. State-of-the-art algorithms for the V-IfO setting include PatchAIL [2], which applies AIL directly on the pixel space utilizing a PatchGAN discriminator [26, 27], and LAIfO [1], where AIL operates on a latent representation of the agent state. Notably, these approaches are built on the assumption that both the expert and the learning agent act within the same decision process, which rarely holds true in real-world scenarios.
Our research focuses on the imitation problem from observations in the presence of domain mismatches between the expert and the learning agent. This framework is referred to in the literature under various names, including third-person IL [28], domain-adaptive IL [29], or cross-domain IL [30]. Previous solutions have approached this problem either by decomposing it into sequential stages [31] or using end-to-end methods. Among end-to-end approaches, some have addressed the fully observable setting [29, 32, 30, 33, 34], while others have tackled the visual (partially observable) setting [28, 35, 6, 36].
In the visual setting, [28] and [6] propose methods to extract domain-independent features useful for inferring a domain-independent reward function. The work in [28] learns domain-independent features using an adversarial approach similar to [37], while DisentanGAIL in [6] achieves a similar result by adding a mutual information constraint to the binary cross-entropy loss used for AIL. Similar to our algorithm, these studies formulate fully end-to-end model-free algorithms that avoid costly generative steps during the imitation process. Our approach adopts similar reasoning to [28, 6]; however, we leverage contrastive learning for domain-independent feature extraction and build the entire AIL pipeline (both reward inference and RL step) on this learned feature space, rather than only the reward inference as done in [28, 6]. As shown in our experiments, this leads to significant improvements in performance.
In contrast, other works rely on expensive generative steps to address the mismatch problem. In [35], imitation is performed using expert observation-action pairs through a learned domain-agnostic recurrent state space model [38]. Our algorithm, on the other hand, is model-free and only requires expert observations. In [36], cycle-consistent adversarial networks [27] are used to generate expert videos in the agent’s domain, thus reducing the problem to the standard V-IfO without mismatches. Our approach does not require such a generative step, as it learns a domain-independent feature space directly.
3 Preliminaries
Unless indicated otherwise, we use uppercase letters (e.g., ) for random variables, lowercase letters (e.g., ) for values of random variables, script letters (e.g., ) for sets, and bold lowercase letters (e.g., ) for vectors. Let be the set of integers such that ; we write such that as . We denote with expectation, with probability, and with an -divergence between two distributions of which the Jensen-Shannon divergence, , is a special case.
Partially Observable Markov Decision Process
We model the decision processes as infinite-horizon discounted Partially Observable Markov Decision Processes (POMDPs) described by the tuple , where is the set of states, is the set of actions, and is the set of observations. is the transition probability function where denotes the space of probability distributions over , is the observation probability function, and is the reward function which maps state-action pairs to scalar rewards. Finally, is the initial state distribution and the discount factor. The true environment state is unobserved by the agent. Given an action , the next state is sampled such that , an observation is generated as , and a reward is computed. Note that an MDP is a special case of a POMDP where the underlying state is directly observed.
Reinforcement learning
Given an MDP and a stationary policy , the RL objective is to maximize the expected total discounted return where . A stationary policy induces a normalized discounted state visitation distribution defined as , and we define the corresponding normalized discounted state-action visitation distribution as . Finally, we denote the state value function of as and the state-action value function as . When a function is parameterized with parameters we write .
Generative adversarial imitation learning
Assume we have a set of expert demonstrations generated by the expert policy , a set of trajectories generated by the policy , and a discriminator network parameterized by . Generative adversarial IL [8] optimizes the min-max objective
(1) |
Maximizing (1) with respect to is effectively an inverse RL step where a reward function, , is inferred by leveraging and . On the other hand, minimizing (1) with respect to can be interpreted as an RL step, where the agent aims to minimize its expected cost. It has been demonstrated that optimizing the min-max objective in (1) is equivalent to minimizing , so we are recovering the expert state-action visitation distribution [39].
Modeling the visual mismatch in POMDPs
Traditionally, the V-IfO problem assumes that both the expert and the agent operate within the same POMDP. Throughout this paper we relax this assumption and define two different decision processes: namely a target-POMDP for the agent and a source-POMDP for the expert. The target-POMDP is characterized by the tuple , whereas the source-POMDP is characterized by the tuple . The primary distinction between these POMDPs lies in their observation probability functions. Despite sharing identical state and action spaces, given the same state , the expert’s observation from the source-POMDP and the agent’s observation from the target-POMDP may be different (i.e., we may have ). We refer to this as visual mismatch.
4 Contrastive Latent Adversarial Imitation from Observations
Considering a target-POMDP and a source-POMDP as introduced in the previous section, we can identify two levels of information in the observation space : information related to task completion, and visual distractors that do not contribute to task completion. Thus, we define as , where represents the goal-completion information that is invariant between source-POMDP and target-POMDP; whereas, represents the set of visual distractors that do not contribute to goal completion. We express the source and target observations respectively as and . Our objective is to filter out the visually-distracting information from both the source and the target observations while retaining the goal-completion information to effectively solve the V-IfO problem. This objective can be achieved by attaining domain invariance in a feature space . As a result, our goal becomes to learn such that only goal-completion information is retained while the visually-distracting information is discarded (cf. Appendix A for formal analysis).
In the following, we present the main components of our algorithm C-LAIfO, which performs imitation directly in a domain-invariant feature space . In order to do so, we learn a domain-invariant encoder that can successfully map and to through two main steps. First, we train the encoder alongside the critic networks where . This step is essential for solving the imitation problem and embedding goal-completion information within the latent space . We further train to optimize an auxiliary contrastive loss and perform randomized augmentation of the observations, taking into account the type of visual mismatch between the source and the target domains. This step is crucial to efficiently discard visually-distracting information from .
Adversarial imitation in latent space
Given a domain-invariant feature space , our AIL pipeline is defined as follows. We initialize two replay buffers and to respectively store the sequences of observations generated by the expert and the agent policies, from which we infer the latent state-transitions . We write to streamline the notation. Then, given a discriminator , we write
(2) |
As mentioned, alternating (2) with an RL step using leads to the minimization of [40]. Therefore, we are effectively imitating the expert in the latent space . Note that the presented AIL can only succeed if is domain-invariant and embeds the relevant goal-completion information necessary to solve the imitation problem. Next, we show how our algorithm C-LAIfO addresses this challenge.
Critic and encoder training step
We define the encoder as , a function mapping sequences of observations to the latent space . Specifically, we write and where . When a data augmentation function is applied to the sequence of observations, we write and . We train to optimize
(3) | ||||
s.t. | (4) |
The steps in (3)–(4) follow the deep -network optimization pipeline [41, 42], where we add a contrastive auxiliary loss in (3) defined on the encoder . In (4), the reward function is computed through the AIL step in (2). Note that we do not perform data augmentation when computing since, during reward inference, we are deploying and not training the encoder . In practice, adding data augmentation to AIL in (2) decreases the performance.
In (3), the encoder is trained together with the critic networks () in order to regress in (4), where stands for stop gradient of the encoder parameters . The value of in (4) is computed by summing with the discounted target critic network at time . In (4), where is a clipped exploration noise with the clipping parameter and a univariate normal distribution with zero mean and standard deviation. and are the slow moving weights for the target critic networks. is a replay buffer initialized to store interactions of the agent with the target environment. Note that the latent state-transitions are inferred from sequences of observations using and, as above, we write to streamline the notation (cf. Appendix B for the complete pseudo-code).
By solving the optimization problem in (3)–(4), our primary goal is to train both the encoder network and the critic networks to solve the RL problem with reward . In other words, this step focuses on retaining the goal-completion information within the latent space such that critic networks are successfully learned. We show in our ablation study that backpropagating the gradient from to is an important step for achieving this goal and solving the imitation problem. Similarly, the types of augmentations performed on the sequences of observations and the choice of auxiliary loss play a crucial role in discarding the visually-distracting information from and dealing with the visual mismatch.
Contrastive loss
In the following, we introduce the data augmentation techniques and the auxiliary loss in (3) for C-LAIfO. We opt for a contrastive method as this leads to good empirical results and good computational efficiency. Contrastive learning constructs low-dimensional representations of high-dimensional data by maximizing agreement between augmented views of the same data example via a contrastive loss in the latent space . In our specific case, we define equivalent data as sequences of observations with the same goal-completion information. The data augmentation is randomized by considering a set of pre-determined functions. We will show in our experiments (Section 5) that both the choice of contrastive loss and the set of augmentation functions play an important role in filtering out visually-distracting information.
First, a stochastic data augmentation module transforms any given sequence of observations into two views, denoted and , which are denoted as the positive pairs. Note that two positive pairs must contain the same goal-completion information. Next, the encoder extracts representation vectors from augmented sequences of observations. We write and . Finally, we apply the contrastive loss function
(5) |
where is an indicator function equal to 1 if , denotes a temperature parameter, and denotes the cosine similarity. We sample a batch of sequences of observations from the buffer and define pairs of augmented sequences derived from the batch, resulting in data points. The negative data points are not sampled explicitly. Instead, given a positive pair, we treat the other augmented data points within the batch as negatives. Note that the loss in (5) is called the normalized temperature-scaled cross entropy loss or the Information Noise-Contrastive Estimation (InfoNCE) loss [43] and represents an upper bound of the negative mutual information between positive pairs. Therefore, by minimizing (5) as in (3) we are maximizing the mutual information between positive pairs in the latent space .
5 Experiments
In this section, we present our experimental results. We begin with an ablation study to justify the key design choices of our algorithm. Next, we demonstrate how C-LAIfO effectively handles various types of visual mismatches in the V-IfO setting. Finally, we showcase how C-LAIfO facilitates learning in challenging robotic manipulation tasks with sparse rewards and realistic visual inputs.
Ablation study
Given the environments in Fig. 2, we perform the following ablations:
- 1.
-
2.
Gradient backpropagation: We highlight the necessity of backpropagating the gradient from to in (3) for solving the imitation problem and embedding the goal-completion information in the latent space .
-
3.
Data augmentation: We emphasize the importance of selecting the appropriate augmentation for a given mismatch, showing that a mismatch-informed augmentation outperforms general augmentations or no augmentation.
The results of these experiments are summarized in Table 1, which includes results for C-LAIfO and its various configurations. All the learning curves are provided in Appendix D. In Table 1, C-LAIfO is implemented as described in Section 4, with the data augmentation function defined as a brightness transformation. In BYOL-LAIfO, we retain the design identical to C-LAIfO except for replacing the InfoNCE loss in (5) with BYOL [4]. In C-LAIfO w/o backprop, we disable gradient backpropagation from to in (3). Lastly, in C-LAIfO full aug and C-LAIfO w/o aug, we respectively modify the data augmentation function to include full augmentation (brightness, color, and geometric transformations, as detailed in Appendix 5) and no augmentation. Notably, without backpropagating the gradient from to , C-LAIfO fails to solve the imitation task even in the simplest mismatch scenario in (2(b)). This result demonstrates the importance of this step for embedding goal-completion information in . Conversely, without a well-designed , C-LAIfO can solve the imitation problem with minor visual mismatch but struggles with more challenging visual distractors. These results highlight the critical role of properly defining for efficient visual generalization in . General might still achieve this goal, but much less efficiently. Finally, the superior performance of the InfoNCE loss in (5) compared to BYOL is particularly evident in handling the most difficult mismatches in (2(e)) and (2(f)).
![Refer to caption](extracted/5676764/Figures/walker_walk_delta=0.20.png)
![Refer to caption](extracted/5676764/Figures/walker_walk_delta=0.15.png)
![Refer to caption](extracted/5676764/Figures/walker_walk_delta=0.05.png)
![Refer to caption](extracted/5676764/Figures/walker_walk_delta=-0.05.png)
![Refer to caption](extracted/5676764/Figures/walker_walk_delta=-0.15.png)
![Refer to caption](extracted/5676764/Figures/walker_walk_delta=-0.25.png)
Source Env (2(a)), Performance = | |||||
Target Env | (2(b)) | (2(c)) | (2(d)) | (2(e)) | (2(f)) |
C-LAIfO | |||||
BYOL-LAIfO | |||||
C-LAIfO w/o backprop | - | - | - | ||
C-LAIfO full aug | - | - | - | ||
C-LAIfO w/o aug | - | - | - |
Visual Imitation from Observations with mismatch
In this section, we test C-LAIfO in the V-IfO setting with different types of mismatches (cf. Fig. 3) and compare it with three baselines: LAIfO [1] and PatchAIL [2], both equipped with the same used for C-LAIfO, and DisentanGAIL [6]. The results, summarized in Table 2, demonstrate that C-LAIfO successfully addresses the V-IfO with mismatch problem, achieving superior performance compared to all the baselines across the proposed mismatches. All the learning curves are provided in Appendix D. For the Light experiment, is defined as a brightness transformation; while for the others it is defined as a color transformation. Details on are provided in Appendix 5. Furthermore, to assess whether C-LAIfO achieves domain invariance in the feature space , we perform PCA and t-SNE analyses on the latent space learned by different algorithms during training. Specifically, focusing on the Light experiment in Table 2, Fig. 4(a) compares the latent space learned by LAIfO in [1], C-LAIfO, and LAIfO with data augmentation. Additionally, Fig. 4(b) investigates whether the latent space learned by C-LAIfO during the Full experiment can generalize to the unseen environment depicted in Fig. 3(g). The results of this analysis are shown in Fig. 4, which includes the PCA visualization. For the t-SNE visualization and additional details refer to Appendix E.
![Refer to caption](extracted/5676764/Figures/walker_walk_delta=-0.25.png)
![Refer to caption](extracted/5676764/Figures/walker_walk_delta=0.20.png)
![Refer to caption](extracted/5676764/Figures/Source_color_body.png)
![Refer to caption](extracted/5676764/Figures/Source_floor.png)
![Refer to caption](extracted/5676764/Figures/Source_bg.png)
![Refer to caption](extracted/5676764/Figures/Source_all_tg.png)
![Refer to caption](extracted/5676764/Figures/source_optimal_colors_unseen.png)
Light | Body | Floor | Background | Full | |
C-LAIfO | |||||
LAIfO [1] w/ data aug | |||||
DisentanGAIL [6] | |||||
PatchAIL [2] w/ data aug |
![Refer to caption](x1.png)
Improving RL using expert videos
In the following section, we examine a scenario where the reward learned by C-LAIfO can be combined with a sparse reward that the agent collects through interaction with the environment. Consequently, the RL problem aims to maximize the total reward with learned through the AIL step in (2). We conduct experiments on the Adroit platform for dynamic dexterous manipulation [7] and compare C-LAIfO with DrQ-v2 [45], an RL-only baseline. In these experiments, both C-LAIfO and DrQ-v2 use an encoder to process pixel observations and extract embeddings in , which are then concatenated with the robot sensory observations. Importantly, expert’s sensory observations are not used in the imitation problem as we only assume access to videos of experts. We denote our approach as RL+C-LAIfO since it maximizes rather than only alone as in the standard imitation learning problem. The results, summarized in Table 3, show that DrQ-v2 fails to learn in two out of three environments. In contrast, C-LAIfO successfully leverages expert videos with visual mismatch to facilitate learning. Note that the goal of these experiments is not merely to compare RL+C-LAIfO with DrQ-v2 but to demonstrate how C-LAIfO can leverage expert videos, even with visual mismatch, to enable learning in challenging robotic tasks. We provide all the learning curves in Appendix D.
![Refer to caption](extracted/5676764/Figures/Door.png)
![Refer to caption](extracted/5676764/Figures/door_mismatch.png)
![Refer to caption](extracted/5676764/Figures/Door_color.png)
![Refer to caption](extracted/5676764/Figures/hammer.png)
![Refer to caption](extracted/5676764/Figures/hammer_mismatch.png)
![Refer to caption](extracted/5676764/Figures/hammer_color.png)
![Refer to caption](extracted/5676764/Figures/pen.png)
![Refer to caption](extracted/5676764/Figures/pen_mismatch.png)
![Refer to caption](extracted/5676764/Figures/pen_color.png)
Door | Hammer | Pen | ||||
Light | Color | Light | Color | Light | Color | |
Expert | ||||||
DrQ-v2 [45] | ||||||
RL+C-LAIfO |
6 Conclusion
In this work, we analyze the V-IfO problem in the presence of visual mismatches and propose a novel algorithm named C-LAIfO as an effective solution. Through comprehensive ablation studies, we provide insights into our design choices and demonstrate the superior performance of our approach compared to a range of baselines in imitation from videos under various mismatch scenarios. Furthermore, we illustrate how C-LAIfO effectively utilizes expert videos with visual mismatches to facilitate learning in challenging hand manipulation tasks characterized by sparse rewards and realistic visual inputs.
Limitations and future work
A main limitation of the current approach is given by the reliance of C-LAIfO on a well-designed, possibly mismatch-informed, data augmentation function. As illustrated in our ablations in Section 5, general augmentation can lead to poor performance or can remarkably reduce the algorithmic sample efficiency. Furthermore, it can be challenging to design effective augmentations for certain types of mismatches. To address this problem, exploring generative models for automatic data augmentation represents an interesting research direction. Generative models could produce diverse, mismatch-informed augmentations, potentially overcoming the limitations of manually designed strategies. Alternatively, investigating different auxiliary losses that are less reliant on augmentation techniques represents another interesting direction. Finally, future work will be devoted to go beyond simulated environments and test our algorithms on hardware in real-world scenarios.
7 Acknowledgements
Vittorio Giammarino and Ioannis Ch. Paschalidis were partially supported by the NSF under grants IIS-1914792, CCF-2200052, and ECCS-2317079, by the ONR under grant N00014-19-1-2571, by the DOE under grant DE-AC02-05CH11231, by the NIH under grant UL54 TR004130, and by Boston University. James Queeney was exclusively supported by Mitsubishi Electric Research Laboratories.
References
- Giammarino et al. [2024] V. Giammarino, J. Queeney, and I. Paschalidis. Adversarial imitation learning from visual observations using latent information. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=ydPHjgf6h0.
- Liu et al. [2022] M. Liu, T. He, W. Zhang, Y. Shuicheng, and Z. Xu. Visual imitation learning with patch rewards. In International Conference on Learning Representations, 2022.
- Chen et al. [2020] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pages 1597–1607. PMLR, 2020.
- Grill et al. [2020] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems, 33:21271–21284, 2020.
- Tassa et al. [2018] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. d. L. Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018.
- Cetin and Celiktutan [2020] E. Cetin and O. Celiktutan. Domain-robust visual imitation learning with mutual information constraints. In International Conference on Learning Representations, 2020.
- Kumar [2016] V. Kumar. Manipulators and Manipulation in high dimensional spaces. PhD thesis, University of Washington, Seattle, 2016. URL https://digital.lib.washington.edu/researchworks/handle/1773/38104.
- Ho and Ermon [2016] J. Ho and S. Ermon. Generative adversarial imitation learning. Advances in Neural Information Processing Systems, 29, 2016.
- Russell [1998] S. Russell. Learning agents for uncertain environments. In Proceedings of the Annual Conference on Computational Learning Theory, pages 101–103, 1998.
- Ng et al. [2000] A. Y. Ng, S. Russell, et al. Algorithms for inverse reinforcement learning. In International Conference on Machine Learning, volume 1, page 2, 2000.
- Abbeel and Ng [2004] P. Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning. In International Conference on Machine Learning, page 1, 2004.
- Syed and Schapire [2007] U. Syed and R. E. Schapire. A game-theoretic approach to apprenticeship learning. Advances in Neural Information Processing Systems, 20, 2007.
- Ziebart et al. [2008] B. D. Ziebart, A. L. Maas, J. A. Bagnell, A. K. Dey, et al. Maximum entropy inverse reinforcement learning. In AAAI Conference on Artificial Intelligence, volume 8, pages 1433–1438. Chicago, IL, USA, 2008.
- Syed et al. [2008] U. Syed, M. Bowling, and R. E. Schapire. Apprenticeship learning using linear programming. In International Conference on Machine Learning, pages 1032–1039, 2008.
- Fu et al. [2017] J. Fu, K. Luo, and S. Levine. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint arXiv:1710.11248, 2017.
- Torabi et al. [2018] F. Torabi, G. Warnell, and P. Stone. Generative adversarial imitation from observation. arXiv preprint arXiv:1807.06158, 2018.
- Yang et al. [2019] C. Yang, X. Ma, W. Huang, F. Sun, H. Liu, J. Huang, and C. Gan. Imitation learning from observations by minimizing inverse dynamics disagreement. Advances in Neural Information Processing Systems, 32, 2019.
- Cheng et al. [2021] Z. Cheng, L. Liu, A. Liu, H. Sun, M. Fang, and D. Tao. On the guaranteed almost equivalence between imitation learning from observation and demonstration. IEEE Transactions on Neural Networks and Learning Systems, 2021.
- Kostrikov et al. [2018] I. Kostrikov, K. K. Agrawal, D. Dwibedi, S. Levine, and J. Tompson. Discriminator-actor-critic: Addressing sample inefficiency and reward bias in adversarial imitation learning. arXiv preprint arXiv:1809.02925, 2018.
- Blondé and Kalousis [2019] L. Blondé and A. Kalousis. Sample-efficient imitation learning via generative adversarial nets. In International Conference on Artificial Intelligence and Statistics, pages 3138–3148. PMLR, 2019.
- Kostrikov et al. [2019] I. Kostrikov, O. Nachum, and J. Tompson. Imitation learning via off-policy distribution matching. arXiv preprint arXiv:1912.05032, 2019.
- Zhu et al. [2020] Z. Zhu, K. Lin, B. Dai, and J. Zhou. Off-policy imitation learning from observations. Advances in Neural Information Processing Systems, 33:12402–12413, 2020.
- Kidambi et al. [2021] R. Kidambi, J. Chang, and W. Sun. Mobile: Model-based imitation learning from observation alone. Advances in Neural Information Processing Systems, 34:28598–28611, 2021.
- Gangwani et al. [2020] T. Gangwani, J. Lehman, Q. Liu, and J. Peng. Learning belief representations for imitation learning in POMDPs. In Uncertainty in Artificial Intelligence, pages 1061–1071. PMLR, 2020.
- Rafailov et al. [2021] R. Rafailov, T. Yu, A. Rajeswaran, and C. Finn. Visual adversarial imitation learning using variational models. Advances in Neural Information Processing Systems, 34:3016–3028, 2021.
- Isola et al. [2017] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1125–1134, 2017.
- Zhu et al. [2017] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 2223–2232, 2017.
- Stadie et al. [2017] B. C. Stadie, P. Abbeel, and I. Sutskever. Third-person imitation learning. arXiv preprint arXiv:1703.01703, 2017.
- Kim et al. [2020] K. Kim, Y. Gu, J. Song, S. Zhao, and S. Ermon. Domain adaptive imitation learning. In International Conference on Machine Learning, pages 5286–5295. PMLR, 2020.
- Raychaudhuri et al. [2021] D. S. Raychaudhuri, S. Paul, J. Vanbaar, and A. K. Roy-Chowdhury. Cross-domain imitation from observations. In International Conference on Machine Learning, pages 8902–8912. PMLR, 2021.
- Zhang et al. [2023] J. Z. Zhang, S. Yang, G. Yang, A. L. Bishop, S. Gurumurthy, D. Ramanan, and Z. Manchester. Slomo: A general system for legged robot motion imitation from casual videos. IEEE Robotics and Automation Letters, 2023.
- Fickinger et al. [2021] A. Fickinger, S. Cohen, S. Russell, and B. Amos. Cross-domain imitation learning via optimal transport. In International Conference on Learning Representations, 2021.
- Gangwani et al. [2022] T. Gangwani, Y. Zhou, and J. Peng. Imitation learning from observations under transition model disparity. arXiv preprint arXiv:2204.11446, 2022.
- Giammarino et al. [2023] V. Giammarino, J. Queeney, L. C. Carstensen, M. E. Hasselmo, and I. C. Paschalidis. Opportunities and challenges from using animal videos in reinforcement learning for navigation. IFAC-PapersOnLine, 56(2):9056–9061, 2023.
- Okumura et al. [2020] R. Okumura, M. Okada, and T. Taniguchi. Domain-adversarial and-conditional state space model for imitation learning. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5179–5186. IEEE, 2020.
- Choi et al. [2023] S. Choi, S. Han, W. Kim, J. Chae, W. Jung, and Y. Sung. Domain adaptive imitation learning with visual observation. In Advances in Neural Information Processing Systems, 2023.
- Ganin and Lempitsky [2015] Y. Ganin and V. Lempitsky. Unsupervised domain adaptation by backpropagation. In International Conference on Machine Learning, pages 1180–1189. PMLR, 2015.
- Hafner et al. [2019] D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pages 2555–2565. PMLR, 2019.
- Ghasemipour et al. [2020] S. K. S. Ghasemipour, R. Zemel, and S. Gu. A divergence minimization perspective on imitation learning methods. In Proceedings of the Conference on Robot Learning, pages 1259–1277. PMLR, 2020.
- Goodfellow et al. [2020] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139–144, 2020.
- Mnih et al. [2013] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
- Mnih et al. [2015] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015.
- Oord et al. [2018] A. v. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.
- Lillicrap et al. [2015] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.
- Yarats et al. [2021] D. Yarats, R. Fergus, A. Lazaric, and L. Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645, 2021.
- Wang et al. [2022] C. Wang, X. Luo, K. Ross, and D. Li. Vrl3: A data-driven framework for visual deep reinforcement learning. Advances in Neural Information Processing Systems, 35:32974–32988, 2022.
Appendix A Analysis
In the following, we show how domain invariance in the feature space leads to the same V-IfO guarantees as in [1], even in the presence of visual mismatches.
Proposition 1.
Consider source and target POMDPs respectively defined by the tuples and . Let , where is an observations set invariant between source and target POMDP, and is a set of visual distractors. We write . Assume such that . Then, the filtering posterior distributions and do not depend on the policy and are invariant between source and target POMDPs.
Proof.
Considering the definition of the latent variable , which only depends on and not the visually-distracting information , we can write
Then, by leveraging Bayes rule we have that
where the denominator can be seen as a normalizing factor. Note that is the same for both the source and target POMDPs by the definition of above. Therefore, has no dependence on the policy and is invariant between source and target POMDP.
Similarly, for we have that
Because does not depend on and is the same for both source and target POMDP, the result also holds for . ∎
Appendix B Pseudo-code and hyperparameters
Hyperparameter Name | Value |
Frames stack | |
Discount factor | |
Image size | |
Batch size | |
Optimizer | Adam |
Learning rate | |
Discriminator learning rate | |
Target update rate | |
Clip value | |
Temperature parameter |
Appendix C Data augmentation
The operations used in our data augmentation functions for the experiments in Section 5 are summarized in Table 5. Fig. 6 shows an example of color augmentation as implemented for the experiments in Table 2, where we randomly perform all the operations in the color transformations row in Table 5. On the other hand, Fig. 7 shows only a brightness transformation as implemented in the ablation study in Table. 1. Full augmentation in Table 1 performs all the operations in Table 5. For additional details refer to our code111https://github.com/VittorioGiammarino/C-LAIfO.
Operations | |
Color transformations | Brightness |
Contrast | |
Saturation | |
Hue | |
Grayscale | |
Gaussian blur | |
Invert | |
Affine transformations | Horizontal flip |
Vertical flip | |
Resized crop |
![Refer to caption](extracted/5676764/Figures/Aug_example_3mln.png)
![Refer to caption](extracted/5676764/Figures/adroit_color_aug.png)
![Refer to caption](extracted/5676764/Figures/dmc_light_aug.png)
![Refer to caption](extracted/5676764/Figures/adroit_light_aug2.png)
Appendix D Learning curves
All the experiments are run using Nvidia-A40 GPUs on an internal cluster. For each algorithm, we run two experiments in parallel on the same GPU and each experiment takes 1 to 4 days depending on the simulated environment and the considered algorithm. For all the implementation details refer to our code.
Fig. 8, Fig. 9, and Fig. 10 show the learning curves for the results in Table 1, Table 2, and Table 3, respectively.
![Refer to caption](extracted/5676764/Figures/walker_walk_delta=0.20.png)
![Refer to caption](extracted/5676764/Figures/walker_walk_delta=0.15.png)
![Refer to caption](extracted/5676764/Figures/walker_walk_delta=0.05.png)
![Refer to caption](extracted/5676764/Figures/walker_walk_delta=-0.05.png)
![Refer to caption](extracted/5676764/Figures/walker_walk_delta=-0.15.png)
![Refer to caption](extracted/5676764/Figures/walker_walk_delta=-0.25.png)
![Refer to caption](x3.png)
![Refer to caption](extracted/5676764/Figures/walker_walk_delta=-0.25.png)
![Refer to caption](extracted/5676764/Figures/walker_walk_delta=0.20.png)
![Refer to caption](extracted/5676764/Figures/Source_color_body.png)
![Refer to caption](extracted/5676764/Figures/Source_floor.png)
![Refer to caption](extracted/5676764/Figures/Source_bg.png)
![Refer to caption](extracted/5676764/Figures/Source_all_tg.png)
![Refer to caption](x4.png)
![Refer to caption](extracted/5676764/Figures/Door.png)
![Refer to caption](extracted/5676764/Figures/door_mismatch.png)
![Refer to caption](extracted/5676764/Figures/Door_color.png)
![Refer to caption](extracted/5676764/Figures/hammer.png)
![Refer to caption](extracted/5676764/Figures/hammer_mismatch.png)
![Refer to caption](extracted/5676764/Figures/hammer_color.png)
![Refer to caption](extracted/5676764/Figures/pen.png)
![Refer to caption](extracted/5676764/Figures/pen_mismatch.png)
![Refer to caption](extracted/5676764/Figures/pen_color.png)
![Refer to caption](x5.png)
Appendix E PCA and t-SNE analysis
In Fig. 11 we compare the latent space learned by LAIfO, C-LAIfO, and LAIfO with data augmentation in the Light setting from Table 2 and Fig. 8. Specifically, we define source and target as the observations generated by an optimal policy in the source and target POMDPs, respectively. Similarly, source random and target random are observations generated by random policies. These observations are processed with the encoder trained using the respective algorithms. We perform PCA and t-SNE on this set of latent variables and plot the first two principal components. The results show that C-LAIfO is the only algorithm capable of filtering out visual distractors and clustering together data points with the same goal-completion information. In Fig. 12, we focus on C-LAIfO and test for generalization to the unseen environment in Fig. 12(c). We train on the Full setting from Table 2 and Fig. 9 and perform PCA and t-SNE as described. In Fig. 12, the Full figures have (12(a))-(12(b)) as source-POMDP and (12(e))-(12(f)) as target-POMDP. Similarly the Full-unseen figures have (12(c))-(12(d)) as source-POMDP and (12(e))-(12(f)) as target-POMDP. The results in the Full-unseen setting match those in the Full experiment, indicating that trained on (12(a)) can successfully generalize to (12(c)).
![Refer to caption](extracted/5676764/Figures/source_optimal_light.png)
![Refer to caption](extracted/5676764/Figures/source_random_light.png)
![Refer to caption](extracted/5676764/Figures/target_optimal_light.png)
![Refer to caption](extracted/5676764/Figures/target_random_light.png)
![Refer to caption](x1.png)
![Refer to caption](x6.png)
![Refer to caption](extracted/5676764/Figures/source_optimal_colors.png)
(Full).
![Refer to caption](extracted/5676764/Figures/source_random_colors.png)
(Full).
![Refer to caption](extracted/5676764/Figures/source_optimal_colors_unseen.png)
(Full-unseen).
![Refer to caption](extracted/5676764/Figures/source_random_colors_unseen.png)
(Full-unseen).
![Refer to caption](extracted/5676764/Figures/target_optimal_colors.png)
![Refer to caption](extracted/5676764/Figures/target_random_colors.png)
![Refer to caption](x2.png)
![Refer to caption](x7.png)