CycleMix: Mixing Source Domains for Domain Generalization in Style-Dependent Data

Let $\mathcal{D}:=\{\mathcal{D}_{i}\}_{i=1}^{S}$ a set of $S$ Source training domains over an input space $\mathcal{X}$. We then observe $n_{i}$ training data points from domain $\mathcal{D}_{i}$, consisting of an input t $\mathbf{x}^{(i)}_{j}$ and label $y^{(i)}_{j}$, i.e. $(\mathbf{x}^{(i)}_{j},y^{(i)}_{j})\sim\mathcal{D}_{i}$. Similarly, let $T:=\{T_{i}\}_{i=1}^{T}$ be a set of $T$ unknown Target domains, while we assume that there exists a single global labeling function $h(x)$ that maps input observations to their labels, $y_{j}^{(i)}=h(\mathbf{x}_{j}^{(i)})$, for all domains $i$ and samples $j$. The aim of Domain Generalization (DG) is to produce a model with parameters $\theta\in\Theta$ which generalizes to both source domains $\mathcal{D}$ and unseen target domains $\mathcal{T}$.

CycleGANs (Zhu et al., 2017) were initially proposed for learning translational image mappings between two domains $X$ and $Y$, in the absence of paired examples. Formally, let $D_{1}$ and $D_{2}$ be source domains for which we aim to learn a mapping $G:D_{1}\rightarrow D_{2}$, such that the distribution of images drawn from $G(D_{1})$ are identical with the distribution of images from $D_{2}$. The novelty of CycleGAN’s lies in the addition of an inverse mapping $F:D_{1}\rightarrow D_{2}$ and Cycle Consistency Loss, to the already established Adversarial Loss of GAN’s. The addition of the Cycle Consistency loss enforces the model to reconstruct a translated image into its original domain.

In our work, we leverage CycleGANs to learn style-mappings between source domains, in the context of Domain Generalization. Our intuition is that by randomly mixing the styles present in source domains and thus synthesizing novel samples, a feature extractor can derive robust representations and learn the features which remain invariant across styles. Specifically, given $S$ source domains we train $S(S-1)/2$ CycleGANs for learning all the possible domain mappings (a single CycleGAN also contains the trained model yielding the inverse mapping between the two domains). An indicative example of translated images between domains is presented in Fig. 2. Having captured the styles of each source domain with the above mappings, we use them as augmentation functions on training data. Specifically, given an image $\mathbf{x}^{(i)}$ drawn from a source domain $D_{i}$, we translate the image to the remaining source domains and then randomly blend them into a single final image. The entire augmentation operation is as follows:

where $G_{ij}$ the GAN for translating images from domain $i$ to domain $j$ and $a_{j}$ a random parameter corresponding to the magnitude of style mixing for each source domain²²2The sum of all parameters $a_{j}$ add up to 1 and are randomly sampled in each minibatch iteration.. After this mixing operation, each image is normalized (i.e., the same preprocessing is applied as for the vanilla model input).

In our experiments we only randomly augment half of the images present in a mini-batch to preserve the information provided by the initial source domains. An example of style mixed images, along with an illustration of our proposed method, is provided in Fig. 1. After augmentation, we pass the mixed images through a feature extractor and train a classifier to attain their final labels.

To validate our approach, we use the publicly available and widely-adopted PACS (Li et al., 2017) dataset. PACS contains images from 4 separate style domains. As its name suggests, samples can either originate from the Photo, Art Painting, Cartoon or Sketch domain and can be one of 7 classes. As in standard DG experimental setups, we follow the leave-one-domain-out cross-validation protocol (Li et al., 2017; Gulrajani and Lopez-Paz, 2021), meaning that in each training iteration we train a model on 3 source domains and evaluate on a single target domain. For our experiments, we use the DomainBed (Gulrajani and Lopez-Paz, 2021) codebase and train a ResNet-50 feature extractor on a single NVIDIA A100 GPU card.

To evaluate the effectiveness of CycleMIX as an augmentation approach, we rank its improvement against established techniques such as CUTOUT, CUTMIX and MIXUP. We also use SagNets as a baseline to demonstrate the efficacy of our method, along with a vanilla ResNet-50³³3Standard image augmentations are used during data loading for each method, such as random resized crop, random horizontal flip, color jitter and normalization.. The results of our experiments are presented in Table 1.

It is apparent that the proposed method has a clear advantage over the baselines, as it surpasses the second best performing model by an average of around 2%. With the exception of Photo, CycleMix yields the best results in every other target domain.