URRL-IMVC: Unified and Robust Representation Learning for Incomplete Multi-View Clustering

KNN Imputation (Section 3.2) provides additional information for missing views, but the retrieved nearest neighbors may contain noise and be unreliable. To address this issue, we propose the Neighbor Dimensional Encoder (NDE), which is a series of customized Transformer Encoders (Vaswani et al., 2017), with each one dedicated to a view to fuse its KNN input and filter out noise, formulated as:

The $v$th Transformer Encoder corresponding to the $v$th view is represented with $f_{\scriptscriptstyle NDE}^{(v)}(\cdot\ ;{\bm{\theta}}_{\scriptscriptstyle NDE}^{(v)})$ in equation 6, and ${\bm{\theta}}_{\scriptscriptstyle NDE}^{(v)}$ is its parameters. The input KNN sequence from the $v$th view $\bar{{\bm{x}}}^{(v)}$ is first processed to add the Cosine Distance-based Positional Encoding (CDPE), then it is passed through the Transformer Encoder with the KNN mask $\bar{\bm{m}}$. Finally, only the first vector from the output sequence is chosen as the output, denoted as $[0,:]$.

Below we introduce the two key customizations of the Transformer Encoders in NDE: the CDPE and the output choice.

The View Dimensional Encoder (VDE) is designed to fuse view representations and obtain unified embedding. As depicted in Figure 2, it consists of two parts, with firstly a Feed-Forward Network (FFN) to map the representations of different dimensions to the same latent space, and then followed by a Transformer Encoder for fusion. The FFN consists of three fully connected (FC) layers, without normalization or dropout layers, which can be detrimental to the stability of training. The FFN of VDE can be formulated as:

In the equation, $\oplus$ represents the concatenate operation, $\sigma$ is the activation function, $\bm{W}$ and $\bm{b}$ are the weight matrix and bias vector of the FC layer respectively.

The Transformer Encoder part of the VDE can be explained as,

in which $f_{\scriptscriptstyle VDE-T}(\cdot\ ;{\bm{\theta}}_{\scriptscriptstyle VDE-T})$ is the VDE Transformer Encoder structure. The view representations ${\bm{x}}_{\scriptscriptstyle VDE-F}$ are passed through the Transformer Encoder along with the generated Three-level Adaptive Mask (TAM) $TAM(\bar{\bm{m}},{\bm{m}})$. The output sequence is averaged for fusion. The Transformer Encoder in VDE is also customized but different from that in NDE. The key difference lies in that views are permutation invariant, i.e., changing the order of views should yield the same output, while KNN has an order. Based on this difference, the two key customizations of the VDE Transformer are designed as:

The Decoder in our model is designed as a compact 4-layer FFN, to reconstruct the input from the unified embedding. Similar to the FFN in VDE, we removed its normalization and dropout layer for better stability. Through our experiments, we have observed that a deep and complex Decoder does not necessarily improve the clustering performance and may even have negative effects in certain cases. One possible explanation for this phenomenon is that a shallow and simple Decoder serves as a regularization technique on the embedding space, and prevents it from collapsing. This regularization effect is similar to the Locality-preserving Constraint proposed by Huang et al. (2014), which helps preserve the local structure of the data. The process of Decoder is formulated as,

The auto-encoder we have designed extracts robust representations and captures the inherent structures of data. However, these inherent structures may not necessarily follow a cluster-oriented distribution. To enhance the clustering performance, we introduce a Clustering Module inspired by DEC (Xie et al., 2016). Below we describe its procedures. First, after auto-encoder pretraining, a traditional clustering method is adopted to initialize the cluster centers ${\bm{\theta}}_{C}\in\mathbb{R}^{d_{c}\times d_{e}}$ from embedding ${\bm{z}}$. Then, during each iteration of joint training, a similarity matrix ${\bm{q}}\in[0,1]^{N\times d_{c}}$ is generated between ${\bm{\theta}}_{C}$ and embedding ${\bm{z}}$ with Student’s t-distribution, in which $q_{ij}$ represents the possibility sample $i$ belongs to cluster $j$, and ${\bm{c}}\in\mathbb{R}^{d_{c}}$ defined in equation 4 is a row of ${\bm{q}}$. With the similarity matrix ${\bm{q}}$, we calculate the target distribution ${\bm{p}}$ as,

in which $f_{j}$ is the soft cluster size $\sum_{i=0}^{N-1}q_{ij}$. The training target is the KL-Divergence between ${\bm{p}}$ and ${\bm{q}}$ (equation 22) which is defined in Section 3.3.1. Finally, after joint training, the clustering result can be obtained by finding the maximum possibility in each row of ${\bm{q}}$. The detailed training procedures are described in Section 3.3.2.

The KNN search is conducted separately in each view, taking into account the incomplete condition. For existing views, the KNN is directly obtained, while for missing views, existing views from the same sample are used for searching KNN. Specifically, for a missing view ${\bm{x}}_{i}^{(v)}$ of sample ${\bm{x}}_{i}$, we first find all existing views ${\bm{x}}_{i}^{(b)}$ of the same sample. Then we iterate through the KNN samples ${\bm{x}}_{j}^{(b)}$ of these existing views to check if ${\bm{x}}_{j}^{(v)}$ exists. If it does, we appended it to the KNN list of the missing view. Finally, we select the top $k$ samples from the KNN list as the imputation for the missing view. If the length of the KNN list is less than $k$, then the remaining positions are filled with zeros. Along with the KNN imputation $\bar{\bm{x}}$, a missing indicator matrix $\bar{\bm{m}}\in\mathbb{R}^{k\times V}$ is generated, where 1 represents a position filled with a KNN sample and 0 represents a position filled with zeros. The detailed procedures are listed in Algorithm 1, Appendix A.

Our framework is inspired by the denoising auto-encoder, which helps to learn robust representations by introducing noise during training. Three types of noise are designed, including Gaussian noise, random dropout, and view dropout. Gaussian noise helps prevent overfitting by introducing variability in the input data. Random dropout, functional as a regularization technique, encourages the model to learn more robust features by forcing it to rely on different subsets of the input data. View dropout is a noise specifically designed for the IMVC task. It randomly drops out one or more views from the input data during training, encouraging the model to learn representations that are more robust to view missing conditions.

Below are the formulations of the three kinds of augmentation we used. By combining equation 17 and 18, the KIDA operation in equation 1 is obtained.

The view dropout augmentation is shown in equation 17, where random views are dropped out with a possibility of $\phi_{1}$, before the KNN Imputation $KNNI(\cdot)$ (Algorithm 1) step. The dropout mask is applied with element-wise multiplication, denoted by $\odot$. The masked views are regarded as missing views in both the KNN Imputation and the auto-encoder network. After the KNN Imputation, in equation 18, Gaussian noise is added to the input data to introduce variability, whose intensity is controlled by hyperparameter $\phi_{2}$. After that, random values in the input are set to zero with a probability of $\phi_{3}$, which is the random dropout augmentation. Finally, the augmented input $\bar{\bm{x}}^{\prime}$ along with the corresponding missing indicator matrix $\bar{\bm{m}}^{\prime}$ can be used for training.

In the training process, we utilize a combination of 3 loss functions formulated as follows:

$L_{rec}$ corresponds to the reconstruction loss of the auto-encoder for learning meaningful latent representations, formulated as:

The missing indicator ${\bm{m}}_{v}$ is element-wisely multiplied so only the mean square errors of existing views are calculated. Note that during training, the network output is $\hat{\bm{x}}^{\prime}$ as the input is the augmented input $\bar{\bm{x}}^{\prime}$ (equation 2 and 3), so the network learns to reconstruct dropped out views with KNN hints and cross-view correlation and thus, learns to predict missing views’ information implicitly.

$L_{aug}$ is the embedding robustness loss, which encourages the learned representations to be consistent when augmentations are applied, promoting the robustness of the learned representations, formulated as:

Though our embedding robustness target is equivalent to minimizing the distance between ${\bm{z}}^{\prime}$ and ${\bm{z}}$ (equation 2), we design $L_{aug}$ based on cross-entropy loss within the training mini-batch. $B$ in the equation represents the training batch size. This design simultaneously minimizes the distance between ${\bm{z}}^{\prime}_{i}$ and ${\bm{z}}_{i}$ and maximizes the distances between ${\bm{z}}^{\prime}_{i}$ and embeddings of other samples ${\bm{z}}_{j},j\neq i$, preventing the embedding space from collapsing.

$L_{clu}$ is the DEC-based clustering loss (Xie et al., 2016) for the Clustering Module (Section 3.1.4), which optimizes embeddings for clustering with gradients from high-confidence samples.

It is formulated as the KL-divergence between distribution ${\bm{p}}^{\prime}$ and ${\bm{q}}^{\prime}$ computed from augmented input ${\bm{x}}^{\prime}$ during training.

Hyperparameters $\lambda_{1}$ and $\lambda_{2}$ control the balance between different loss components. By jointly minimizing the three loss terms, our network can learn representations that are both informative and clustering-friendly.

The training process is divided into two stages. In the first stage, the auto-encoder is pre-trained using $L_{rec}$ and $L_{aug}$, focusing on learning robust representations. Once the pre-training is complete, the Clustering Module is initialized. In the second stage, $L_{clu}$ is added for joint training to learn a clustering-friendly representation.

The training process is controlled by 2 hyperparameters: $E_{p}$, which represents the number of pre-training epochs, and $E_{j}$, which represents the number of joint training epochs. For a detailed description of the training process, please refer to Algorithm 2 in Appendix A.

In Table 3, we present the results of our ablation study on the main modules we designed. First of all, our Unified auto-encoder framework sets a solid baseline. Then, our designed robustness strategies, KNN Imputation (KNNI), and Data Augmentation (Aug) significantly improve clustering performance and have approximately equal contributions. The Clustering Module (CM) also plays a vital role in some datasets, by learning clustering-friendly representations. However, directly applying it can result in unstable performance, as the DEC-based training is sensitive to initialization. While our tailored components, i.e., CDPE&TAM help stabilize the learning. To summarize, the ablation study on modules reveals that the KNN Imputation, Augmentation, and Clustering Module are the three key components for improving clustering performance, while CDPE&TAM is essential for stability.

Figure 5f presents a T-SNE visualization of the embeddings during one training process. Initially in Figure 5a, the multi-view raw data are concatenated as embeddings, and the visualization appears to be disorganized. After 200 iterations of training, in Figure 5b, inherent structures start to be captured, and the pre-training peak accuracy (89.86) occurs at 1600 iterations 5c. At 2200 iterations 5d, the Clustering Module is initialized, and joint training with DEC-based clustering loss commences. The clusters become more compact after 200 iterations of joint training, as depicted in Figure 5e. Finally, at the end of the training, as shown in Figure 5f, the clusters become very compact, numerous samples initially incorrectly clustered with low confidence are now corrected, and the accuracy reaches 94.14.