URRL-IMVC: Unified and Robust Representation Learning for Incomplete Multi-View Clustering
Abstract.
Incomplete multi-view clustering (IMVC) aims to cluster multi-view data that are only partially available. This poses two main challenges: effectively leveraging multi-view information and mitigating the impact of missing views. Prevailing solutions employ cross-view contrastive learning and missing view recovery techniques. However, they either neglect valuable complementary information by focusing only on consensus between views or provide unreliable recovered views due to the absence of supervision. To address these limitations, we propose a novel Unified and Robust Representation Learning for Incomplete Multi-View Clustering (URRL-IMVC). URRL-IMVC directly learns a unified embedding that is robust to view missing conditions by integrating information from multiple views and neighboring samples. Firstly, to overcome the limitations of cross-view contrastive learning, URRL-IMVC incorporates an attention-based auto-encoder framework to fuse multi-view information and generate unified embeddings. Secondly, URRL-IMVC directly enhances the robustness of the unified embedding against view-missing conditions through KNN imputation and data augmentation techniques, eliminating the need for explicit missing view recovery. Finally, incremental improvements are introduced to further enhance the overall performance, such as the Clustering Module and the customization of the Encoder. We extensively evaluate the proposed URRL-IMVC framework on various benchmark datasets, demonstrating its state-of-the-art performance. Furthermore, comprehensive ablation studies are performed to validate the effectiveness of our design.
1. Introduction
Multi-view data (Fu et al., 2020) is commonly collected and utilized in various domains, making multi-view clustering (MVC) a crucial tool for analyzing such data and uncovering its underlying structures (Chao et al., 2021; Chen et al., 2022). Previous research has proposed several approaches (Xu et al., 2022, 2021) achieving promising performance by exploiting consensus or complementary information between views. However, in real-world applications, some views may be partially unavailable due to sensor malfunctions or other practical reasons. Existing MVC methods heavily rely on complete views to learn a comprehensive representation for clustering, making them inadequate under such conditions. To address this issue, Incomplete Multi-view Clustering (IMVC) methods have been introduced to reduce the impact of missing views (Wen et al., 2023). Various IMVC approaches have been proposed, including matrix decomposition (Li et al., 2014), kernel-based (Liu et al., 2017), and graph-based (Gao et al., 2016) methods. With the superior feature representation ability demonstrated by deep learning, some IMVC methods have integrated deep learning techniques, known as Deep Incomplete Multi-view Clustering (DIMVC) methods, which we will mainly discuss below. The key challenges in the IMVC task revolve around two problems: i) effectively utilizing multi-view information, and ii) mitigating the impact of missing views. Previous DIMVC works (Wang et al., 2018; Lin et al., 2021, 2023; Jin et al., 2023; Liu et al., 2023) have employed two mainstream strategies to address these problems: 1) cross-view contrastive learning, and 2) missing view recovery. However, these strategies have inherent drawbacks.
A general framework for cross-view contrastive learning is illustrated in Fig 1a, which originates from the MVC approaches. In this framework, Deep Neural Network (DNN) auto-encoders are employed to extract embeddings for each view. The embeddings are then aligned using a contrastive loss, aiming to minimize the distance between embeddings from the same sample across different views while simultaneously maximizing the distance with other samples (Jin et al., 2023; Lin et al., 2021; Yang et al., 2023). However, this framework primarily focuses on extracting consensus information in multi-view data, overlooking the valuable complementary information present. Additionally, the efficiency of the pair-wise contrastive strategy suffers as the number of views increases, and the effectiveness of this strategy diminishes due to less overlapped information between views (See Table 5 for experimental analysis). Theoretical analysis by Trosten et al. (2023) supports these observations, highlighting that contrastive alignment can reduce the number of separable clusters in the representation space, with this effect worsening as the number of views increases.
![Refer to caption](extracted/5727039/Fig/Fig_1_1.png)
![Refer to caption](extracted/5727039/Fig/Fig_1_2.png)
![Refer to caption](extracted/5727039/Fig/Fig_1_3.png)
Our framework avoids the drawbacks of cross-view contrastive learning and missing view recovery, by directly learning a unified and robust embedding.
The missing view recovery framework, as depicted in 1b, is commonly adopted in IMVC approaches. Typically, a DNN is employed to recover the missing view, either in the data or latent space. Subsequently, MVC methods or another view fusion network are utilized for clustering based on the recovered views. However, the reliability of the recovered views is a concern since the recovery ability of DNNs relies on unsupervised training. Meanwhile in some instances, (Liu et al., 2023) for example, missing views are recovered by a fused embedding in the first stage, and subsequently used to generate another fused embedding for clustering in the second stage, introducing unnecessary complexity and inefficiency to the pipeline. We propose that a well-designed recovery-free method can achieve comparable performance to recovery-based methods while offering the advantages of simplicity and reduced computational overhead.
To address the aforementioned challenges, we propose a Unified and Robust Representation Learning framework for Incomplete Multi-View Clustering (URRL-IMVC). Our framework, depicted in Fig 1c, is designed to be cross-view contrastive learning-free and missing view recovery-free. First, to overcome the limitations of cross-view contrastive learning, we propose a new framework that fuses multi-view information into a unified embedding instead of contrasting each view’s information. We achieve this by designing an attention-based auto-encoder network, which captures both consensus and complementary information and intelligently fuses them. Moreover, it is naturally scalable to different numbers of views. Second, to tackle the issue of missing views, we aim to directly enhance the robustness of the unified embedding against view-missing conditions without explicitly recovering the missing views. We introduce two strategies to achieve this robustness. 1) We treat view missing as a form of noise and draw inspiration from successful applications of denoising and masked auto-encoders (Vincent et al., 2008; He et al., 2022). Our proposed approach randomly drops out existing views as a form of data augmentation to simulate the view missing condition. By reconstructing denoised input data from the unified embedding and imposing constraints between the augmented and un-augmented embeddings, we enhance the robustness of the unified representation. 2) As the old saying goes, “One cannot make bricks without straw”, it is hard to learn to reconstruct a dropped-out view directly. We introduce k-nearest neighbors (KNN) as additional inputs, with a cross-view imputation strategy to fill in the missing or dropped-out views, providing valuable hints for reconstruction. We want to highlight that while previous methods have focused on either fusing multi-view information (Wang et al., 2021; Lin et al., 2022) or incorporating neighborhood information (Nguyen et al., 2021; Wang et al., 2019; Yang et al., 2020; Tu et al., 2021) for clustering, our approach represents one of the initial endeavors to fuse both aspects. Finally, we conduct experiments based on this framework and make incremental improvements to enhance clustering performance and stability. Some of the key enhancements include the customization of the Transformer-based Encoder to filter out noise and emphasize critical information, and the introduction of the Clustering Module to learn clustering-friendly representations.
To summarize, our main contributions are:
-
•
Unified: We propose a unified representation learning framework that efficiently fuses both multi-view and neighborhood information, allowing for better capturing of consensus and complementary information while avoiding the limitations of cross-view contrastive learning.
-
•
Robust: We proposed novel strategies, including KNN imputation and data augmentation, to directly learn a robust representation capable of handling view-missing conditions without explicit missing view recovery.
-
•
Improvements: Multiple incremental improvements are introduced for better clustering performance and stability, including the extra Clustering Module and the customization of the Transformer-based Encoder.
-
•
Experiments: Through comprehensive experiments on diverse benchmark datasets, we demonstrate the state-of-the-art performance of our unified representation learning framework. Thorough ablation studies are also conducted to provide valuable insights for future research in this field.
2. Related works
Deep neural networks (DNNs) have shown good performance in learning feature representation, which is beneficial for the IMVC task. Various IMVC approaches have integrated DNNs into their framework, denoted as DIMVC approaches. In terms of network architecture, DIMVC approaches can be divided into four categories. (1) Auto-encoder-based approaches (Lin et al., 2022; Jin et al., 2023; Lin et al., 2021, 2023). These approaches utilize auto-encoders to extract high-level features of each view, which are usually combined with contrastive learning or cross-view prediction to handle the incompleteness problem. (2) Generative network-based approaches. For the IMVC task, an intuitive solution is to complete the missing views with generative models, transforming it into an MVC task. Adversarial learning (Goodfellow et al., 2014) is commonly adopted by generative IMVC approaches including AIMVC Xu et al. (2019), PMVC-CG Wang et al. (2018), and GP-MVC Wang et al. (2021) to improve data distribution learning in the context of IMVC. (3) Graph Neural Network-based (GNN-based) approaches (Wang et al., 2022, 2018). These approaches aim to learn consensus representations from the structure information contained in the graphs constructed for each view. (4) Transformer (Vaswani et al., 2017) or attention-based approaches. The Transformer network has gained attention in recent years due to its successful application in various domains. Its architecture, along with its Multi-head Attention mechanism, has been particularly effective in capturing complex relationships. In the field of DIMVC, RecFormer (Liu et al., 2023) proposed a Transformer auto-encoder with a mask to recover missing views, while MCAC (Zhang and Zhu, 2023) and IMVC-PBI (Li et al., 2023) incorporated attention mechanisms into their frameworks. In this paper, we leverage an auto-encoder architecture based on the Transformer framework to address the challenges of the IMVC task.
3. The Proposed Method
Notations. An incomplete multi-view dataset with samples and views is denoted as , where denotes the dimension of -th view. The view missing condition can be described by a binary missing indicator matrix , where indicates the -th view of the -th sample is missing and just the opposite. An extra restriction is imposed: , ensuring that at least one view is available for each sample, which is essential for the clustering task.
3.1. Framework
![Refer to caption](extracted/5727039/Fig/Fig_2.png)
The architecture can be better understood together with the equations below.
Unlike many prior approaches in the field of MVC that employ view-specific auto-encoders for each view, we propose a novel framework using a unified auto-encoder that effectively fuses multi-view data. The network architecture, depicted in Fig 2, consists of three key modules: the Encoder , the Decoder , and the Clustering Module . To provide a formal description, the framework operates as follows. Given an incomplete multi-view data sample from dataset with its missing indicator vector , we apply KNN Imputation and Data Augmentation (KIDA), as described in section 3.2, to obtain the input for the auto-encoder network,
(1) |
where , is the data and mask after KNN Imputation, while , is the augmented version of , , and is the hyperparameter in KNN. Note that though KNN Imputation is widely applied in prior IMVC works, it is mainly used for recovering missing views, which is different from our usage as a pre-process. Next, these inputs are fed into the Encoder network to obtain the augmented and un-augmented embeddings, denoted as and respectively,
(2) |
where represents the Encoder’s parameters and is the dimension of the embedding. Then, the Decoder maps the augmented embedding back to the data space to reconstruct the data sample,
(3) |
where represents the parameters of the Decoder. Simultaneously, clustering is performed using the un-augmented embedding,
(4) |
where is the clustering result, and represents the probabilities of the data sample belonging to cluster centers. During training, the loss function defined in equation 19 is computed to optimize parameters ; During testing, is regarded as the final clustering result.
In the following sections, we will introduce the Encoder module, including its two submodules: the Neighbor Dimensional Encoder (NDE) and the View Dimensional Encoder (VDE), the Decoder module, and the Clustering Module respectively.
3.1.1. Neighbor Dimensional Encoder
KNN Imputation (Section 3.2) provides additional information for missing views, but the retrieved nearest neighbors may contain noise and be unreliable. To address this issue, we propose the Neighbor Dimensional Encoder (NDE), which is a series of customized Transformer Encoders (Vaswani et al., 2017), with each one dedicated to a view to fuse its KNN input and filter out noise, formulated as:
(5) |
(6) |
The th Transformer Encoder corresponding to the th view is represented with in equation 6, and is its parameters. The input KNN sequence from the th view is first processed to add the Cosine Distance-based Positional Encoding (CDPE), then it is passed through the Transformer Encoder with the KNN mask . Finally, only the first vector from the output sequence is chosen as the output, denoted as .
Below we introduce the two key customizations of the Transformer Encoders in NDE: the CDPE and the output choice.
Cosine Distance-based Positional Encoding (CDPE)
The order or distance of the KNN instances contains vital information regarding the reliability of the inputs, with farther neighbors noisier and less reliable. To capture this information for the permutation invariant Transformer structure, we introduce Positional Encoding (PE) to provide this extra KNN order information. We explored various positional encoding (PE) designs considering the data sources and their combination with data. Among these configurations, concatenating cosine distance-based (inspired by Nguyen et al. (2021)) or learnable PE with the input yielded the best results. For better interpretability, Cosine Distance-based Positional Encoding (CDPE) is chosen as our final design. The CDPE can be explained as,
(7) |
in which is the function calculating the pair-wise distance of vectors and return a distance matrix, and stands for matrix concatenation. Given two input vectors and , the pair-wise cosine distance is formulated as,
(8) |
We conjecture that CDPE is the most suitable for two reasons. First, it contains the KNN distance information rather than simply providing order information. Second, its value range is 0-2, which is more stable than other distance functions, e.g., Euclidean distance.
Output choice
(Figure 3a) Generally, for fusing information with a Transformer Encoder, an additional token like [CLS] can be added (Devlin et al., 2019). However, in our unsupervised task, adding such a meaningless token can introduce noise and lead to performance degradation. Instead, we adopt the first vector of the output sequence. This design not only avoids extra noise but also introduces a bias on the first input. The first input is always the most reliable sample, i.e., the center sample for an existing view or the nearest neighbor for a missing view. By introducing this bias, important information is emphasized while fusing KNN information.
3.1.2. View Dimensional Encoder
The View Dimensional Encoder (VDE) is designed to fuse view representations and obtain unified embedding. As depicted in Figure 2, it consists of two parts, with firstly a Feed-Forward Network (FFN) to map the representations of different dimensions to the same latent space, and then followed by a Transformer Encoder for fusion. The FFN consists of three fully connected (FC) layers, without normalization or dropout layers, which can be detrimental to the stability of training. The FFN of VDE can be formulated as:
(9) |
(10) |
In the equation, represents the concatenate operation, is the activation function, and are the weight matrix and bias vector of the FC layer respectively.
The Transformer Encoder part of the VDE can be explained as,
(11) |
in which is the VDE Transformer Encoder structure. The view representations are passed through the Transformer Encoder along with the generated Three-level Adaptive Mask (TAM) . The output sequence is averaged for fusion. The Transformer Encoder in VDE is also customized but different from that in NDE. The key difference lies in that views are permutation invariant, i.e., changing the order of views should yield the same output, while KNN has an order. Based on this difference, the two key customizations of the VDE Transformer are designed as:
Three-level Adaptive Masking (TAM)
We employ a masking mechanism in VDE to emphasize the reliability of the inputs, instead of the positional encoding used in NDE to maintain permutation invariant. The self-attention in Transformer is formulated as,
(12) |
where is the mask applied, with negative infinity for masking and 0 for not. The input view representations can be roughly divided into 3 categories based on data completeness: (1) complete, (2) missing view with KNN imputation, and (3) missing view without imputation. Therefore, instead of the original binary mask, we design a Three-level Adaptive Mask (TAM) for the 3 categories, the mask values range from completely unmasked (1) to fully masked (3), with an intermediate level (2) in between, formulated as,
(13) |
in which and are the original and KNN imputation generated missing matrix respectively. is a negative hyperparameter to control the emphasizing intensity.
Output choice
(Figure 3b) To ensure permutation invariance and avoid bias towards any views, the embedding is generated by averaging all Transformer output vectors. The output choices of NDE and VDE are depicted in Figure 3b for intuitive understanding.
![Refer to caption](extracted/5727039/Fig/Fig_7_1.png)
![Refer to caption](extracted/5727039/Fig/Fig_7_2.png)
The effectiveness of this design is proved by ablation studies in the Appendix.
3.1.3. Decoder
The Decoder in our model is designed as a compact 4-layer FFN, to reconstruct the input from the unified embedding. Similar to the FFN in VDE, we removed its normalization and dropout layer for better stability. Through our experiments, we have observed that a deep and complex Decoder does not necessarily improve the clustering performance and may even have negative effects in certain cases. One possible explanation for this phenomenon is that a shallow and simple Decoder serves as a regularization technique on the embedding space, and prevents it from collapsing. This regularization effect is similar to the Locality-preserving Constraint proposed by Huang et al. (2014), which helps preserve the local structure of the data. The process of Decoder is formulated as,
(14) |
(15) |
3.1.4. Clustering Module
The auto-encoder we have designed extracts robust representations and captures the inherent structures of data. However, these inherent structures may not necessarily follow a cluster-oriented distribution. To enhance the clustering performance, we introduce a Clustering Module inspired by DEC (Xie et al., 2016). Below we describe its procedures. First, after auto-encoder pretraining, a traditional clustering method is adopted to initialize the cluster centers from embedding . Then, during each iteration of joint training, a similarity matrix is generated between and embedding with Student’s t-distribution, in which represents the possibility sample belongs to cluster , and defined in equation 4 is a row of . With the similarity matrix , we calculate the target distribution as,
(16) |
in which is the soft cluster size . The training target is the KL-Divergence between and (equation 22) which is defined in Section 3.3.1. Finally, after joint training, the clustering result can be obtained by finding the maximum possibility in each row of . The detailed training procedures are described in Section 3.3.2.
3.2. Data Augmentation and Imputation
In this section, we introduce Data Augmentation and KNN Imputation ( in equation 1), which are two key data processing strategies of our approach for handling incomplete multi-view data.
3.2.1. KNN Imputation
The KNN search is conducted separately in each view, taking into account the incomplete condition. For existing views, the KNN is directly obtained, while for missing views, existing views from the same sample are used for searching KNN. Specifically, for a missing view of sample , we first find all existing views of the same sample. Then we iterate through the KNN samples of these existing views to check if exists. If it does, we appended it to the KNN list of the missing view. Finally, we select the top samples from the KNN list as the imputation for the missing view. If the length of the KNN list is less than , then the remaining positions are filled with zeros. Along with the KNN imputation , a missing indicator matrix is generated, where 1 represents a position filled with a KNN sample and 0 represents a position filled with zeros. The detailed procedures are listed in Algorithm 1, Appendix A.
3.2.2. Data Augmentation
Our framework is inspired by the denoising auto-encoder, which helps to learn robust representations by introducing noise during training. Three types of noise are designed, including Gaussian noise, random dropout, and view dropout. Gaussian noise helps prevent overfitting by introducing variability in the input data. Random dropout, functional as a regularization technique, encourages the model to learn more robust features by forcing it to rely on different subsets of the input data. View dropout is a noise specifically designed for the IMVC task. It randomly drops out one or more views from the input data during training, encouraging the model to learn representations that are more robust to view missing conditions.
Below are the formulations of the three kinds of augmentation we used. By combining equation 17 and 18, the KIDA operation in equation 1 is obtained.
(17) |
(18) |
The view dropout augmentation is shown in equation 17, where random views are dropped out with a possibility of , before the KNN Imputation (Algorithm 1) step. The dropout mask is applied with element-wise multiplication, denoted by . The masked views are regarded as missing views in both the KNN Imputation and the auto-encoder network. After the KNN Imputation, in equation 18, Gaussian noise is added to the input data to introduce variability, whose intensity is controlled by hyperparameter . After that, random values in the input are set to zero with a probability of , which is the random dropout augmentation. Finally, the augmented input along with the corresponding missing indicator matrix can be used for training.
3.3. Training Strategy and Loss Function
3.3.1. Loss Function
In the training process, we utilize a combination of 3 loss functions formulated as follows:
(19) |
corresponds to the reconstruction loss of the auto-encoder for learning meaningful latent representations, formulated as:
(20) |
The missing indicator is element-wisely multiplied so only the mean square errors of existing views are calculated. Note that during training, the network output is as the input is the augmented input (equation 2 and 3), so the network learns to reconstruct dropped out views with KNN hints and cross-view correlation and thus, learns to predict missing views’ information implicitly.
is the embedding robustness loss, which encourages the learned representations to be consistent when augmentations are applied, promoting the robustness of the learned representations, formulated as:
(21) |
Though our embedding robustness target is equivalent to minimizing the distance between and (equation 2), we design based on cross-entropy loss within the training mini-batch. in the equation represents the training batch size. This design simultaneously minimizes the distance between and and maximizes the distances between and embeddings of other samples , preventing the embedding space from collapsing.
is the DEC-based clustering loss (Xie et al., 2016) for the Clustering Module (Section 3.1.4), which optimizes embeddings for clustering with gradients from high-confidence samples.
(22) |
It is formulated as the KL-divergence between distribution and computed from augmented input during training.
Hyperparameters and control the balance between different loss components. By jointly minimizing the three loss terms, our network can learn representations that are both informative and clustering-friendly.
3.3.2. Training Strategy
The training process is divided into two stages. In the first stage, the auto-encoder is pre-trained using and , focusing on learning robust representations. Once the pre-training is complete, the Clustering Module is initialized. In the second stage, is added for joint training to learn a clustering-friendly representation.
4. Experiments
Please refer to Appendix B.1 for the hyperparameter settings and design details in our experiments.
Name | Ansichten | Clusters | Samples | Dimensions | Typ |
---|---|---|---|---|---|
Handwritten (Duin, 2023) | 6 | 10 | 2000 | 240/76/216/47/64/6 | Image |
Caltech101-7 (Cai et al., 2013) | 5 | 7 | 1400 | 40/254/1984/512/928 | Image |
ALOI_Deep (Liu et al., 2023) | 3 | 100 | 10800 | 2048/4096/2048 | Image |
Scene15 (Fei-Fei and Perona, 2005; Cai et al., 2013) | 2 | 15 | 4485 | 20/59 | Image |
BDGP (Cai et al., 2012; Tang and Liu, 2022) | 2 | 5 | 2500 | 1750/79 | Image/Text |
Reuters (Amini et al., 2009; Yang et al., 2023) | 2 | 6 | 18758 | 10/10 | Text |
Datasets | Handwritten | Caltech101-7 | ALOI_Deep | ||||||
---|---|---|---|---|---|---|---|---|---|
Metrics | Acc(%) | NMI(%) | ARI(%) | Acc(%) | NMI(%) | ARI(%) | Acc(%) | NMI(%) | ARI(%) |
Completer (Lin et al., 2021) | |||||||||
DSIMVC (Tang and Liu, 2022) | |||||||||
SURE (Yang et al., 2023) | |||||||||
DCP (Lin et al., 2023) | |||||||||
CPSPAN (Jin et al., 2023) | |||||||||
RecFormer (Liu et al., 2023) | 82.67 0.99 | ||||||||
URRL-IMVC (ours) | 93.86 2.51 | 89.90 1.38 | 88.93 2.58 | 93.19 1.54 | 86.94 1.46 | 86.45 1.98 | 91.48 2.21 | 97.50 1.04 | 90.91 2.70 |
Datasets | Scene15 | BDGP | Reuters | ||||||
Metrics | Acc(%) | NMI(%) | ARI(%) | Acc(%) | NMI(%) | ARI(%) | Acc(%) | NMI(%) | ARI(%) |
Completer (Lin et al., 2021) | 42.09 1.54 | ||||||||
DSIMVC (Tang and Liu, 2022) | 17.48 0.67 | 94.63 1.53 | 85.62 2.29 | 87.53 2.74 | 33.27 2.10 | 23.69 2.20 | |||
SURE (Yang et al., 2023) | 46.68 3.63 | ||||||||
DCP (Lin et al., 2023) | 38.28 1.63 | 41.69 1.23 | 22.22 1.70 | ||||||
CPSPAN (Jin et al., 2023) | 41.38 2.04 | ||||||||
RecFormer (Liu et al., 2023) | |||||||||
URRL-IMVC (ours) | 41.18 1.77 | 41.87 0.95 | 24.09 1.09 | 89.15 5.07 | 77.14 6.25 | 77.64 8.03 | 48.63 2.64 | 28.94 1.65 | 24.78 2.59 |
4.1. Datasets and Metrics
Experiments were performed on 6 multi-view datasets varying in number of views and modal to validate the effectiveness of our method. The dataset characteristics are summarized in Table 1. We report the widely used metrics Clustering Accuracy (Acc), Normalized Mutual Information (NMI), and Adjusted Rand Index(ARI) as results. We run each experiment 10 times and report the average value and standard deviation (after ). Details about our experiment and view-missing settings can be found in Appendix B.2.
4.2. Comparison with State-of-the-arts
We compare our approach with several state-of-the-art DIMVC approaches listed in Table 2. Other comparisons about different numbers of views, traditional IMVC methods, model parameters, and computational costs can be found in Appendix B.3.
Comparison on different datasets
URRL-IMVC achieved state-of-the-art performance on the 6 benchmark datasets, surpassing most existing approaches, as indicated in Table 2. Our approach consistently outperformed other SOTA methods across all evaluation metrics, except for the BDGP dataset and NMI on Scene15 and Reuters, where our approach is sub-optimal. This excellent clustering performance and stability can be attributed to our representation learning framework, which effectively captures the underlying data structure while remaining robust in the presence of missing views. Additionally, URRL-IMVC exhibited stability compared to other SOTA methods, with a relatively lower standard deviation across 10 experiments, thanks to the tailored components in the network to filter out noise. Notably, our approach excelled on datasets with more views, such as Handwritten and Caltech101-7. Together with experimental results in Table 5, it showed that our framework successfully overcomes the drawbacks of cross-view contrastive learning.
![Refer to caption](extracted/5727039/Fig/Fig_5_1.png)
![Refer to caption](extracted/5727039/Fig/Fig_5_2.png)
![Refer to caption](extracted/5727039/Fig/Fig_5_3.png)
Our approach consistently outperforms other approaches under different missing rates.
Comparison with different missing rates
As depicted in Figure 4c, URRL-IMVC consistently outperformed other approaches, establishing an upper bound for clustering performance regardless of the missing rate (). Our approach displayed better stability compared to other methods, with a gradual decrease in accuracy as the missing rate increased. In contrast, other approaches exhibited more fluctuation, rendering their results less predictable. Notably, DCP and Completer experienced a significant decline in performance when the missing rate reached 0.75, as they only trained their cross-view contrastive and recovery networks using complete samples. Insufficient training samples led to unsatisfactory recovery outcomes and fragile representations for clustering. In contrast, our approach focused on the robustness of the unified representation, allowing us to circumvent these limitations and achieve stable and high performance across varying missing rates.
4.3. Ablation Studies
Unless otherwise specified, the experiments were conducted on the Caltech101-7 dataset with the missing rate . In certain experiments, the Clustering Module was disabled to provide clearer observations of specific phenomena. Additional ablation studies regarding detailed designs and hyperparameters (e.g., output choice, in KNN Imputation, view dropout probability) can be found in Appendix B.4.
4.3.1. Ablation on Modules
KNNI | Aug | CDPE&TAM | CM | Acc(%) | NMI(%) | ARI(%) |
64.79 2.18 | 56.52 3.99 | |||||
✓ | 72.76 2.21 | 68.60 4.48 | ||||
✓ | 83.96 3.11 | 74.39 2.24 | 70.41 3.63 | |||
✓ | 85.60 6.52 | 78.10 4.46 | 75.29 7.14 | |||
✓ | ✓ | 74.92 2.54 | 71.76 3.69 | |||
✓ | ✓ | 82.05 3.71 | 70.74 3.44 | 63.92 6.82 | ||
✓ | ✓ | 89.95 0.74 | 80.01 0.96 | 79.55 1.19 | ||
✓ | ✓ | 89.39 3.12 | 81.74 1.81 | 80.60 2.76 | ||
✓ | ✓ | ✓ | 75.25 2.02 | 71.69 2.44 | ||
✓ | ✓ | ✓ | 90.22 5.20 | 83.21 3.89 | 82.15 5.67 | |
✓ | ✓ | ✓ | 91.90 2.99 | 84.29 2.06 | 83.76 3.96 | |
✓ | ✓ | ✓ | ✓ | 93.35 0.37 | 86.50 0.61 | 86.25 0.64 |
In Table 3, we present the results of our ablation study on the main modules we designed. First of all, our Unified auto-encoder framework sets a solid baseline. Then, our designed robustness strategies, KNN Imputation (KNNI), and Data Augmentation (Aug) significantly improve clustering performance and have approximately equal contributions. The Clustering Module (CM) also plays a vital role in some datasets, by learning clustering-friendly representations. However, directly applying it can result in unstable performance, as the DEC-based training is sensitive to initialization. While our tailored components, i.e., CDPE&TAM help stabilize the learning. To summarize, the ablation study on modules reveals that the KNN Imputation, Augmentation, and Clustering Module are the three key components for improving clustering performance, while CDPE&TAM is essential for stability.
4.3.2. Visualization
![Refer to caption](x1.png)
![Refer to caption](x2.png)
![Refer to caption](x3.png)
![Refer to caption](x4.png)
![Refer to caption](x5.png)
![Refer to caption](x6.png)
The first half of iterations capture the inherent data structure and provide a good initialization for the Clustering Module.
Figure 5f presents a T-SNE visualization of the embeddings during one training process. Initially in Figure 5a, the multi-view raw data are concatenated as embeddings, and the visualization appears to be disorganized. After 200 iterations of training, in Figure 5b, inherent structures start to be captured, and the pre-training peak accuracy (89.86) occurs at 1600 iterations 5c. At 2200 iterations 5d, the Clustering Module is initialized, and joint training with DEC-based clustering loss commences. The clusters become more compact after 200 iterations of joint training, as depicted in Figure 5e. Finally, at the end of the training, as shown in Figure 5f, the clusters become very compact, numerous samples initially incorrectly clustered with low confidence are now corrected, and the accuracy reaches 94.14.
5. Conclusion
In this paper, we proposed URRL-IMVC, a novel unified and robust representation learning framework for the incomplete multi-view clustering task. By leveraging an attention-based auto-encoder framework, we successfully fuse the multi-view information into a unified embedding, offering a more comprehensive solution compared to potentially limiting cross-view contrastive learning. Through the utilization of KNN imputation and data augmentation strategies, we directly acquire robust embeddings that effectively handle the view-missing condition, eliminating the need for explicit missing view recovery and its associated computation and unreliability. Furthermore, incremental improvements, such as the Clustering Module and customization of the Encoder, enhance the clustering stability and performance, achieving state-of-the-art results. This improved robust and unified representation learning framework acts as a powerful tool for addressing the challenges of IMVC and provides valuable insights for future research in this domain.
Acknowledgements.
This work was supported by the Zhejiang Provincial Natural Science Foundation of China under Grant No. LDT23F01013F01, and by the Fundamental Research Funds for the Central Universities.References
- (1)
- Amini et al. (2009) Massih R. Amini, Nicolas Usunier, and Cyril Goutte. 2009. Learning from Multiple Partially Observed Views - an Application to Multilingual Text Categorization. In Advances in Neural Information Processing Systems, Y. Bengio, D. Schuurmans, J. Lafferty, C. Williams, and A. Culotta (Eds.), Vol. 22. Curran Associates, Inc.
- Cai et al. (2013) Xiao Cai, Feiping Nie, and Heng Huang. 2013. Multi-view k-means clustering on big data. In Twenty-Third International Joint conference on artificial intelligence.
- Cai et al. (2012) Xiao Cai, Hua Wang, Heng Huang, and Chris Ding. 2012. Joint stage recognition and anatomical annotation of drosophila gene expression patterns. Bioinformatics 28, 12 (2012), i16–i24.
- Chao et al. (2021) Guoqing Chao, Shiliang Sun, and Jinbo Bi. 2021. A Survey on Multiview Clustering. IEEE Transactions on Artificial Intelligence 2, 2 (2021), 146–168. https://doi.org/10.1109/TAI.2021.3065894
- Chen et al. (2022) Man-Sheng Chen, Jia-Qi Lin, Xiang-Long Li, Bao-Yu Liu, Chang-Dong Wang, Dong Huang, and Jian-Huang Lai. 2022. Representation learning in multi-view clustering: A literature review. Data Science and Engineering 7, 3 (2022), 225–241.
- Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 [cs.CL]
- Duin (2023) Robert Duin. 2023. Multiple Features. UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C5HC70.
- Fei-Fei and Perona (2005) L. Fei-Fei and P. Perona. 2005. A Bayesian hierarchical model for learning natural scene categories. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), Vol. 2. 524–531 vol. 2. https://doi.org/10.1109/CVPR.2005.16
- Fu et al. (2020) Lele Fu, Pengfei Lin, Athanasios V Vasilakos, and Shiping Wang. 2020. An overview of recent multi-view clustering. Neurocomputing 402 (2020), 148–161.
- Gao et al. (2016) Hang Gao, Yuxing Peng, and Songlei Jian. 2016. Incomplete Multi-view Clustering. In Intelligent Information Processing VIII, Zhongzhi Shi, Sunil Vadera, and Gang Li (Eds.). Springer International Publishing, Cham, 245–255.
- Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In Advances in Neural Information Processing Systems, Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (Eds.), Vol. 27. Curran Associates, Inc.
- He et al. (2022) Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2022. Masked Autoencoders Are Scalable Vision Learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 16000–16009.
- He et al. (2015) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).
- Huang et al. (2014) Peihao Huang, Yan Huang, Wei Wang, and Liang Wang. 2014. Deep Embedding Network for Clustering. In 2014 22nd International Conference on Pattern Recognition. 1532–1537. https://doi.org/10.1109/ICPR.2014.272
- Jin et al. (2023) Jiaqi Jin, Siwei Wang, Zhibin Dong, Xinwang Liu, and En Zhu. 2023. Deep Incomplete Multi-View Clustering With Cross-View Partial Sample and Prototype Alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 11600–11609.
- Li et al. (2023) Haobin Li, Yunfan Li, Mouxing Yang, Peng Hu, Dezhong Peng, and Xi Peng. 2023. Incomplete Multi-view Clustering via Prototype-based Imputation. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, Edith Elkind (Ed.). International Joint Conferences on Artificial Intelligence Organization, 3911–3919. https://doi.org/10.24963/ijcai.2023/435 Main Track.
- Li et al. (2014) Shao-Yuan Li, Yuan Jiang, and Zhi-Hua Zhou. 2014. Partial Multi-View Clustering. Proceedings of the AAAI Conference on Artificial Intelligence 28, 1 (Jun. 2014). https://doi.org/10.1609/aaai.v28i1.8973
- Lin et al. (2022) Fangfei Lin, Bing Bai, Kun Bai, Yazhou Ren, Peng Zhao, and Zenglin Xu. 2022. Contrastive multi-view hyperbolic hierarchical clustering. arXiv preprint arXiv:2205.02618 (2022).
- Lin et al. (2023) Yijie Lin, Yuanbiao Gou, Xiaotian Liu, Jinfeng Bai, Jiancheng Lv, and Xi Peng. 2023. Dual Contrastive Prediction for Incomplete Multi-View Representation Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 4 (2023), 4447–4461. https://doi.org/10.1109/TPAMI.2022.3197238
- Lin et al. (2021) Yijie Lin, Yuanbiao Gou, Zitao Liu, Boyun Li, Jiancheng Lv, and Xi Peng. 2021. COMPLETER: Incomplete Multi-View Clustering via Contrastive Prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 11174–11183.
- Liu et al. (2023) Chengliang Liu, Jie Wen, Zhihao Wu, Xiaoling Luo, Chao Huang, and Yong Xu. 2023. Information Recovery-Driven Deep Incomplete Multiview Clustering Network. IEEE Transactions on Neural Networks and Learning Systems (2023), 1–11. https://doi.org/10.1109/TNNLS.2023.3286918
- Liu et al. (2017) Xinwang Liu, Miaomiao Li, Lei Wang, Yong Dou, Jianping Yin, and En Zhu. 2017. Multiple Kernel k-Means with Incomplete Kernels. Proceedings of the AAAI Conference on Artificial Intelligence 31, 1 (Feb. 2017). https://doi.org/10.1609/aaai.v31i1.10893
- Nguyen et al. (2021) Xuan-Bac Nguyen, Duc Toan Bui, Chi Nhan Duong, Tien D. Bui, and Khoa Luu. 2021. Clusformer: A Transformer Based Clustering Approach to Unsupervised Large-Scale Face and Visual Landmark Recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 10847–10856.
- Tang and Liu (2022) Huayi Tang and Yong Liu. 2022. Deep Safe Incomplete Multi-view Clustering: Theorem and Algorithm. In Proceedings of the 39th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 162), Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (Eds.). PMLR, 21090–21110.
- Trosten et al. (2023) Daniel J. Trosten, Sigurd Løkse, Robert Jenssen, and Michael C. Kampffmeyer. 2023. On the Effects of Self-Supervision and Contrastive Alignment in Deep Multi-View Clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 23976–23985.
- Tu et al. (2021) Wenxuan Tu, Sihang Zhou, Xinwang Liu, Xifeng Guo, Zhiping Cai, En Zhu, and Jieren Cheng. 2021. Deep Fusion Clustering Network. Proceedings of the AAAI Conference on Artificial Intelligence 35, 11 (May 2021), 9978–9987. https://doi.org/10.1609/aaai.v35i11.17198
- Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems, I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc.
- Vincent et al. (2008) Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. 2008. Extracting and Composing Robust Features with Denoising Autoencoders. In Proceedings of the 25th International Conference on Machine Learning (Helsinki, Finland) (ICML ’08). Association for Computing Machinery, New York, NY, USA, 1096–1103. https://doi.org/10.1145/1390156.1390294
- Wang et al. (2018) Qianqian Wang, Zhengming Ding, Zhiqiang Tao, Quanxue Gao, and Yun Fu. 2018. Partial Multi-view Clustering via Consistent GAN. In 2018 IEEE International Conference on Data Mining (ICDM). 1290–1295. https://doi.org/10.1109/ICDM.2018.00174
- Wang et al. (2021) Qianqian Wang, Zhengming Ding, Zhiqiang Tao, Quanxue Gao, and Yun Fu. 2021. Generative Partial Multi-View Clustering With Adaptive Fusion and Cycle Consistency. IEEE Transactions on Image Processing 30 (2021), 1771–1783. https://doi.org/10.1109/TIP.2020.3048626
- Wang et al. (2022) Yiming Wang, Dongxia Chang, Zhiqiang Fu, Jie Wen, and Yao Zhao. 2022. Graph Contrastive Partial Multi-View Clustering. IEEE Transactions on Multimedia (2022), 1–12. https://doi.org/10.1109/TMM.2022.3210376
- Wang et al. (2019) Zhongdao Wang, Liang Zheng, Yali Li, and Shengjin Wang. 2019. Linkage Based Face Clustering via Graph Convolution Network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
- Wen et al. (2023) Jie Wen, Zheng Zhang, Lunke Fei, Bob Zhang, Yong Xu, Zhao Zhang, and Jinxing Li. 2023. A Survey on Incomplete Multiview Clustering. IEEE Transactions on Systems, Man, and Cybernetics: Systems 53, 2 (2023), 1136–1149. https://doi.org/10.1109/TSMC.2022.3192635
- Xie et al. (2016) Junyuan Xie, Ross Girshick, and Ali Farhadi. 2016. Unsupervised Deep Embedding for Clustering Analysis. In Proceedings of The 33rd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 48), Maria Florina Balcan and Kilian Q. Weinberger (Eds.). PMLR, New York, New York, USA, 478–487.
- Xu et al. (2019) Cai Xu, Ziyu Guan, Wei Zhao, Hongchang Wu, Yunfei Niu, and Beilei Ling. 2019. Adversarial Incomplete Multi-view Clustering. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19. International Joint Conferences on Artificial Intelligence Organization, 3933–3939. https://doi.org/10.24963/ijcai.2019/546
- Xu et al. (2021) Jie Xu, Yazhou Ren, Huayi Tang, Xiaorong Pu, Xiaofeng Zhu, Ming Zeng, and Lifang He. 2021. Multi-VAE: Learning Disentangled View-Common and View-Peculiar Visual Representations for Multi-View Clustering. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 9234–9243.
- Xu et al. (2022) Jie Xu, Huayi Tang, Yazhou Ren, Liang Peng, Xiaofeng Zhu, and Lifang He. 2022. Multi-Level Feature Learning for Contrastive Multi-View Clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 16051–16060.
- Yang et al. (2020) Lei Yang, Dapeng Chen, Xiaohang Zhan, Rui Zhao, Chen Change Loy, and Dahua Lin. 2020. Learning to Cluster Faces via Confidence and Connectivity Estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
- Yang et al. (2023) Mouxing Yang, Yunfan Li, Peng Hu, Jinfeng Bai, Jiancheng Lv, and Xi Peng. 2023. Robust Multi-View Clustering With Incomplete Information. IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 1 (2023), 1055–1069. https://doi.org/10.1109/TPAMI.2022.3155499
- Zhang and Zhu (2023) Yanhao Zhang and Changming Zhu. 2023. Incomplete multi-view clustering via attention-based contrast learning. International Journal of Machine Learning and Cybernetics (2023), 1–17.
Appendix A Method Appendix
The detailed KNN Imputation algorithm is described in Algorithm 1, and the training procedure is described in Algorithm 2.
Input: Target , which is the th view of the th sample, dataset , missing indicator matrix , hyperparameter .
Output: KNN Imputation
Input: Dataset , missing indicator matrix , hyperparameters.
Output: Model parameters , , , final clustering result
Appendix B Experiments Appendix
B.1. Implementation Details
We set most of the hyperparameters empirically with grid search, and the same setting is used for all experiments if not specifically mentioned. in equation 13 is set to -10. The hyperparameter in KNN Imputation is set to 4. and , the data augmentation hyperparameter in equation 18, are fixed at 0.05, while in equation 17 which controls view dropout possibility is set to be growing with the actual missing rate of the dataset, defined as,
(23) |
in which we set . The loss weight hyperparameters and are set to 0.001 and 0.1 respectively. The embedding dimension is set to 256. Batch size is set to 64 for both training and testing and the learning rate is fixed at 3e-4 throughout training. A small weight decay of 4e-5 is used for less over-fitting. The training epoch parameter (Section 3.3.2) is set to 100, 100, 15, and 50 respectively for the four datasets in Table 1 to maintain roughly the same training iteration. As for , we found that training with DEC-based loss on some datasets (ALOI_Deep and Scene15 in this paper) can diverge, possibly due to imbalanced cluster size. For these datasets, we simply skip the second stage’s joint training, i.e., , while for other datasets .
PReLU (He et al., 2015) is used as the activation function in the VDE and the Decoder. Dropout is not used in any modules of our network. Agglomerative clustering with “ward” linkage is used to initialize cluster centers in the Clustering Module.
Parameter | Equation 23 | |||||
89.71 1.19 | ||||||
88.56 1.55 | ||||||
86.12 1.57 | ||||||
82.89 4.51 | ||||||
80.21 2.94 |
Ansichten | CPSPAN (Jin et al., 2023) | URRL-IMVC (ours) | ||||
---|---|---|---|---|---|---|
Acc(%) | NMI(%) | ARI(%) | Acc(%) | NMI(%) | ARI(%) | |
2 | ||||||
3 | 73.17 4.27 | 61.40 4.29 | 55.37 5.46 | 67.61 0.98 | 63.97 1.33 | |
4 | 84.89 2.15 | 75.37 2.45 | 71.79 3.26 | 91.73 0.47 | 83.57 0.68 | 83.26 0.76 |
5 | 77.62 4.74 | 69.70 4.04 | 63.23 5.51 | 92.95 2.60 | 86.29 1.76 | 86.02 2.91 |
B.2. Datasets and Experiments Setting
Our chosen datasets vary in views (2–6), clusters (7–100), samples (1400–18758), modal (image/text), and feature types (deep/hand-crafted), providing a comprehensive evaluation of approaches. Two parameters missing number and missing rate are defined to control the missing conditions. We first select samples as incomplete samples, then randomly select views of each incomplete sample as missing views. We fix and vary in our experiments, are fixed at 4, 3, 2, 1, 1, 1 for the 6 datasets respectively. Importantly, it should be noted that within the same set of experiments, we ensured that the input data and missing indicator matrix remained consistent across different methods, ensuring fair comparisons. For comparison with state-of-the-art methods in Table 2 we reproduce the results with their published code. Several prior works are difficult to adapt to different numbers of views, which could hinder real applications. We randomly select views when the dataset has more views than the model requires.
B.3. Comparison with State-of-the-art Methods
B.3.1. Comparison with a different number of views
As we mentioned in the introduction, the effectiveness of the cross-view contrastive learning strategy diminishes due to less overlapped information between views. Observing from Table 5, adding more views may harm the clustering performance of the cross-view contrastive learning-based approach, proving this point of view, and also being consistent with the theoretical analysis from (Trosten et al., 2023). On the other hand, our approach stably benefits from more views in the dataset, overcoming this drawback.
B.4. Ablation Studies
Output/Module | NDE | VDE |
---|---|---|
93.360.89 | 89.733.26 | |
91.871.01 | 88.753.56 | |
90.473.15 | 87.233.67 | |
89.364.33 | 89.944.15 | |
- | 87.143.88 | |
Mean | 90.093.61 | 93.360.98 |
Concat+Linear | 83.237.16 | 91.133.38 |
B.4.1. Ablation on Output Choice
We conducted the ablation test in Table 6 to find the best output choice of both the Neighbor Dimensional Encoder (NDE, section 3.1.1) and the View Dimensional Encoder (VDE, section 3.1.2). It can be observed that choosing the first vector of the Transformer output sequence significantly outperforms other choices, and using the latter output vectors results in worse and worse performance. It is consistent with our point of view that NDE needs a bias on the most confident input (the center sample or the nearest neighbor), and further neighbors contain more noise to harm the final performance. For VDE the situation is different, using the average of all output vectors outperforms other choices, which is consistent with our point of view that VDE needs to be unbiased. Concatenation with linear layer does not perform well in both Encoders, possibly due to lack of supervision.
B.4.2. Ablation on View Dropout Probability
We conducted a grid search to determine the best value range of view dropout augmentation probability under different missing rates , and the results are shown in Table 4. For view complete condition, have similar performance, while for view incomplete condition, the desired ascends as the missing rate increases. According to this observation, we designed the mapping function in equation 23 to follow this ascending trend, and its value is listed in the last column of the table.
87.31 2.01 |
B.4.3. Ablation on hyperparameter for KNN
We conduct an ablation study on in KNN Imputation (3.2.1) to examine its effect. A large increment can be observed comparing with . However, the performance starts to drop as , which we infer can be caused by the noise brought by further neighbors. On the other hand, larger also seems to benefit the stability of clustering.