Investigating Calibration and Corruption Robustness of Post-hoc Pruned Perception CNNs: An Image Classification Benchmark Study

When assessing CNNs, most works concentrate on standard performance metrics such as accuracy and F1 score (for classification) or mean average precision (for object detection). To ensure the safety of CNNs, one needs to consider their technology-specific insufficiencies. Sämman et al. [43] and Schwalbe et al. [46] categorized them into: lack of generalized performance; incorrect internal logic; lack of robustness against perturbations that do not change the semantic content of the input; lack of efficiency for the respective hardware, usually correlated with the CNN size; and CNN opaqueness, which, however, is hard to quantify across use-cases objectively [62, 44]. This catalogue was extended to consider incorrect uncertainty estimate outputs separately measured by variants of the calibration error metric - expected, average, and maximum calibration error [15]. Some safety metrics found in the literature are specifically tailored to the safety needs as well as potential logical issues tied to a use-case [56, 45]. Examples are consistency with logical constraints [47] or standard detection accuracy weighted by safety-relevance of objects [6]. However, such metrics require a concrete reference system and are hardly generalized across use cases. Robustness against perturbations, however, gives rise to a rich set of application-agnostic metrics. As of [46] and [56], these are divided into two classes: adversarial and natural corruption robustness. The first considers robustness against adversarial attacks, i.e., targeted and maliciously crafted changes to the input [5, 54, 37], and sets focus on security. On the other side, natural corruption robustness considers naturally occurring non-semantic corruptions of inputs, such as noise and translations from sensor degradation, adverse weather conditions like fog and rain [46]. Percentage change in accuracy, the relative change in failure rate, mean performance under corruption, and mean corruption error are the most popular metrics for both robustness against adversarial attacks and natural corruption robustness [37, 24]. Our work focuses on expected calibration error and mean performance under corruption as the safety metrics for uncertainty calibration and natural corruption robustness, respectively. The aforementioned safety measures are explored mostly for vanilla networks without considering model compression.

In order to leverage the power of large state-of-the-art computer vision CNNs in resource-constrained environments, compression techniques must be employed to reduce the number of parameters and, thus, computation operations and memory of the deployed CNN. As of He et al. [22], typical and complementary means of compression are quantization, i.e., compression of the weight value representation [42], and CNN pruning. Pruning started as early as the 1980s [41], and, unlike hardware-dependent quantization, aims to remove parameters on an architectural level. Following [34]; pruning methods can be differentiated by their application time (dynamic during operation versus static before deployment), their pruning criterion, the removed elements, and whether they are structured or unstructured. Operation-time pruning employs decision logic at runtime to dynamically remove computational paths [34], but cannot decrease CNN memory consumption.

Static pruning approaches can be sub-classified according to their instance of interference before deployment: 1. Post-hoc: Pruning after training is a three-step process: first, train the initial network to convergence; second, prune redundant parameters based on specific criteria; and finally, retrain the pruned model (fine-tuning) to recover any performance loss incurred during the pruning process [17, 33, 35]. 2. Ante-hoc: (a)Pruning during training: compared to pruning after training, connections are dynamically deactivated during training based on their importance. But later weights can adapt and potentially be reactivated based on gradient updates [64]. (b)Pruning before training resp. weight rewinding: motivated by the Lottery Ticket Hypothesis [9] some recent studies aim to identify a sparsity mask that can be used for weight initialization, e.g., using information from a previous training run, and subsequently train the pruned network from scratch while maintaining the original mask throughout the training process [32, 53]. In this article, we have adopted approach 1., i.e., pruning after training or post-hoc pruning, which not only has the largest body of research [31], but also most practical relevance due to the separation of concerns of model training and integration. In post-hoc pruning, the model can be pruned with different compression ratios without the necessity of initiating training from its initial state. This affords flexibility in modifying compression ratios without undertaking a comprehensive model retraining. Conversely, when employing ante-hoc algorithms for model pruning with diverse compression ratios, it is imperative to initiate model retraining to accommodate the specific compression ratio, contrasting the more adaptable approach offered by post-hoc pruning.

In our work, we consider two fundamental categories of static post-hoc pruning: unstructured pruning and structured pruning. Unstructured pruning involves the removal of individual weights which are assigned the least importance for the network functioning, resulting in a sparse network without sacrificing predictive performance [18, 17]. However, since the positions of non-zero weights are irregular and random, the sparse network pruned by unstructured pruning cannot be presented in a structured fashion. This means that it cannot lead to compression and speedup without dedicated hardware or libraries [19]. To address this, so-called structured pruning typically prunes complete channels, filters, or layers within the CNN architectural structure according to an importance criterion [33, 21, 35]. As a result, the pruned network retains the original convolutional structure without introducing sparsity, and no sparse libraries or specialized hardware are required to realize the benefits of pruning. Since both have practical advantages, we cover and compare these two pruning paradigms in this study.

For ease of notation, pruning refers to post-hoc pruning in the remainder of this paper.

There is a limited exploration of the impact of pruning on uncertainty calibration. While a study conducted by Sun et al. [50] indicates that sparsification is beneficial for enhancing the calibration of CNNs, their study specifically focused on various unstructured pruning methods applied to smaller residual networks (ResNet-20, ResNet-32) with lower sparsity levels (up to 20%). Consequently, these investigations lack a comprehensive analysis of diverse pruning methodologies involving larger residual networks and higher sparsity levels. This gap in research underscores the fact that the impact of pruning on uncertainty calibration remains both active and largely unexplored which we aim to tackle here.

There is a growing body of literature on the robustness of pruning methods against adversarial attacks, focusing mainly on the improvement of the adversarial robustness of pruned models even without adversarial training [27, 16]. According to these results, pruned models typically do not inherit the susceptibility to adversarial attacks observed in the original models. The previously cited studies demonstrate the positive effect of pruning concerning robustness against adversarial attacks. However, this is limited to adversarial robustness, not taking into account robustness against naturally occurring corruptions like fog or random noise from sensor degradation.

Compared to research on adversarial robustness, there is a limited exploration of the impact of pruning on natural corruption robustness. Regarding the relationship between pruning and natural corruption robustness, the study conducted by Hooker et al. [26] revealed that the corruption performance significantly deteriorates when higher pruning ratios are applied for the magnitude pruning method. According to Hoffmann et al. [25], unstructured (global weight) pruning preserves more robustness regarding performance under corruption than the structured ($L_{1}$-norm filter) pruning counterparts. However, such studies lack a comprehensive examination of diverse pruning methodologies, leaving the influence of pruning on natural corruption robustness an active and unexplored research area.

All static pruning methods commonly consist of a step 1, in which they determine an importance score for candidate network units, step 2, in which this is used to select and remove items with a low score, and an optional last step of fine-tuning the newly obtained pruned network. The decision threshold in the second step influences the final pruning ratio, i.e., the percentage of removed parameters, which can serve as a measure for the achieved compression.

As outlined in Section 2.2, static pruning approaches that apply after model training can be divided into unstructured pruning of weights, and structured pruning of filters or channels in case of CNNs [34]. To evaluate pruning with respect to uncertainty calibration and natural corruption robustness and to compare different pruning paradigms with respect to these aspects, we consider the following three pruning methods, which are commonly used and chosen to cover a broad range of static pruning approaches [30].

Besides efficiency, the two main application-agnostic safety targets for computer vision CNNs are correct uncertainty calibration and robustness against naturally occurring corruptions [46, 37, 62]. These are quantified by means of expected calibration error and mean performance under corruption on a benchmark corruption dataset, respectively.

Within the network pruning literature, CIFAR-10 stands as the established benchmark dataset, and VGG, ResNet serve as the prevalent network architectures. Our assessment of the three pruning methods aligns with the target models and dataset pairs presented in the original paper, ensuring the comparability of our results.

We adopt the implementation and hyperparameters for weight pruning, filter pruning, and channel pruning from the publicly available codebase by Liu et al. [36], demonstrating comparable results to the original works. Using a stochastic gradient descent optimizer, the original models are trained for 160 epochs with a batch size of 64. An exponentially decreasing learning rate is applied, starting at 0.1 for epochs [1, 80), 0.01 for epochs [80, 120), and finally 0.001 until epoch 160. Simple data augmentation involving random crop and random horizontal flip is used on training images as a standard means to foster natural corruption robustness in the original models. For fine-tuning the pruned models, we use a constant learning rate set to the last one used for training the original model (0.001) and apply this for 40 epochs.

The pruning ratios vary among the selected pruning methods, each characterized as follows:

To examine the impact of pruning on network uncertainty calibration, we compared $\mathit{ECE}$ and accuracy under different pruning ratios for the three selected pruning techniques. The results are shown in Figure 1 for weight, filter, and channel pruning, respectively (we conduct each experiment three times and report $\text{mean}\pm\text{std}$).

The results suggest that even high pruning ratios do not impact the uncertainty calibration compared to that of the original unpruned model. In unstructured (weight) pruning, the calibration error for all pruning ratios is less than the calibration error of the original unpruned model. Hence, unstructured pruning can even enhance the uncertainty calibration of the VGG-19 and ResNet-110 model. Whereas, in filter and channel pruning, the calibration error for pruned models up to a certain pruning ratio (filter pruning: 50% on VGG-16 and 50% on ResNet-110, channel pruning: 40% on VGG-19 and 50% on ResNet-164) is less than or similar to the calibration error of the original unpruned model. After that, the calibration errors of pruned models do not increase substantially with respect to the calibration error of the unpruned model.

To answer this question, the choice of pruning methods and ratio are kept similar, as mentioned in Section 5.1, but naturally corrupted data is considered as test data. Here, $\mathit{mPC}$ is measured for unpruned and pruned models for the selected pruning methods from Section 3.1. $\mathit{mPC}$ is calculated separately for each of the five corruption severity levels from CIFAR-10-C, each pruning ratio step, and each pruning method. Figure 2 illustrates how natural corruption robustness is influenced by pruning for different severity levels for weight, filter, and channel pruning, respectively.

The observation implies that the natural corruption robustness of weight pruned and filter pruned models, as assessed through $\mathit{mPC}$, is better or similar compared to the original unpruned model. Notably, the robustness against natural input corruption remains unaffected by the weight pruning of VGG-19, ResNet-110, filter pruning of VGG-16, ResNet-110, and channel pruning of VGG-19 across all severity levels. In contrast, for channel pruning, the $\mathit{mPC}$ of ResNet-164 starts to degrade from ca. 60% pruning ratio for all corruption levels. Consequently, the robustness experiences deterioration with higher pruning rates for channel pruning in the ResNet-164 model across all severity levels. This suggests that information needed to compensate for corruption is highly distributed over channels, i.e., wider networks (in terms of the number of channels) might have better chances of achieving zero-shot natural corruption robustness than narrow ones.

While the pruning ratio seems to play a negligible role in accuracy in the presence of corruption, one should, however, note the severe drop in accuracy for increasing corruption severity levels (from more than 90% without corruption in Figure 1 down to less than 55%). This attests to the generally weak overall robustness of CNNs against strong natural corruption.

Here, $\mathit{ECE}$ is measured for unpruned and pruned models using the different pruning methods from above in the presence of natural corruption of different severity levels. Figure 3 illustrates how the uncertainty calibration is influenced by pruning in the presence of natural corruption for weight, filter, and channel pruning, respectively.

The results demonstrate that pruning does not negatively impact the model uncertainty calibration compared to the original unpruned model, even when additionally confronted with natural corruptions. In weight, filter, and channel pruning, the measured calibration error for all pruned models at different pruning ratios is similar to or less than the calibration error of the original unpruned VGG-19, VGG-16, ResNet-110 and ResNet-164 models for all five severity levels of corruption. Nevertheless, as for $\mathit{mPC}$, $\mathit{ECE}$ increases rapidly with increasing severity levels and worse $\mathit{mPC}$, reaching more than 400% increase of $\mathit{ECE}$ for the highest level. Hence, natural corruptions seem not only to pose a challenge to robust accuracy but also to trustworthiness in terms of uncertainty calibration.