Sparse Explanations of Neural Networks Using Pruned Layer-Wise Relevance Propagation

Let $f:\mathbb{R}^{d}\rightarrow\mathbb{R}^{C},\boldsymbol{x}=(x_{1},\ldots,x_{d})^{\top}\mapsto f(\boldsymbol{x})=(f_{1}(\boldsymbol{x}),\ldots,f_{C}(\boldsymbol{x}))^{\top}$ be the pre-softmax prediction function of a trained DNN for a classification task with $C$ classes, which we also refer to as model. For our setting, we are only interested in the prediction score of the winning class, denoted by $f_{c^{*}}(\boldsymbol{x})$. For a given model, an explanation method is a function $e:\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}:\boldsymbol{x}\mapsto e(\boldsymbol{x})=(r_{1},...,r_{d})^{\top}$ that assigns a relevance score $r_{i}$ to every input dimension $i=1,\ldots,d$. We denote by $W_{l}\in\mathbb{R}^{m\times n}$ the weight matrix of the $l$-th layer of the model and by $w_{jk}^{(l)}$ the weight of the edge connecting the $j$-th neuron in layer $l-1$ with the $k$-th neuron in layer $l$. The corresponding activations are denoted by $a_{j}^{(l-1)}$ and $a_{k}^{(l)}$, respectively. Hence, we can write $\boldsymbol{a}^{(l)}=\big{(}W^{\top}_{l}\boldsymbol{a}^{(l-1)}\big{)}^{+}$, where $(\cdot)^{+}$ refers to the entry-wise application of the ReLU function. Since for our purposes we consider only one layer at a time, we will drop the layer-specific superscripts for readability and simply write $w_{jk}$. Furthermore, let $(\cdot)^{-}$ be the function that only keeps the absolute value of the negative part and is zero otherwise.

Finally, for the relevance attribution, we refer to the relevance of the $k$-th neuron in layer $l$ as $r_{k}^{(l)}$ and write $\boldsymbol{r}^{(l)}=(r_{1}^{(l)},\ldots,r_{n}^{(l)})^{\top}$, in particular $\boldsymbol{r}^{(0)}=\boldsymbol{r}=e(\boldsymbol{x})$. For the last layer $L$ by definition, we have $r_{c^{*}}^{(L)}:=f_{c^{*}}(\boldsymbol{x})$ and $r_{i}^{(L)}:=0,i\neq c^{*}$.

As described earlier, LRP propagates the relevance backward from one layer to its predecessor, starting with the prediction score of the output layer $f_{c^{*}}(\boldsymbol{x})$. The total relevance mass is preserved in every layer by the so-called conservation property (Bach et al.,, 2015). The relevance propagation from layer $l$ to $l-1$ can be expressed by a linear map (Sixt et al.,, 2020), i.e., a multiplication with the matrix $M_{l}:\mathbb{R}^{n}\rightarrow\mathbb{R}^{m}:\boldsymbol{r}^{(l)}\mapsto\boldsymbol{r}^{(l-1)}=M_{l}\boldsymbol{r}^{(l)}$. Thereby, the matrix entry in the $i$-th row and $j$-th column represents the proportion of relevance of neuron $j$ in layer $l$ that is allocated to neuron $i$ in layer $l-1$. As for the weights, we will disregard the layer subscript and only write $M$. There are different LRP rules for how these proportions are calculated (Montavon et al.,, 2018). Specifically, for the LRP-0 rule, the matrix $M$ is given by

Note that by definition, the columns of $M$ are normalized corresponding to redistributing the whole relevance of a neuron. For choosing a rule that yields measurably appropriate relevance attributions, we follow the composite strategy of Kohlbrenner et al., (2020) that combines various rules for different layer types.

The goal of pruned layer-wise relevance propagation (PLRP) is to create sparser explanations by reducing noise and simultaneously increasing the attribution to the most relevant features. We do this by pruning the explanation directly in every layer. This consists of two steps: first, determining the neurons to be pruned, and second, redistributing their relevance mass among the remaining neurons. The criterion for pruning the neurons is the relevance itself, i.e., the neurons with the lowest relevance scores get pruned. Hence, first, we conduct a preliminary unpruned relevance attribution to layer $l$, prune it, and then after redistribution, use it as input for the attribution to layer $l-1$. As it is unclear in advance how the relevance is distributed among the neurons, we prune a proportion of the total relevance mass instead of a proportion of the neurons.

By redistributing the pruned relevance mass among the remaining neurons, we maintain the relevance conservation of LRP. As we are potentially interested in strong negative contributions, we consider the positive and negative relevance separately, i.e., we aim to prune neurons with small relevance in absolute terms. Hence, in the following by $\boldsymbol{r}^{(l)}$ we either mean $\big{(}\boldsymbol{r}^{(l)}\big{)}^{+}$ oder $\big{(}\boldsymbol{r}^{(l)}\big{)}^{-}$. Note that the proportion of pruned mass can be different for the positive and negative parts of the relevance vector. We do not prune the relevance propagation in the last step, i.e., the first layer is left unpruned. This would correspond to a simple thresholding of the input because it is not further propagated through further layers and thus is neither desired nor meaningful.

Sparsity and localization are intended to measure the efficacy of our method, while faithfulness and robustness serve as sanity checks for its trustworthiness.