Hadamard Adapter: An Extreme Parameter-Efficient Adapter Tuning Method for Pre-trained Language Models

We employ eight tasks in the GLUE benchmark to conduct the first analysis of how PLM’s self-attention output changes before and after fine-tuning. For PLM, we adopt Roberta-large model, which has 24 hidden layers and outputs 1024-dimensional tensors in the encoder, as an example to make analysis. Specifically, in order to compare the changes of self-attention outputs in each layer among all tasks, we adopt the norm of self-attention outputs instead of the original self-attention outputs. We analyze the distribution of the norm of self-attention outputs among all tasks before and after fine-tuning, and the changes during fine-tuning on each layer as shown in Fig. 1. The process is shown as the following equations:

where $\|\bm{A_{b}}\|_{2}$ and $\|\bm{A_{b}}\|_{2}$ represent the norm of self-attention outputs among all tasks before and after fine-tuning in a hidden layer, and $\lambda_{max}(\bm{A_{b}}^{T}\bm{A_{b}})$ is the eigenvalue of the matrix $\|\bm{A_{b}}\|_{2}$.

One box in Fig. 1(a)(b) and Fig. 1(c) represents the distribution of the norm of self-attention outputs and the corresponding changes in the layer, respectively. As can be observed in Fig. 1, the norm of self-attention outputs of all tasks significantly increase from an average of 60 to an average of 100 after fine-tuning, especially in the middle and back layer (Fig 1(a)(b)). After the fifteen layers, the changes become more significant as the number of layers increases, reaching the greatest changes at the last layer (Fig 1(c)). The above observations indicate that self-attention outputs change significantly during the fine-tuning process, which inspire us with the answer to Q1 as follows: It is proper to inject an adapter right after the self-attention outputs to achieve similar performance gains with fine-tuning while updating much fewer parameters.

We design fitting functions for self-attention outputs to make adapter tuning, which aims at letting the values of self-attention outputs approximate those in full fine-tuning of PLMs. We first optimize the parameters in the classifier modules. Next, we reload them and train different fitting functions, including linear function, quadratic function and higher order function (i.e. cubic function), respectively, to obtain new self-attention outputs. After that, we calculate the average value of each token in a sequence through dividing by the hidden size of the PLM in the new self-attention outputs as shown in Fig. 2(a). The process is shown in the following equations:

where $H$ represents the hidden size of a PLM. $a_{ij}^{\prime}$ is the value of the $i^{th}$ dimension of the $j^{th}$ token in the new self-attention outputs in a hidden layer based on a task. $a_{j}^{\prime}$ is the average value of each token in a sequence of a task. More detailed, we then calculate the average value of each sequence through dividing by sequence length in the new self-attention outputs. In this way, we obtain a characteristic value for each task which represents its respective average self-attention outputs. We analyze the distribution of the characteristic value among all tasks in each layer as shown in Fig. 2(b) with the process shown in the following equations:

where $L$ represents the sequence length fed for the PLM. $a^{\prime}$ is the characteristic value of a task. We also analyze the average characteristic values of all tasks in each layer as shown in Fig. 2(c).

As shown in Fig. 2(a), dots of the same color represents the average value of each token in a sequence of all downstream tasks corresponding to one of the three fitting functions and fine-tuning. The dots of four settings are covered by each other, which also proves that fitting functions of different orders are similar in approximating the performance of fine-tuning. As shown in Fig. 2(b), one box represents the distribution of characteristic values among all downstream tasks in a layer. Median, quartile ranges which correspond to the characteristic value distribution of different fitting functions and fine-tuning are similar in each hidden layer. When we analyze the average characteristic value of all tasks, trends of linear function, quadratic function and higher order function are still similar, but slightly different from that of fine-tuning (Fig 2(c)). One dot represents the average characteristic values of all downstream tasks in a layer. As the order increases, the values between the fitting function and fine-tuning are closer, but the difference in distance can be ignored compared with the increase in the number of parameters. Therefore, we have answer to Q2 as follows: A linear function is qualified enough to act on self-attention outputs to fit the performance of fine-tuning.

We output the gradient and unit gradient of the top five layers in the first and last epoch during training, respectively, of a PLM (such as BERT-base model). Two representative datasets MRPC (similarity and paraphrase task, 3.7k) and SST-2 (single-sentence classification, 67k) from the GLUE benchmark are selected for analysis, and the results are shown in Table  1.

From the results of the gradients, we speculate that the classifier weights, embedding weights and intermediate weights of all tasks are more important in fine-tuning than the other modules because their gradients contribute the most in the first and last epochs. However, we find that some modules which contribute most during the fine-tuning have a large number of parameters, such as the intermediate module, which accounted for nearly half of all parameters. If we still fine-tune them, a lot of redundant computation is inevitable. We further analyze the results of unit gradient, which is the gradient divided by the number of parameters, and the results show that the classifier weights, embedded weights, and normalized weights are more important because their unit gradients contribute the most in the first and last epochs.

Therefore, we have answer to Q3 as follows: We select out the classifier and the normalization as trainable modules in the adapter tuning. We also make a theoretical analysis of the selection of these parameters as follows: i) The classifier makes prediction in the downstream tasks and directly affects the performance of tasks. ii) Normalization is to limit the data within a certain range and unify the distribution of each batch of training data. Since the input data distribution of each batch of the network is constantly changing, the changes of training pattern without normalization will make it difficult for the network to find a balance point, thus affecting the convergence of the network.

The framework of the Hadamard adapter is shown in Fig. 3(A). It is equivalent to a linear transformation based on feature dimension as shown in the following equation:

Different feature dimensions correspond to different linear transformation parameters, but different positions (i.e. each token in a sequence) share the same parameters. We put such an adapter in each layer of the PLMs. Adapters of different layers have different parameters. Based on the empirical analysis in Sec. 2.1, we put this adapter right after the self-attention outputs. Specifically, we multiply the weight vector $\bm{W_{j}}$ of the Hadamard adapter to the self-attention outputs $\bm{A_{ij}}$, and then add the bias vector $\bm{b_{j}}$ to the product to obtain new self-attention outputs $\bm{A_{ij}^{\prime}}$. The weight vector and the bias vector in the Hadamard adapter are both 1-d vectors of the same shape as the hidden size of the PLM, such as 768 for the base version or 1024 for the large version for BERT model. The number of the Hadamard adapter in a PLM is also the same as the number of PLMs’ layers, such as 12 for the base version and 24 for the large version in BERT model. All weight vectors are initialized as 1.0 and all bias vectors are initialized as 0.0. The initial value is equivalent to not adding any adapter. The approximate number of parameters of this adapter is from 30,000 to 100,000 according to different size of PLMs and 3,000 to 4,000 in each layer of PLMs, which is only 0.03% of the full fine-tuning.

The parameter-efficient adapter tuning method is to inject adapters to PLMs and make continuous fine-tuning as shown in Fig. 3(a)(b). Specifically, we first only unfreeze and train the pooling and classifier modules on downstream tasks. Next, we inject the Hadamard adapter right after self-attention outputs. After that, we reload the trained pooling and classifier layers, and only fine-tune the Hadamard adapter and the normalization module in each layer. Although the above-mentioned two-stage training process can be time-consuming and computationally expensive, the performance is better than joint training in our experiments. We have analyze the possible reason in the paper that the proposed Hadamard Adapter has not been optimized in the pre-training stage which will affect the effect of classifier layer. We also find other research, such as LP-FT (Kumar et al., 2022), demonstrating the better performance of two-stage training.

Train the classifier module. For each given downstream task, we first learn only the classifier module on the training dataset, including the pooling and linear output layers. While learning the classifier module, the other parameters of the PLMs are frozen. Since all layers except the classifier module do not participate in backward propagation and gradient update, they can be shared by different tasks. In addition, the computational cost of this step is very small compared to the overall cost of full fine-tuning.

Inject the Hadamard adapter and unfreeze the normalization module. Secondly, we inject the Hadamard adapter right after the self-attention outputs. Multi-head self-attention is an important mechanism in PLMs, which first obtains representation of queries $Q$, keys $K$ and values $V$, and then makes scaled dot-product for $heads=16$ times to obtain self-attention score of each attention head. Outputs of all attention heads are finally concatenated to obtain self-attention output $A_{i}$. Based on multi-head self-attention, we first transform the 3-D dimensional self-attention outputs into 2-D dimensional matrices, that is, making the first and second dimension flatten. Next, we input the 2-D dimensional self-attention outputs into the Hadamard adapter and transform the self-attention outputs into 3-D dimensional matrix $A_{i}^{\prime}$. The process is as follows:

After that, We reload the parameters of the trained classifier module and only unfreeze the normalization module besides the Hadamard adapter. Based on the empirical analysis in Sec. 2.3, we argue that the normalization module is very important to improve the accuracy of downstream tasks. The reason is that the value range of the self-attention outputs changes after being transformed with the Hadamard adapter. Obviously, it is necessary to relearn the parameters of the normalized module and make them adapt to the value distribution of the new self-attention outputs in order to achieve better results. In order to further scale the parameter size and reach the limit of parameter efficiency, we only unfreeze the normalization module right after the intermediate outputs rather than unfreeze that right after the self-attention output.

The experiments are carried out on Tesla V100 GPUs with Pytorch in Python. Similar with previous work (Devlin et al., 2018; Liu et al., 2019; Lewis et al., 2019; He et al., 2020; Clark et al., 2020), the batch size is set to 16 or 32, and the sequence length is set to 128. We maintain the hyper parameters such as 0.01 for weight decay, 0.9 for $\beta_{1}$, and 0.999 for $\beta_{2}$, etc. We set the learning rate from 2e-03 to 4e-03 in training the classifier module, from 2e-05 to 4e-05 in full fine-tuning, and from 1e-03 to 9e-03 in training with the Hadamard adapter to obtain the best performance in each training setting. All tasks are training for 20 epochs with each PLM.

Datasets and Baselines. We adopt GLUE benchmark (Wang et al., 2018), including eight tasks from different domains, as the datasets. They are divided into single-sentence classification (CoLA, SST-2), similarity and paraphrase (MRPC, STS-B, QQP), and inference (MNLI, QNLI, RTE). We ignore the WNLI dataset whose data amount is too small. Same as the previous research (Ben Zaken et al., 2022; Hu et al., 2021) on GLUE benchmark, we also adopt Matthews correlation coefficient ¹¹1https://en.wikipedia.org/wiki/Phi_coefficient and Pearson coefficient ²²2https://en.wikipedia.org/wiki/Pearson_correlation_coefficient to evaluate the performance on the CoLA dataset and STS-B dataset, respectively, and use accuracy for other datasets. We carry out our experiments on several SOTA PLMs, including BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019), BART (Lewis et al., 2019), DeBERTa (He et al., 2020), and ELECTRA (Clark et al., 2020). Moreover, we use powerful parameter-efficient adapters shown in Table 3 as baselines to compare the parameter amount and performance on GLUE benchmark.

We first report the performance of the classifier, Hadamard adapter and full fine-tuning in GLUE benchmark based on several SOTA PLMs in Table 2, where the results all come from our own machines, which represent only fine-tuning the classifier module, the proposed adapter tuning method, and full fine-tuning, respectively. We find that only training the classifier module achieves 77.5% in performance compared with that of the corresponding full fine-tuning averagely in selected PLMs and tasks. It represents that the classifier module is important for solving downstream tasks, so it is necessary to be fine-tuned alone. Next, we reload the trained classifier module, inject the Hadamard adapter for self-attention outputs, and only unfreeze the normalization module. The performance increase a large degree, which achieves 99.4% compared with that of full fine-tuning averagely in selected PLMs and tasks. Some results are even better than the corresponding full fine-tuning, such as the MPRC dataset with the BERT-base model. It represent that the Hadamard adapter has a stunning effect which has competitive performance and achieve extreme parameter efficiency (0.033% parameters of full fine-tuning).

We also compare the performance between the proposed adapter tuning method with Hadamard adapter and other parameter-efficient baselines as shown in Table 3, where baselines’ results are replicated from their corresponding published papers. As can be observed, there is not much difference in performance, but the Hadamard adapter uses the fewest amount of parameters. This comparison results prove that the Hadamard adapter could uses the fewest parameters to achieve competitive performance with full fine-tuning and the existing parameter-efficient tuning methods.

In this part, we carry out ablation study to clarify the effect of each module in the Hadamard adapter as shown in Table 4, as well as the influence of different number of unfreezing layers to the performance as shown in Fig. 4.

The effect of each module. The results for the effect of each module are shown in Table 4. We first unfreeze any one module of the adapter tuning method (see row 2-5). The results represent that the bias vectors in the Hadamard adapter and the normalization module contribute more than the weight vectors (see row 2-5). We also unfreeze the normalization module (those right after intermediate outputs, short as normalization module) and attention-based normalization module (those right after self-attention outputs, short as attention-based normalization module), respectively. We find that the coarse-grained normalization modules (i.e. normalization module) are more necessary than the fine-grained normalization modules (i.e. attention-based normalization module) (see row 4-5). Next, we unfreeze any two modules of the adapter tuning method (see row 6-10). The results represent that the most two effective modules, i.e., bias vectors in the Hadamard adapter and the normalization module, also achieve the best performance after being unfrozen simultaneously. After that, we unfreeze three or four modules of the adapter tuning method (see row 11-13). Compared with the final three modules, we observe that when we add the attention-based normalization module, the performance decreases a little bit. It represents that some parameters are not valuable enough to improve PLMs’ performance in the downstream tasks.

The effect of the number of unfreezing layers. The results for the number of unfreezing layers of the Hadamard adapter on downstream tasks are shown in Table 5. We select two tasks to visualize the trend as shown in Fig. 4. We find that as we unfreeze more layers, the performance increase consistently. And they achieve satisfying performance when we unfreeze over a half of all layers (8 for models of base version and 16 for those of large version). The experiment inspires us that parameters in some layers of the Hadamard adapter are still redundant which can be removed to achieve more parameter-efficient with 0.022% paramters, but this conjecture need to be validated with more datasets and PLMs.