Simple Transferability Estimation for Regression Tasks

Consider a source training set $\mathcal{D}_{s}=\{(x^{s}_{i},y^{s}_{i})\}_{i=1}^{n_{s}}$ and a target training set $\mathcal{D}_{t}=\{(x^{t}_{i},y^{t}_{i})\}_{i=1}^{n_{t}}$ consisting of $n_{s}$ and $n_{t}$ examples respectively, where $x^{s}_{i},x^{t}_{i}\in\mathbb{R}^{d}$ are $d$-dimensional input vectors, $y^{s}_{i}\in\mathbb{R}^{d_{s}}$ is a $d_{s}$-dimensional source label vector, and $y^{t}_{i}\in\mathbb{R}^{d_{t}}$ is a $d_{t}$-dimensional target label vector. Here we allow multi-output regression tasks (with $d_{s},d_{t}\geq 1$) where the source and target labels may have different dimensions ($d_{s}\neq d_{t}$). In the simplest case, the source and target tasks are both single-output regression tasks where $d_{s}=d_{t}=1$.

In this paper, we will refer to a model (such as $w$, $w^{*}$, $h$, $h^{*}$, $k$, or $k^{*}$) and its parameters interchangeably. Using the source dataset $\mathcal{D}_{s}$, we train a deep learning model $(w^{*},h^{*})$ consisting of an optimal feature extractor $w^{*}$ and an optimal regression head $h^{*}$ that minimizes the empirical MSE loss:²²2Here we assume $(w^{*},h^{*})$ is a global minimum of Eq. (1). However, practical optimization algorithms often only return a local minimum for this problem. The same is also true for Eq. (3).

where $w:\mathbb{R}^{d}\rightarrow\mathbb{R}^{d_{r}}$ is a feature extractor network that transforms a $d$-dimensional input vector into a $d_{r}$-dimensional feature vector, $h:\mathbb{R}^{d_{r}}\rightarrow\mathbb{R}^{d_{s}}$ is a source regression head network that transforms a $d_{r}$-dimensional feature vector into a $d_{s}$-dimensional output vector, and $\mathcal{L}(w,h;\mathcal{D}_{s})$ is the empirical MSE loss of the whole model $(w,h)$ on the dataset $\mathcal{D}_{s}$:

with $\|\cdot\|$ being the $\ell_{2}$ norm. In practice, we usually consider a source model (e.g., a ResNet [He et al., 2016]) as a whole and use its first $l$ layers from the input (for some chosen number $l$) as the feature extractor $w$. The regression head $h$ is the remaining part of the model from the $l$-th layer to the output layer, and the prediction for any input $x$ is $h(w(x))$.

After training the optimal source model $(w^{*},h^{*})$, we perform transfer learning to the target task by freezing the optimal feature extractor $w^{*}$ and re-training a new regression head $k^{*}$ using the target dataset $\mathcal{D}_{t}$, also by minimizing the empirical MSE loss:

where $k:\mathbb{R}^{d_{r}}\rightarrow\mathbb{R}^{d_{t}}$ is a target regression head network that may have a different architecture than that of $h$. In general, the regression heads $h$ and $k$ may contain multiple layers and are not necessarily linear.

This transfer learning algorithm, usually called head re-training, has been widely used for deep learning models [Donahue et al., 2014, Oquab et al., 2014, Sharif Razavian et al., 2014, Whatmough et al., 2019] and will be used for our theoretical analysis. In practice and in our experiments, we also consider another transfer learning algorithm, widely known as fine-tuning, where we fine-tune the trained feature extractor $w^{*}$ on the target set, and then train a new target regression head $k^{*}$ with this fine-tuned feature extractor [Agrawal et al., 2014, Girshick et al., 2014, Chatfield et al., 2014, Dhillon et al., 2020].

As our first contribution, we propose a definition of transferability for regression tasks and a new formulation for the transferability estimation problem. For this purpose, we make the standard assumption that the target data $\mathcal{D}_{t}$ are drawn iid from the true but unknown distribution $\mathbb{P}_{t}:=\mathbb{P}(X^{t},Y^{t})$; that is, $(x^{t}_{i},y^{t}_{i})\stackrel{{\scriptstyle\mathrm{iid}}}{{\sim}}\mathbb{P}_{t}$. We do not make any assumption on the distribution of the source data $\mathcal{D}_{s}$, but we assume a source model $(w^{*},h^{*})$ is pre-trained on $\mathcal{D}_{s}$ and then transferred to a target model $(w^{*},k^{*})$ using the procedure in Section 3.1.

We now define the transferability between the source dataset $\mathcal{D}_{s}$ and the target task represented by $\mathbb{P}_{t}$. In our Definition 3.1 below, the transferability is the expected negative $\ell_{2}$ loss of the target model $(w^{*},k^{*})$ on a random example drawn from $\mathbb{P}_{t}$. From this definition, the lower the loss of $(w^{*},k^{*})$, the higher the transferability.

In the above definition, transferability is also equivalent to the negative expected (true) risk of $(w^{*},k^{*})$. Next, we formulate the transferability estimation problem. Previous work [Tran et al., 2019, Huang et al., 2022] defined this problem as estimating $\mathrm{Tr}(\mathcal{D}_{s},\mathbb{P}_{t})$ from the training sets $(\mathcal{D}_{s},\mathcal{D}_{t})$, i.e., to derive a real-valued metric $\mathcal{T}(\mathcal{D}_{s},\mathcal{D}_{t})\in\mathbb{R}$ such that ${\mathcal{T}(\mathcal{D}_{s},\mathcal{D}_{t})\approx\mathrm{Tr}(\mathcal{D}_{s},\mathbb{P}_{t})}$. However, in most applications of transferability estimation such as task selection [Tran et al., 2019, Huang et al., 2022, You et al., 2021] or model ranking [Huang et al., 2021, Li et al., 2021], an accurate approximation of $\mathrm{Tr}(\mathcal{D}_{s},\mathbb{P}_{t})$ is usually not required since $\mathcal{T}(\mathcal{D}_{s},\mathcal{D}_{t})$ is only used for comparing tasks or models. Thus, we propose below an alternative definition for this problem that better aligns with its practical usage.

In our new definition, a transferability estimator $\mathcal{T}(\mathcal{D}_{s},\mathcal{D}_{t})$ is a function of $(\mathcal{D}_{s},\mathcal{D}_{t})$ that can be used for comparing or ranking transferability. It does not need to be an approximation of $\mathrm{Tr}(\mathcal{D}_{s},\mathbb{P}_{t})$. This is a generalization of previous definitions [Nguyen et al., 2020, Huang et al., 2022] and can be used for source task selection (when $\mathbb{P}_{t}=\mathbb{P}^{\prime}_{t}$ and ${\mathcal{D}_{t}=\mathcal{D}^{\prime}_{t}}$) as well as target task selection (when $\mathcal{D}_{s}=\mathcal{D}^{\prime}_{s}$). It is consistent with the usage of transferability estimators and the way they are evaluated in the literature by correlation analysis [Tran et al., 2019, Nguyen et al., 2020, You et al., 2021, Huang et al., 2022].

To reduce the cost of computing $\mathcal{L}(w^{*},k^{*};\mathcal{D}_{t})$, a simple idea is to approximate it with an $\ell_{2}$-regularized linear regression (Ridge regression) head. This leads to our first simple transferability estimator, Linear MSE, which is defined as the negative regularized MSE of this Ridge regression head. In this definition, $\|\cdot\|_{F}$ is the Frobenius norm.

Here we add a regularizer to avoid overfitting when the target dataset $\mathcal{D}_{t}$ is small. Previous work such as LogME [You et al., 2021] proposed to prevent overfitting by taking a Bayesian approach, which is more complicated and expensive. We will show empirically in our experiments (Section 6.3) that our simple regularization approach can tackle the issue more effectively and efficiently.

Given a pre-trained feature extractor $w^{*}$ and a target set $\mathcal{D}_{t}$, we can compute $\mathcal{T}^{\mathrm{lin}}_{\lambda}(\mathcal{D}_{s},\mathcal{D}_{t})$ efficiently using the closed form solution for Ridge regression or using second-order optimization [Bishop, 2006]. If the target regression head $k^{*}$ is a linear regression model, $\mathcal{T}^{\mathrm{lin}}_{0}(\mathcal{D}_{s},\mathcal{D}_{t})$ with $\lambda=0$ is the negative MSE of the transferred target model $(w^{*},k^{*})$ on $\mathcal{D}_{t}$. If $k^{*}$ has more than one layer with a non-linear activation, $\mathcal{T}^{\mathrm{lin}}_{\lambda}(\mathcal{D}_{s},\mathcal{D}_{t})$ can be regarded as using a regularized linear model to approximate this non-linear head.

Although the Linear MSE transferability score above can be computed efficiently, this computation may still be relatively expensive if the feature vectors $w^{*}(x^{t}_{i})$ are high-dimensional. To further reduce the costs, we propose another transferability estimator, Label MSE, which replaces $w^{*}(x^{t}_{i})$ by the “dummy” source label $z_{i}=h^{*}(w^{*}(x^{t}_{i}))$. Using dummy labels from the pre-trained source model $(w^{*},h^{*})$ is a technique previously used to compute the LEEP transferability score for classification [Nguyen et al., 2020]. We define our Label MSE estimator below.

In practice, since the size of $z_{i}$ is usually much smaller than that of $w^{*}(x^{t}_{i})$ (i.e., $d_{s}\ll d_{r}$), computing the Label MSE is usually faster than computing the Linear MSE.

In this definition, the Shared Inputs Label MSE is computed by training a Ridge regression model directly from the true label pairs $(y^{s}_{i},y^{t}_{i})$, which is less expensive than the original Label MSE since we do not need to train the source model $(w^{*},h^{*})$ or compute the dummy labels.

Intuitively, our estimators use a weaker version of the actual target model that helps trade off the estimators’ accuracy for computational speed. Our estimators can also be viewed as instances of the kernel Ridge regression approach [Smale and Zhou, 2007, Hastie et al., 2009]. While the Linear MSE can be interpreted as a linear approximation to $-\mathcal{L}(w^{*},k^{*};\mathcal{D}_{t})$, properties of the Label MSE and Shared Inputs Label MSE are not well understood. In the next section, we shall prove novel theoretical properties for these estimators.

This experiment considers the general case where source models are trained on one dataset (OpenMonkey) and then transferred to another (CUB-200-2011). Specifically, we train a source model for each of the 17 keypoints of the OpenMonkey dataset and transfer them to each of the 15 keypoints of the CUB-200-2011 dataset, resulting in a total of 255 final models. Since each keypoint consists of x and y positions, all source and target tasks in this experiment have two dimensional labels. The actual MSEs of these models are computed on the respective test sets and then used to calculate the Pearson correlation coefficients with the transferability estimators. In this experiment, LabMSE0, LabMSE1, LabLogME, and LabTransRate are computed from the dummy source labels and the actual target labels.

Results for this experiment are in Table 1. In this setting, TransRate and LabTransRate perform poorly, while our methods are equal or better than LogME and LabLogME in most cases, especially when using $\lambda=1$ (LinMSE1) or dummy labels (LabMSE0 and LabMSE1). The results show our approaches improve up to 25.9% in comparison with SotA (LogME) while being 12.9% better on average.

It is interesting to observe that LabMSE0 and LabMSE1 provide competitive or even better correlations than LinMSE0 and LinMSE1 in this experiment. This shows that the dummy labels (i.e., body parts of monkeys) can provide as much information about the target labels (i.e., body parts of birds) as the extracted features.

In the Appendix C.2, we also report additional results where both source and target tasks have 10-dimensional labels (i.e., each task predicts 5 keypoints simultaneously). We also achieve better correlations than the baselines in this case.

In this experiment, we consider the setting where the source and target tasks have the same inputs (the special setting in Section 4.2). Since images in our datasets contain multiple labels (15 keypoints for CUB-200-2011 and 17 keypoints for OpenMonkey), we can use any two different keypoints on the same dataset as source and target tasks. In total, we construct 210 source-target pairs for CUB-200-2011 and 272 pairs for OpenMonkey that all have the same source and target inputs but different labels. The labels for all tasks are also two dimensional real values.

We repeat the experiment in Section 6.1 with these source-target pairs for CUB-200-2011 and OpenMonkey separately. The main difference in this experiment is that we use the true source labels (instead of dummy labels) when computing LabLogME, LabTransRate, LabMSE0, and LabMSE1. Under this setting, the LabMSE estimators here are the Shared Inputs Label MSE estimators in Definition 4.3. These estimators can be computed without any source models, and thus incurring very low computational costs in this setting.

Results for these experiments are in Table 2. In the results, both versions of TransRate perform poorly on CUB-200-2011, while TransRate is slightly better than LogME on OpenMonkey. In most settings, LabMSE0 and LabMSE1 both outperform LabLogME and LabTransRate, while LinMSE0 and LinMSE1 both outperform LogME and TransRate. In the setting where we transfer by full fine-tuning on the CUB-200-2011 dataset, all methods perform poorly. From these results, our approaches improve up to 113% in comparison with SotA (LogME) while being 36.6% better on average.

We also report in the Appendix C.3 additional results for each individual source task. The results show that our methods are consistently better than LogME, LabLogME, TransRate, and LabTransRate for most source tasks on both datasets. Furthermore, our methods are also better than these baselines when transferring to higher dimensional target tasks (tasks that predict 5 keypoints simultaneously and have 10-dimensional labels). These additional results further confirm the effectiveness of our approaches.

In many real-world transfer learning scenarios, the target set is usually small. This experiment will evaluate the effectiveness of the feature-based transferability estimators (LogME, TransRate, LinMSE0, and LinMSE1) in this small data regime where the number of samples is smaller than the feature dimension. For this experiment, we fix a source task (Belly for CUB-200-2011 and Right eye for OpenMonkey) and transfer to all other tasks in the corresponding dataset using head re-training. These source tasks are chosen since they have fewer missing labels and thus can be used to train reasonably good source models for transfer learning. For each target task, instead of using the full data, we randomly select a small subset of 100 to 400 images to perform transfer learning and to compute the transferability scores. The actual MSEs of the transferred models are still computed using the full target test sets.

Figure 3 compares the correlations of the 4 methods on different target set sizes between 100 and 400. The results are averaged over 10 runs with 10 different random seeds. From the figure, LogME and LinMSE1 are better than TransRate and LinMSE0. This is expected since LogME and LinMSE1 are designed to avoid overfitting on small data. Both LogME and LinMSE1 are also more stable, but LinMSE1 is slightly better than LogME on all dataset sizes.

One of the main strengths of our methods is their efficiency due to the simplicity of training the Ridge regression head. In this experiment, we first use the settings in Section 6.2 to compare the running time of our methods with that of the baselines on the CUB-200-2011 dataset. Figure 3 (left) reports the results (averaged over 5 runs with different random seeds) for this experiment. From these results, our methods, LabMSE0, LabMSE1, LinMSE0, and LinMSE1, are all faster than the corresponding label-based or feature-based baselines. The figure also shows that LabMSE1 and LinMSE1 achieve the best running time among the label-based and feature-based methods respectively.

In Figure 3 (right), we also compare the average running time of the 4 transferability estimators using the CUB-200-2011 experiment in Section 6.3. This figure clearly shows that our methods, LinMSE0 and LinMSE1, are more computationally efficient than LogME and TransRate. Both results in Figure 3 show that LinMSE1 and LabMSE1 are significantly faster than other corresponding feature-based and label-based methods. In these experiments, LinMSE1 and LabMSE1 converge faster than LinMSE0 and LabMSE0 respectively, and thus are more efficient.

Source task selection is important for applying transfer learning since the right source task can improve transfer learning performance [Nguyen et al., 2020]. In this experiment, we examine the application of our transferability estimation methods for selecting source tasks on the CUB-200-2011 dataset. We use the head re-training setting similar to Section 6.2, but fix one of the tasks as the target and choose the best source task from the rest of the task pool. We repeat this process for all 15 target tasks and measure the top-$k$ matching rate of each transferability estimator.

The top-$k$ matching rate is defined as $m_{\text{match}}/m_{\text{target}}$, where $m_{\text{target}}$ is the total number of target tasks (15 in our case), and $m_{\text{match}}$ is the number of times the selected source task gives a target model within the best $k$ models. Here the best $k$ models are determined by the actual test MSE on the target task.

Results for this experiment are in Table 3. From the results, our methods are better than the baselines in terms of top-$3$ and top-$5$ matching rates. When comparing top-$1$ matching rates, our methods are competitive with LogME and LabLogME for the feature-based and label-based approaches respectively. This experiment shows that our transferability estimators are useful for source task selection.

In this experiment, we investigate the effects of $\lambda$ on our proposed transferability estimators. We use the setting in Section 6.2 with the CUB-200-2011 dataset and vary the value of $\lambda$ in [0, 20] for both LabMSE and LinMSE. Table 4 reports the results for all three transfer learning settings.

For head re-training, we observe that the best correlations are achieved at $\lambda=1$ for both LabMSE and LinMSE. For half fine-tuning, $\lambda\geq 5$ gives the best result for LabMSE, while $\lambda=0.001$ gives the best result for LinMSE. For full fine-tuning, we do not observe significant correlations for both transferability estimators.

Notably, from the results in Table 4 for the head re-training and half fine-tuning settings (where we have significant correlations for at least one transferability estimator), LabMSE with any tested $\lambda$ value in [0, 20] is better than LabLogME and LabTransRate, while LinMSE with any tested $\lambda$ value in this range is better than LogME and TransRate. These results show that our methods are better than the baselines for a wide range of $\lambda$ values.

Although our paper mainly focuses on regression tasks, the main idea of using the negative regularized MSE of a Ridge regression model for transferability estimation goes beyond regression. In principle, this idea can be applied for transferring between classification tasks (in this case, we should train a linear classifier and use its regularized log-likelihood as the transferability estimator) or between a classification and a regression task.

In this section, we demonstrate that our idea can be applied for transferability estimation between a classification and a regression task. Particularly, we use 8 source models pre-trained on ImageNet [Deng et al., 2009] and transfer to a target regression task on the dSprite dataset [Matthey et al., 2017] using full fine-tuning. This setting is similar to You et al. [2021] where the target is a regression task with 4-dimensional labels: x and y positions, scale, and orientation. We compute the transferability scores from the extracted features and the labels of the target training set. More details about this experiment are in the Appendix B.2.

From the results in Figure 3, the trends for LogME, LinMSE0, and LinMSE1 are correct (i.e., transferability scores have negative correlations with actual MSEs), while that of TransRate is incorrect. Note that there is a discrepancy between the ranges of the transferability and the transferred MSE because of two reasons: (1) The transferability estimators are computed from the target training set, while the transferred MSEs are computed from the target test set, and (2) there is a mismatch between the source task (ImageNet classification) and the target task (dSprite shape regression).

To compare the transferability estimation methods, we fit a linear regression to the points in each plot and compute its RMSE to these points, where we obtain: $6.12\times 10^{-3}$ (LogME), $6.16\times 10^{-3}$ (TransRate), $6.10\times 10^{-3}$ (LinMSE0), and $\textbf{5.46}\times 10^{-3}$ (LinMSE1). These results show that LinMSE0 and LinMSE1 are better than LogME and TransRate.