Understanding Forgetting in Continual Learning with Linear Regression:
Overparameterized and Underparameterized Regimes

Continual learning, also known as lifelong learning, is a subfield of machine learning that focuses on developing a model capable of learning continuously from a stream of data, which are i.i.d sampled from different tasks and presented sequentially to the model. A primary challenge in continual learning is the catastrophic forgetting phenomenon (McCloskey & Cohen, 1989), wherein the model forgets previously acquired knowledge when exposed to new data.

Previous research addressing catastrophic forgetting in continuous learning primarily focuses on empirical studies, which can be broadly classified into three categories: expansion-based methods, regularization-based methods, and memory-based methods. Expansion-based methods (Yoon et al., 2017, 2019; Yang et al., 2021) mitigate catastrophic forgetting by allocating distinct subsets of network parameters to individual tasks. Regularization-based methods (Kirkpatrick et al., 2017; Aljundi et al., 2018; Serra et al., 2018; Liu & Liu, 2022) employee structural regularization in fixed capacity models to counteract forgetting, which penalize significant changes in parameters that are crucial for previous tasks. Memory-based methods (Shin et al., 2017; Chaudhry et al., 2018; Riemer et al., 2018; Saha et al., 2021; Lin et al., 2022; Hao et al., 2023) alleviate forgetting by storing subsets of previous task data or synthesizing pseudo-data without data-replay.

Recently, there has been a growing body of work focused on understanding the behavior of catastrophic forgetting from a theoretical standpoint. For example, Bennani et al. 2020; Doan et al. 2021 analyze the generalization of continual learning for Orthogonal Gradient Descent (OGD) (Farajtabar et al., 2020) in the Neural Tangent Kernel (NTK) (Jacot et al., 2018) regime. Lee et al. 2021; Asanuma et al. 2021 explore the impact of task similarity in a teacher-student setting. Evron et al. 2022; Lin et al. 2023 provide a detailed forgetting analysis of the minimum-norm interpolator for the overparameterized linear regression model. However, the existing analyses of forgetting often rely on relatively stringent assumptions that may not be applicable in many scenarios. For example, Bennani et al. 2020; Doan et al. 2021; Evron et al. 2022; Lin et al. 2023 necessitate an overparameterized regime for their analysis, which may be invalid when involving large datasets. Moreover, Lee et al. 2021; Asanuma et al. 2021; Lin et al. 2023; Swartworth et al. 2023 assume that data follows a Gaussian distribution that may not hold in real-world datasets exhibiting more complex distributions. Evron et al. 2022; Lin et al. 2023 focus on the minimum-norm interpolator, where each task requires achieving zero loss on its training samples and hence can find a closed-form solution.

In this paper, we investigate the behavior of forgetting under the linear regression model via the more practical Stochastic Gradient Descent (SGD) method and provide a general theoretical analysis that is applicable to both over-parameterized and under-parameterized regimes. Our main contributions can be summarized as follows:

Firstly, our work provides a theoretical analysis for multi-step SGD algorithms in both underparameterized and overparameterized regimes, with the population data covariance matrix satisfying the general fourth moment instead of Gaussian distribution as in existing studies. In specific, we provide a novel upper bound on the model forgetting, as well as a matching lower bound that shows the tightness of our characterization. Our bounds derive the forgetting bound that is stated as a function of $\mathbf{1)}$ the spectrum of the population data covariance matrices for each task, $\mathbf{2)}$ the step size, $\mathbf{3)}$ the number of training samples and $\mathbf{4)}$ the effective dimensions on the forgetting.

Second, our study provides some interesting insights into the impact of task sequence and algorithmic parameters on the degree of forgetting. Specifically, we show that when the data size is sufficiently large, forgetting tends to escalate when we postpone the training of tasks, whose population data covariance matrices possess larger eigenvalues. It is intuitive that when tasks with larger eigenvalues are trained later, the model might overfit these tasks due to their high variance. In addition, our findings reveal that an appropriate choice of step size can help mitigate forgetting in both underparameterized and overparameterized settings. Note that these results cannot be derived from existing works due to their restrictive data distribution assumptions or closed-form updating rules. More detailed discussions can be found in Section 4.

Finally, we conducted simulation experiments on both linear regression models and Deep Neural Networks (DNNs) to validate our theoretical analysis. Our simulation results indicate that both linear regression models and DNNs exhibit increased forgetting when tasks with larger eigenvalues are encountered later. Additionally, we demonstrate that smaller step sizes in training can also mitigate forgetting across task sequences, especially in under-parameterized settings. Interestingly, we observe that in over-parameterized DNNs, higher dimensionality does not necessarily equate to more forgetting if the dataset size is fixed, as opposite to the linear regression case.

In our setup, we consider a sequence of tasks, denoted as $\mathbb{M}=\{1,2,\ldots,M\}$. For each task $m$ in this sequence, we have a corresponding dataset $D_{m}$, which consists of $N$ data points. Each of these data points, denoted as $(\mathbf{x}_{m,i},y_{m,i})$, is drawn independently and identically distributed (i.i.d.) from a specific distribution $\mathcal{D}_{m}=$ $\mathcal{X}_{m}\times\mathcal{Y}_{m}\subset\mathbb{R}^{d}\times\mathbb{R}$. Here, $\mathbf{x}_{m,i}$ represents the feature vector, and $y_{m,i}$ is the response variable for each data point in the dataset $D_{m}$. Assume that $\{(\mathbf{x}_{m,i},y_{m,i})\}_{i=1}^{N}$ are i.i.d. sampled from a linear regression model, i.e., each pair $(\mathbf{x}_{m,i},y_{m,i})$ is a realization of the linear regression model $y_{m}=(\mathbf{x}_{m}^{\top}\mathbf{w}_{*})+z_{m}$, where $z_{m}$ is some randomized noise and $\mathbf{w}_{*}\in\mathbb{R}^{d}$ is the optimal model parameter.

Our goal is to output a model $\mathbf{w}_{MN}$ minimizing the degree of forgetting (Evron et al., 2022) for $M$ tasks, i.e.

$\mathbf{w}_{MN}$ represents the final output after sequentially training on $M$ tasks, each updated via SGD over $N$ iterations for each task. Equation 1 quantifies an average excess population risk on the final output $\mathbf{w}_{MN}$ across all tasks. For each task $m$, the loss $\mathcal{L}_{m}$ evaluate how well $\mathbf{w}_{MN}$ performs on it, thus assessing the degree of the model’s forgetting on previous tasks in continual learning scenarios.

Let $\mathbf{H}_{m}$ denote the eigen decomposition of the data covariance for task $m$, given by $\mathbf{H}_{m}=\sum_{i}\lambda_{m}^{i}\mathbf{v}_{m}^{i}{\mathbf{v}_{m}^{i}}^{\top}$, where $(\lambda_{m}^{i})_{i\geq 1}$ are eigenvalues in a nonincreasing order and $(\mathbf{v}_{m}^{i})_{i\geq 1}$ are the corresponding eigenvectors. Define $\mathbf{H}_{m,k_{1}:k_{2}}$ as $\mathbf{H}_{m,k_{1}:k_{2}}:=\sum_{k_{1}<i\leq k_{2}}\lambda_{m}^{i}\mathbf{v}_{m}^{i}{\mathbf{v}_{m}^{i}}^{\top},$ and allow $k_{2}=\infty$ to imply that $\mathbf{H}_{m,k:\infty}=\sum_{i>k}\lambda_{m}^{i}\mathbf{v}_{m}^{i}{\mathbf{v}_{m}^{i}}^{\top}$.

Compared to the concept of covariate shift in transfer learning (Pathak et al., 2022), Definition 2.2 provides a more general scenario applicable to a series of tasks $M\geq 2$. For simplicity, in our analysis, we assume that each task $m$ in our model consists of $N$ data points, differentiating it from transfer learning approaches that typically consider the total dataset size as $N$.

Similar to previous works, we assume that $z_{m}$ is some randomized noise that satisfies $\mathbb{E}[z_{m}|\mathbf{x}]=0$ and $\mathbb{E}[z_{m}^{2}]=\sigma^{2}$ for each task $m$.

Before presenting our upper bound, we shall establish the following notations to facilitate comprehension of the results.

where $(\lambda_{m}^{i})_{i\geq 1}$ are eigenvalues of $\mathbf{H}_{m}$ in a nonincreasing order and $k_{m}^{*}=\max\{i:\lambda_{m}^{i}\geq\frac{1}{N\eta}\}$ represents the cut-off index for $\mathbf{H}_{m}$. Here, $\Gamma_{(p,q)}^{i}$ and $\boldsymbol{\Gamma}_{p}^{q}$ can be regarded as a projection accumulation from task $p$ to task $q$, and basically capture the impact of the learning dynamic of previous tasks on the subsequent task. $\mathbf{U}_{k_{m}^{*}}$ is defined with respect to the cut-off index $k_{m}^{*}$ for each task’s data covariance matrix $\mathbf{H}_{m}$ that captures both the dominant eigenvalues and the tail of the spectrum, and $\Lambda^{i}$ denotes the sum of the $i$-th eigenvalue across all tasks.

In the following, we first provide our upper bound for the behavior of forgetting via SGD in the linear regression model.

In Theorem 3.1, we establish an upper bound on the forgetting behavior of a model trained using SGD in the continual learning with various data distribution settings. It highlights that the model’s performance is influenced by both $\text{err}_{\text{var}}$ and $\text{err}_{\text{bias}}$, where $\text{err}_{\text{var}}$ stems from the inherent noise intrinsic to the model itself and $\text{err}_{\text{bias}}$ represents the bias associated with the initial value during the learning process. Notice that both of them are determined jointly by the spectrum of the covariance matrices as well as the stepsizes for continual learning.

To provide a more intuitive explanation, we explore a simplified scenario by setting $\eta=0$. Specifically, this setting simplifies our analysis by reducing the error terms to only the first term in bias error, which appears to depend solely on the initial weight $\mathbf{w}_{0}$ and the data. However, this simplification might misleadingly imply that a minimal $\eta$ would result in optimal learning outcomes. A crucial aspect overlooked in this interpretation is the role of the projection term $\Gamma_{1}^{M}=\prod_{j=1}^{M}(\mathbf{I}-\eta\mathbf{H}_{j})^{2N}$, which becomes an identity matrix $\mathbf{I}$ when $\eta=0$. Thus, while setting $\eta=0$ eliminates other error terms, it also exacerbates the first term of bias error, potentially making it the most significant error contributor. Consequently, there exists a trade-off in choosing the step size.

The subsequent theorem presents a nearly matching lower bound.

Analogous to the Theorem 3.1, our lower bound also consists of the bias term and the variance term. It is noteworthy that our lower bound is tight with the upper bound in terms of variance term, differing only by absolute constants. Additionally, our lower bound closely matches the upper bound in terms of the bias term, with some differences arising from the following quantities

Specifically, $\hat{\Phi}_{1}^{m-1}$ here differs from ${\Phi}_{1}^{m-1}$ in Theorem 3.1 only by a factor of constants (i.e. $\alpha_{k}$ and $\beta_{k}$ defined in 2.3). The term $\|\mathbf{w}_{0}-\mathbf{w}^{*}\|_{(\mathbf{I}-\eta\mathbf{H}_{m})^{2N}}$ has a different subscript of $(\mathbf{I}-\eta\mathbf{H}_{m})^{2N}$ compared to that of the upper bound. Nevertheless, it can be regarded as a part of the projection accumulation $\bm{\Gamma}_{1}^{M}$ that exists in the subscript of both results simultaneously.

More importantly, we show that the upper and lower bounds converge, ignoring constant factors, under the conditions

which can be satisfied that the signal-to-noise ratios $\|\mathbf{w}_{0}-\mathbf{w}^{*}\|_{\mathbf{U}_{k_{m}^{*}}}^{2}/\sigma^{2}$ is bounded and the step size is appropriate small.

Building on Theorem 3.1 and Theorem 3.2, we aim to offer a more comprehensive understanding of our findings from three key perspectives: 1) Technical Understanding Under Simplified Cases; 2) Comparison with Existing Work; 3) The Impact of Task Ordering and Parameters on Forgetting.