Unleashing the Power of Multi-Task Learning: A Comprehensive Survey Spanning Traditional, Deep, and Pretrained Foundation Model Eras

The increasing popularity of MTL over the past few decades is evident in Fig. 3, which displays the trend in the number of papers associated with “allintitle: ‘multitask learning’ OR ‘multi-task learning’ ” as a keyword search, according to data from Google Scholar¹¹1https://scholar.google.com.

As the name suggests, MTL is a subfield of ML where multiple tasks are jointly learned. In this manner, we hope to leverage useful information across these related tasks and break from the tradition of performing different tasks in isolation. In Single-Task Learning (STL), data specific to the task at hand is the only source to couch a learner. However, MTL can conveniently transfer extra knowledge learned from other tasks. The essence of MTL is to exploit consensual and complementary information among tasks by combining data resources and sharing knowledge. This sheds light on a better learning paradigm that can reduce memory burden and data consumption, and improve training speed and testing performance. For instance, learning the monocular depth estimation (scaling the distance to the camera) (eigen2014depth) and semantic segmentation (assigning a class label to every pixel value) (fu1981survey) simultaneously in images is beneficial since both tasks need to perceive meaningful objects. MTL has become increasingly ubiquitous as experimental and theoretical analyses continue to validate its promising results. For example, using Face ID to unlock an iPhone is a typical but imperceptible MTL application that involves simultaneously locating the user’s face and identifying the user. In general, multitasking occurs when we attempt to handle two or more objectives during the optimization stage in practice.

Consequently, MTL exists everywhere in ML, even when performing STL with regularization. This can be understood as having one target task and an additional artificial task of human preference, such as learning a constrained model via $\ell_{2}$ regularizer or a parsimonious model via $\ell_{1}$ regularizer. These hypothesis preferences can serve as an inductive bias to enhance an inductive learner (caruna1993multitask). In the early exploration of MTL (caruana1997multitask), the extra information that the involved tasks provide is regarded as a domain-specific inductive bias for the other tasks. Since collecting training signals from other tasks is more practical than acquiring inductive bias from model design or human expertise, we can thus empower any ML models via this MTL paradigm.

Having established a formal definition of MTL grounded in fundamental ML elements, a thorough understanding can be achieved by analytically comparing it with related domains. These include Transfer Learning (TL), Meta-Learning, and In-Context Learning (ICL), among others. This comparison not only clarifies the distinct characteristics of MTL but also situates it within the broader context of these interconnected fields.

MTL can be motivated from the following five perspectives with different benefits: cognitive/social psychology, data augmentation, learning efficacy, real-world scenarios, and learning theory.

• •

  Psychologically, humans are inherent with flexible adaptability to new problems and settings, as the human learning process can transfer knowledge from one experience to another (national2000people). Therefore, MTL is inspired by simulating this process to empower a model with the potentiality of multitasking. Coincidentally, another example of this knowledge transfer happens among organizations (argote2000knowledge). It is proved that organizations with more effective knowledge transfer are more productive and likely to survive than those with less. These prior successes of transfers or mutualizations in other areas encourage the joint learning of tasks in ML (caruana1997multitask).

• •

  In the pre-big data era, real-world problems were usually represented by small but high-dimensional datasets ($\#$ samples$<\#$ features). This data bottleneck forces early methods to learn a sparse-structured model, which always leads to a parsimonious solution to a problem with insufficient data. However, the MTL emerged to aggregate labeled data from different domains or tasks to enlarge the training dataset against overfitting.

• •

  The pursuit of efficiency and effectiveness is also one of the motivations. MTL can aggregate data from different sources together, and the joint training process of multiple tasks can save both computation and storage resources. In addition, the potential of performance enhancement makes it popular in research communities. In brief, universal representations for any tasks can be learned from multi-source data, and benefit all tasks in terms of both the learning cost and performance.

• •

  Motivated by the majority of real-world problems naturally being multimodal or multitasking, MTL is proposed to remedy the suboptimal achieved by STL that only models parts of the whole problem separately. For example, predicting the progression of Alzheimer’s Disease (AD) biomarkers for Mild Cognitive Impairment (MCI) risk and clinical diagnosis is simultaneously based on multimodal data such as computed tomography (CT), Magnetic Resonance Imaging (MRI), and Positron Emission Tomography (PET) (jie2015manifold; kwak2018multi; chen2022machine). Autonomous driving, another example, also involves multiple subtasks to calculate the final prediction (yang2018end; chowdhuri2019multinet), including the recognition of surrounding objects, adjustments to the fastest route according to the traffic conditions, the balance between efficiency and safety, etc.

• •

  From the perspective of learning theory, bias-free learning is proved to be impossible (mitchell1980need), so we can motivate the MTL by using the extra training signals for related tasks. Generally, MTL is one of the ways to achieve inductive transfer via multitasking assistance, which improves both learning speed and generalization. Specifically, during the process of the combined training of multiple tasks, some tasks can be provided inductive bias from other related tasks, and these stronger inductive biases (compared with universal regularizers, e.g., $\ell_{2}$) enable the knowledge transfer and yield more generalization abilities on a fixed training dataset. In other words, task-related biases make a learner prefer hypotheses that can explain more than one task and prevent specific task from overfitting.

In this section, we explore three key mechanisms – regularization, inductive bias, and feature sharing – shedding light on how MTL operates to achieve enhanced performance across multiple tasks.

Existing Surveys. ruder2017overview is a pioneering survey in MTL, offering a broad overview of MTL and focusing on advances in deep neural networks from 2015 to 2017. thung2018brief reviews MTL methods from a taxonomy perspective of input-output variants, mainly concentrating on traditional MTL prior to 2016. These two reviews can be complementary materials to each other. vafaeikia2020brief is an incomplete survey that briefly reviews recent deep MTL approaches, particularly focusing on the selection of auxiliary tasks for enhanced learning performance. crawshaw2020multi presents the well-established and advanced MTL methods before 2020 from the perspective of applications. vandenhende2021multi provides a comprehensive review of deep MTL in dense prediction tasks, which generate pixel-level predictions such as in semantic segmentation and monocular depth estimation. zhang2021survey first give a comprehensive overview of MTL models from the taxonomy of feature-based and parameter-based approaches, but with limited inclusion of deep learning (DL) methods. Notably, all these surveys overlook the development of MTL in the last three or four years, named the era of large PFMs (bommasani2021opportunities; zhou2023comprehensive), exemplified by the GPT-series models (radford2018improving; radford2019language; brown2020language; openai2023gpt4).

Roadmap. This survey adopts a well-organized structure, distinguishing it from its predecessors, to demonstrate the evolutionary journey of MTL from traditional methods to DL and the innovative paradigm shift introduced by PFMs, as shown in Fig. 1. In § 2.1, we provide a comprehensive summary of traditional MTL techniques, including feature selection, feature transformation, decomposition, low-rank factorization, priori sharing, and task clustering. Moving forward, § 2.2 is devoted to exploring the critical dimensions of deep MTL methodologies, encompassing feature fusion, cascading, knowledge distillation, cross-task attention, scalarization, multi-objective optimization (MOO), adversarial training, Mixture-of-Experts (MoE), graph-based methods, and NAS. The recent advancements in PFMs are introduced in § 2.3, categorized based on task-generalizable fine-tuning, task promptable engineering, as well as task-agnostic unification. Additionally, we provide a concise overview of the miscellaneous aspects of MTL in § 3. § 4 provides valuable resources and tools to enhance the engagement of researchers and practitioners with MTL. Our discussions and future directions are presented in § 5, followed by our conclusion in § 6. The goal of this review is threefold: 1) to provide a comprehensive understanding of MTL for newcomers; 2) to function as a toolbox or handbook for engineering practitioners; and 3) to inspire experts by providing insights into the future directions and potentials of MTL.

To establish a unified formulation, we start the review of traditional methods by defining a common framework. The notations for subsequent discussions are summarized in Table 3. Building upon this, we initiate our discussion with multiple standard regression models for each task as a paradigm. The weights of these homogeneous models can be arranged into one weight matrix, catalyzing a series of MTL studies through matrix regularization techniques in the traditional era. We denote by $\{(\boldsymbol{X}^{(t)},\boldsymbol{y}^{(t)})\}_{t=1}^{T}$ our dataset across $T$ tasks. For each task indexed by $t={1,2,\cdots,T}$, we are given $N_{t}$ samples with $D$ features, i.e., $\boldsymbol{X}^{(t)}\in\mathbb{R}^{N_{t}\times D}$, and the corresponding response values $\boldsymbol{y}^{(t)}\in\mathbb{R}^{N_{t}}$.

The single-task setting of these multiple linear regression problems is

where $\boldsymbol{w}^{(t)}\in\mathbb{R}^{D}$ for any $t\in\{1,\cdots,T\}$, $\epsilon^{(t)}\sim\mathcal{N}(0,\sigma_{t}^{2}\mathbb{I})$ is the error term independent of $\boldsymbol{X}^{(t)}$, and $\sigma_{t}$ is determined by the system state for $t$-th task. Each model is separately learned from independent samples $\{({\boldsymbol{x}_{1}^{(t)}}^{\top},y_{1}^{(t)}),\cdots,({\boldsymbol{x}_{N_{t}}^{(t)}}^{\top},y_{N_{t}}^{(t)})\}$.

A trivial simplification of the above linear regressions is that all tasks maintain the same feature size $D$, thus leading to a natural idea of stacking weight vectors for these tasks: $\boldsymbol{W}=[\boldsymbol{w}^{(1)},\cdots,\boldsymbol{w}^{(T)}]\in\mathbb{R}^{D\times T}$, where the matrix-based regularizers come into play. To estimate as $\boldsymbol{W}$, the MTL method minimizes the objective function:

where we consider the weights of multiple models, i.e., $\boldsymbol{W}$, as a union, and denote by $\boldsymbol{w}^{t}$ the $t$-th column of $\boldsymbol{W}$. Normally, an identical loss function, e.g., mean squared error (MSE), is always applied to $\{{\mathcal{L}}^{(t)}\}_{t=1}^{T}$, which originates from the $i.i.d.$ assumption of $\{\epsilon^{(t)}\}_{t=1}^{T}$. To capture task relatedness from the Assumption 1 that multiple models are similar to each other, $\Omega$ is designed to take various regularization forms in traditional MTL. The overview of regularization techniques used in the traditional ML era for MTL (will be discussed in the following subsections) is presented in Table 2.1.