An Improved Finite-time Analysis of Temporal Difference
Learning with Deep Neural Networks

For large-scale reinforcement learning (RL) problems, appropriate parameterization of the Q function is crucial for better scalability of the TD algorithms. Common examples include linear (Tesauro et al., 1995), general smooth nonlinear (Maei et al., 2009), and neural network (Mnih et al., 2013) function approximations. However, it is well known that the naive extension of TD learning and Q-learning algorithms can diverge under the general function approximation Tsitsiklis & Van Roy (1996). To encourage convergence, numerous variants of TD and Q-learning have been proposed, including Least-squares TD (LSTD) (Bradtke & Barto, 1996; Boyan, 2002) and gradient TD (GTD) (Sutton et al., 2009a, b), to name a few.

The applications of neural network function approximation have witnessed huge empirical success in many real-world tasks, including Deep Q-network (DQN) algorithms (Mnih et al., 2013; Van Hasselt et al., 2016), policy improvement method (Sutton et al., 1999), trust region policy optimization (Schulman et al., 2015) and the actor-critic algorithms (Konda & Tsitsiklis, 1999; Lillicrap et al., 2015; Fujimoto et al., 2018), etc. However, due to the analysis difficulties brought by the function approximation, a significant gap exists between the empirical success and the theoretical understanding of these algorithms. Hence analyzing the convergence and sample complexity of TD learning and Q-learning under various Q function parameterizations has always been an active topic in the RL community during the past decades.

Early works focus on the asymptotic convergence of the algorithms with tabular or linear function approximation. For the tabular (stochastic) TD or Q learning method, Jaakkola et al. (1993) established the asymptotic convergence for the first time. Later on, the asymptotic convergence of algorithms with linear function approximation has been extensively discussed using ODE-based methods, see e.g. Tsitsiklis & Van Roy (1996); Perkins & Pendrith (2002); Borkar (2009). Meanwhile, in contrast to the convergent results for RL algorithms under the tabular or linear settings, TD with nonlinear function approximation is known to diverge in general Tsitsiklis & Van Roy (1996); Brandfonbrener & Bruna (2019). To overcome this issue, Maei et al. (2009) proposed to optimize the Mean Squared Projected Bellman Error (MSPBE) via a gradient-based algorithm. Due to the problem nonconvexity, only asymptotic convergence to stationary points can be guaranteed.

More recently, benefiting from the improved techniques for analyzing stochastic optimization algorithms, there has been a growing number of research on providing finite-time analysis for TD and Q-learning algorithms with function approximations.

For linear function approximation, the non-asymptotic results of TD learning and its variants are relatively well-understood, including TD Bhandari et al. (2018); Dalal et al. (2018); Zou et al. (2019), gradient TD Dalal et al. (2018); Touati et al. (2018); Liu et al. (2020a), and Least-Squares TD Lazaric et al. (2010); Prashanth et al. (2014); Tagorti & Scherrer (2015), etc. In particular, Bhandari et al. (2018) established the first finite-time analysis of linear Q-learning under both i.i.d. sampling and Markovian sampling settings.

For neural network function approximation, which is directly related to this paper, we provide a more detailed discussion. Based on the recent advances in the understanding of optimizing ReLU network Jacot et al. (2018); Du et al. (2018); Allen-Zhu et al. (2019a, b); Cao & Gu (2019, 2020), a few recent works have successfully developed the finite-time analysis of the neural TD and neural Q-learning algorithms, as long as the Q network is sufficiently wide. Let $Q^{*}$ be the true action-value function and let $Q(s,a;\boldsymbol{\theta})$ denote the action-value function parameterized by a neural network with weights $\boldsymbol{\theta}$, at any state action pair $(s,a)$. Then we aim to find some $\epsilon$-optimal parameter $\bar{\boldsymbol{\theta}}$ such that $\mathbb{E}\big{[}(Q(s,a;\bar{\boldsymbol{\theta}})-Q^{*}(s,a))^{2}\big{]}\leq\epsilon+\epsilon_{\mathcal{F}}$, where the expectation is taken over the possible randomness in the output $\bar{\theta}$ as well as the distribution over the state-action pairs $(s,a)$, and $\epsilon_{\mathcal{F}}$ is the optimal approximation error of the parameterization function class. In (Xu & Gu, 2020), a neural Q-learning algorithm with a general $L$-layer ReLU network is analyzed, and an $\tilde{\mathcal{O}}(\epsilon^{-2})$ sample complexity is guaranteed given that the network is sufficiently wide. In (Cai et al., 2023), the authors studied both the neural TD learning and neural Q-learning algorithms for minimizing the MSPBE for policy evaluation and policy optimization, respectively. For policy evaluation, the $Q^{*}$ in the definition of an $\epsilon$-optimal solution is defaulted as $Q^{\pi}$ with $\pi$ being the policy to be evaluated. For both cases, an $\tilde{\mathcal{O}}(\epsilon^{-2})$ sample complexity is guaranteed for wide two-layer ReLU networks. In (Sun et al., 2022), an $\tilde{O}(\epsilon^{-\frac{2}{2-\alpha}})$ complexity has been achieved by an adaptive neural TD algorithm with multi-layer ReLU networks, where $\alpha\in(0,1]$ is a constant that characterizes the sparsity and decay rate of the stochastic semi-gradients. However, without additional assumption, only an $\tilde{O}(\epsilon^{-2})$ complexity with $\alpha=1$ can be theoretically guaranteed. Finally, for policy evaluation problems, there are also several works that aim at reducing the width of the over-parameterized Q networks in the existing works (Tian et al., 2022; Cayci et al., 2023). In terms of complexity, both of them requires $\tilde{\mathcal{O}}(\epsilon^{-2})$ samples to obtain an $\epsilon$-optimal solution.

Despite the fact that existing analysis of the neural TD or neural Q-learning algorithms merely provides the $\tilde{\mathcal{O}}(\epsilon^{-2})$ sample complexity under various settings, an $\tilde{\mathcal{O}}(\epsilon^{-1})$ sample complexity should be expected. In fact, a double-loop fitted Q-iteration (FQI) method (Fan et al., 2020) and its single-loop Gauss-Newton variant (Ke et al., 2023) can achieve an $\tilde{\mathcal{O}}(\epsilon^{-1})$ sample complexity is obtained for two-layer Q networks. Let $\mathcal{T}$ be the Bellman (optimality) operator, then the FQI method repeatedly solves a nonlinear least square subproblem to obtain the next iteration: $\boldsymbol{\theta}_{k+1}\approx\mathop{\mathrm{argmin}}_{\boldsymbol{\theta}\in\Theta}\mathbb{E}\big{[}(Q(s,a;\boldsymbol{\theta})-\mathcal{T}Q(s,a;\boldsymbol{\theta}_{k}))^{2}\big{]}$. Compared to the single-loop neural TD or neural Q-learning method that takes only one sample (or a mini-batch) to update the weights of Q networks, the update scheme of FQI requires repeatedly solving a subproblem to sufficiently high accuracy to enable convergence, which makes it inefficient and less favorable in practice. Therefore, we would like to raise a question: {mdframed}[leftmargin=1cm,rightmargin=1cm, backgroundcolor=gray!10] Can we improve the existing analysis of the neural temporal difference learning algorithm and obtain an $\tilde{\mathcal{O}}(\epsilon^{-1})$ sample complexity under general multi-layer Q neural networks?

To answer this question, we revisit the convergence analysis of the neural TD learning or Q-learning algorithms under the non-i.i.d. Markovian observations where a general $L$-layer neural network is used for Q function parameterization. By proposing a new subspace analysis technique, under suitable conditions, we derive a brand new $\tilde{\mathcal{O}}(\epsilon^{-1})$ sample complexity for neural TD learning or Q-learning, improving the state-of-the-art $\tilde{\mathcal{O}}(\epsilon^{-2})$ sample complexity in the existing works. Our contributions are summarized as follows.

• •

  Under the non-i.i.d. Markovian sampling setting, we derive an $\tilde{\mathcal{O}}(\epsilon^{-1})$ sample complexity for both neural TD learning and Q-learning methods under the multi-layer network approximation for Q functions. Our result also improves the best known $\tilde{\mathcal{O}}(\epsilon^{-2})$ sample complexity in the existing works.

• •

  Based on our newly developed techniques, we further provide a finite-sample analysis for a minimax neural Q-learning algorithm that solves two-player zero-sum Markov games. An $\tilde{\mathcal{O}}(\epsilon^{-1})$ sample complexity is obtained under the non-i.i.d. Markovian sampling setting.

Technically, the subspace analysis approach that we propose to establish the $\tilde{O}(\epsilon^{-1})$ sample complexity is by itself of independent interest. We believe this technique can potentially be applied to linear Q-learning algorithms and linear Actor-Critic algorithms without requiring the positive definiteness assumption of the feature covariance matrix (Bhandari et al., 2018; Zou et al., 2019; Barakat et al., 2022), while maintaining the $\tilde{O}(\epsilon^{-1})$ complexity.

In summary, we provide a comprehensive comparison between our work and the most related works in their respective settings and sample complexity in Table 1. Our work establishes an optimal sample complexity analysis within a broader contextual framework.

We consider the infinite-horizon discounted Markov decision process (MDP), which is denoted as $\mathcal{M}=(\mathcal{S},\mathcal{A},\mathbb{P},r,\gamma)$. We consider a general state space $\mathcal{S}$ and a finite action space $\mathcal{A}$. At any state $s\in\mathcal{S}$, if the agent takes an action $a\in\mathcal{A}$, it will receive a reward $r(s,a)\in[-R_{\max},R_{\max}]$ and transition to the next state $s^{\prime}\in\mathcal{S}$ with probability $\mathbb{P}(s^{\prime}|s,a)$. We call $r$ the reward function and $\mathbb{P}$ the transition kernel. Let $\gamma\in(0,1)$ be a discount factor, then an MDP aims to find a sequence of actions $\{a_{t}\}_{t\geq 0}$ to maximize the expected and discounted cumulative reward $\mathbb{E}\big{[}\sum_{t=0}^{\infty}\gamma^{t}\!\cdot r(s_{t},a_{t})|s_{0}\sim\mu\big{]}$, where $\mu$ is the distribution of the initial state $s_{0}$.

Let $\Delta_{\mathcal{A}}$ denote the set of all probability distributions over the action space $\mathcal{A}$, and let a policy $\pi:\mathcal{S}\mapsto\Delta_{\mathcal{A}}$ be a mapping that returns a probability distribution $\pi(\cdot|s)\in\Delta_{\mathcal{A}}$ given any state $s\in\mathcal{S}$. If an agent follows a policy $\pi$, then at any state $s_{t}$, it will act by sampling an action $a_{t}\sim\pi(\cdot|s_{t})$. Therefore, the action-value function (Q-function) under the policy $\pi$ is

for $\forall(s,a)\in\mathcal{S}\times\mathcal{A}$, where all actions except $a_{0}$ are sampled according to $\pi$. For any mapping $Q:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$, let the Bellman operator $\mathcal{T}^{\pi}$ be

Then $\mathcal{T}^{\pi}$ is a $\gamma$-contraction under the infinity norm and $Q^{\pi}$ is the unique solution to the fixed-point equation $Q=\mathcal{T}^{\pi}Q$ (Bertsekas, 2012). If the Q function is parameterized by some function $Q(s,a;\boldsymbol{\theta})$ to gain better scalability for large-scale RL problems, popular approaches for finding a good $\boldsymbol{\theta}$ include minimizing the the Mean-Squared Bellman Error (MSBE):

and minimizing the Mean-Squared Projected Bellman Error (MSPBE):

where $\Theta$ is a feasible domain of the parameter $\boldsymbol{\theta}$, $\mu$ is some distribution over state action pairs, and $\Pi_{\mathcal{F}}$ is the projection onto some function class $\mathcal{F}$. Typical choices of $\mathcal{F}$ includes the Q function parameterization class itself $\mathcal{F}:=\{Q(\cdot;\boldsymbol{\theta}):\theta\in\Theta\}$ (Maei et al., 2009), and some local linearization of the parameterization function class (Cai et al., 2023).

In this paper, we study the neural temporal difference learning method where the action-value function is parameterized by some multi-layer neural network. Let us define a feedforward neural network by the following recursion:

where $\boldsymbol{W}_{1}\in\mathbb{R}^{m\times d}$, $\boldsymbol{W}_{l}\in\mathbb{R}^{m\times m}$ for $2\leq l\leq L$ are the weight matrices of the network, $\sigma(\cdot)$ is an activation function, and the input is a feature map $\boldsymbol{x}^{(0)}=\phi(s,a)\in\mathbb{R}^{d}$ for any state action pair $(s,a)$. For simplicity of notation, we write $\boldsymbol{x}=\boldsymbol{x}^{(0)}$, then $Q(s,a)$ is parameterized by

where the parameter $\boldsymbol{\theta}=\left(\mbox{Vec}(\boldsymbol{W}_{1});\cdots;\mbox{Vec}(\boldsymbol{W}_{L})\right)$ denotes the collection of all weight matrices, and $\boldsymbol{b}$ is given by a random initialization. $\mbox{Vec}(\cdot)$ stands for the vetorization operator that reshapes a matrix to a column vector by stacking its columns one by one and the “;” separator in $\boldsymbol{\theta}$ stands for the vertical stacking of the elements. That is, we reshape $\boldsymbol{\theta}$ to a long column vector for the notational convenience in later discussion.

Assumption 2.1 indicates that our results below are not based on the popular ReLU activation function. However, we primarily focus on some twice-differentiable activation functions (such as Sigmoid, ELU, GeLU, etc.), which are smooth approximations of the ReLU function and are frequently utilized in practical problems (Devlin et al., 2018; Godfrey, 2019). Such a setup aligns with (Liu et al., 2020b), and provides a $\mathcal{O}(m^{-\frac{1}{2}})$-smooth property for the neural Q-function.

Let $\boldsymbol{\theta}^{0}=\left(\mbox{Vec}(\boldsymbol{W}_{1}^{0});\cdots;\mbox{Vec}(\boldsymbol{W}_{L}^{0})\right)$ be the initial solution. For each $l$, we initialize the weights of $\boldsymbol{W}_{l}^{0}$ element-wise from a normal distribution $\mathcal{N}(0,1)$ and each element of $\boldsymbol{b}$ is drawn uniformly from $\{-1,+1\}$. The parameter $\boldsymbol{b}$ will not be optimized during training. For regularity purpose, we would like to restrict the iterations to a bounded set around $\boldsymbol{\theta}^{0}$, which is defined as

In each iteration $t$, the neural Q-learning algorithm obtains a sample of state-action-reward-transition tuple $(s_{t},a_{t},r_{t},s_{t+1},a_{t+1})$ and computes the TD error by

with $\boldsymbol{x}_{t}=\phi(s_{t},a_{t}),\boldsymbol{x}_{t+1}=\phi(s_{t+1},a_{t+1}).$ Then a projected stochastic semi-gradient step is performed to update the weight matrices:

with

We formally describe the neural TD learning method in Algorithm 1.

One remark is that, under the non-i.i.d. Markovian sampling setting, the agent is only able to generate a trajectory of samples following some given learning policy $\pi$, which is very common in the offline RL (Wu et al., 2019; Levine et al., 2020; Kostrikov et al., 2021) where the data trajectories are generated by some learning policy.

In later sections, we will revisit the Algorithm 1 and design a novel subspace analysis technique for this method and achieve an improved sample complexity of $\tilde{\mathcal{O}}(\epsilon^{-1})$. Moreover, by replacing the TD error induced by the Bellman operator (5) with TD error induced by the Bellman optimality operator: $\Delta_{t}=Q(\boldsymbol{x}_{t};\boldsymbol{\theta}^{t})-\big{(}r_{t}+\gamma\max_{b\in\mathcal{A}}Q(s^{\prime},b;\boldsymbol{\theta}^{t})\big{)}$, Algorithm 1 can be reduced to the neural Q-learning method for finding optimal state-action value $Q^{*}$. Our analysis for neural TD learning can be extended to the neural Q-learning analogously and obtain the same $\tilde{\mathcal{O}}(\epsilon^{-1})$ sample complexity.