Stochastic First-Order Methods with Non-smooth and Non-Euclidean Proximal Terms for Nonconvex High-Dimensional Stochastic Optimization ^††thanks: Research is supported from HKU-IDS start-up fund; and Guangdong Province Fundamental and Applied Fundamental Research Regional Joint Fund, 2022B1515130009.

In this paper, we consider a stochastic optimization (SO) problem formulated as below:

where $f:{\mathbb{R}}^{d}\to{\mathbb{R}}$ and $F:{\mathbb{R}}^{d}\times\Omega\to{\mathbb{R}}$. $(\Omega,\mathcal{F},\mathbb{P})$ is the probability space. $X$ is a closed and convex set. Denote by ${\mathbf{x}}^{*}\in{\mathbb{R}}^{d}$ an optimal solution to (1). We do not have explicit assumption on problem dimension $d$, but we are particularly interested in a large $d$. We assume that the expectation ${\mathbb{E}}\left[F({\mathbf{x}},\zeta)\right]=\int_{\Omega}F({\mathbf{x}},\zeta)\,\text{d}\mathbb{P}(\zeta)$ is well-defined and finite-valued for every ${\mathbf{x}}\in{\mathbb{R}}^{d}$ and that it is possible to generate independent and identically distributed realizations, $\zeta_{1},\,\zeta_{2},....,$ of the random vector $\zeta$. For this problem, we are particularly interested in solution algorithms when $f({\mathbf{x}})$ is smooth but potentially nonconvex.

In view of stochasticity and potential nonconvexity of the problem, we aim to compute an stationary point with small expected value of first-order optimality residual.

Many stochastic first-order methods (S-FOMs) apply to smooth and nonconvex optimization problems in the form of (1). These methods date back to robbins1951stochastic ; chung1954stochastic ; sacks1958asymptotic , and have been much studied (e.g., by polyak1990new ; polyak1992acceleration ; nemirovski2009robust ; lan2012optimal ; ghadimi2013stochastic ; rosasco2019convergence ; devolder2011stochastic ; ghadimi2012optimal ; chambolle2018stochastic ; chambolle2011first ; bach2013non ; metel2019simple ; xu2019stochastic ; nitanda2017stochastic ; davis2019stochastic ; li2019ssrgd ; nguyen2017sarah ; wang2019spiderboost ; fang2018spider ; pham2020proxsarah ; horvath2020adaptivity , among others). In terms of the stochastic first-order oracle or sample, most existing S-FOMs incur analytical complexities that grow polynomially in dimensionality.

Indeed, for S-FOMs in smooth and nonconvex optimization, (See ghadimi2013stochastic ; ghadimi2016mini ; pham2020proxsarah and references therein.) Here $\sigma^{2}$ is defined as an upper bound on ${\mathbb{E}}[\|\nabla F({\mathbf{x}},\zeta)-\nabla f({\mathbf{x}})\|^{2}]$. In the worst case, $\sigma^{2}$ is linear in $d$ because

where the right-hand side is often $\mathcal{O}(d)$. This then leads to an analytical complexity in terms of SFO as following:

More recent results such as by fang2018spider ; wang2019spiderboost further enhances (3) into:

However, with the ever-increasing demand for more sophisticated and comprehensive models, (SO) problems of excessively high dimensionality are rapidly emerging. Based on the existing theories, the SGD and many other S-FOM alternatives may have to perform at least hundreds of millions of iterations or stochastic first-order oracles to achieve the desired solution quality. Without fundamental innovations vis-à-vis high dimensionality, the SO problem (1) will become extremely difficult to solve in the near future.

Dimension-insensitive S-FOMs have been discussed by nemirovski2009robust ; lan2020first ; zhang2018convergence , especially about mirror descent algorithm (MDA) and its analysis. The authors of agarwal2012stochastic base their algorithm on successively solving a series of $\ell_{1}$-regularized optimization problems using Nesterov’s dual averaging algorithm and obtain the dimension insensitive result. However, agarwal2012stochastic ; nemirovski2009robust ; lan2020first focused on convex SO problems. zhang2018convergence extended the MDA to nonconvex settings and showed a complexity of

This bound is verifiable to be much better than (3) and (4) in terms of the dependence on $d$ ($\mathcal{O}(\log d)$ instead of $\mathcal{O}(d)$ because of the infinity norm) under certain mild assumptions on $\zeta$ (e.g. sub-Gaussian). To our knowledge, this is the only dimension-insensitive algorithm available in the nonconvex regime from the literature. However, the result is lacking in two key respects: (i). The authors assume that the distance generating function should be continuously differentiable. This is somewhat restrictive and does not allow non-smooth alternatives to enhance the algorithm. (ii). The term $\epsilon^{-4}$ in (5) can be further improved.

In this work, we propose dimension-insensitive stochastic first-order methods (DISFOMs) that allow non-Euclidean and non-smooth distances to construct the proximal term. Throughout this work, the residual function in Def. 1 is naturally defined via the following:

We point out that although the residual function is given in $\|\cdot\|_{\infty}$, the dimension-insensitive result is not trivial. In fact, the residual function is stronger than a residual function discussed in literature (zhang2018convergence ) and defined in $\|\cdot\|_{1}$:

where ${\rm prox}_{\lambda f}({\mathbf{x}})$ is the proximal operator w.r.t. Bregman divergence. Detailed discussions are in Section 3.

Under mild assumptions, we show that DISFOM using minibatches to estimate the gradient enjoys sample complexity of $\mathcal{O}\left((\log d)/\epsilon^{4}\right)$ to obtain an $\epsilon$-stationary point. Furthermore, we prove that DISFOM employing variance reduction can sharpen this bound to $\mathcal{O}\left((\log d)^{2/3}/\epsilon^{10/3}\right)$, which further improves over (5) in terms of dependence on both $d$ and $\epsilon$, and perhaps leads to the best-known sample complexity result in terms of $d$ in the nonconvex regime. We discuss two choices of the non-smooth distance functions, both of which allow closed-form solutions to the proximal step. Numerical experiments are conducted to illustrate the dimension-insensitive property of the proposed frameworks and show comparison with other popular algorithms.

In Section 2 we clarify some mild assumptions used in this article. We discuss the residual function defined in (6) and how it compares with others in Section 3. In Section 4, we formally introduce the DISFOMs. Theoretical analysis are provided in Section 5. Section 6 includes numerical experiments. A final conclusion can be found in Section 7.

We consider problem 1 where the following assumptions are satisfied:

By Assumption 2, we immediately have

By Assumption 3, we have that $\nabla f({\mathbf{x}})$ is also Lipschitz continuous with constant $L$. Therefore, the following inequality holds:

Assumption 1 stipulates that the oracle accessible here is the tractable computation of a stochastic gradient. Assumption 2 assumes that each component of the stochastic gradient is unbiased and its error is a sub-Gaussian random variable with parameter $\sigma_{\infty}$. Assumption 3 states that function $\nabla F(\cdot,\zeta)$ is Lipschitz smooth. All these assumptions above are standard or equivalent to the common conditions in the literature of stochastic approximation or S-FOMs. For example, Assumptions 1 and 2 are imposed by nemirovski2009robust ; ghadimi2013stochastic ; lan2020first (though some important results therein do not rely on Assumption 2). Note that if a random variable has a bounded support, it is sub-Gaussian. Assumption 3 or its slightly weaker version (inequality holds after taking expectation on the left) is necessary for the discussions by wang2019spiderboost ; nguyen2017sarah ; pham2020proxsarah .

It is true that the dimension-insensitive complexity result is dependent on the criterion used to measure optimality. If the criterion is weak, then the dimension-insensitive result could be meaningless. We demonstrate that the dimension-insensitive result associated with the residual function (6) is nontrivial by comparing $r({\mathbf{x}})$ in (6) with the following two discussed in zhang2018convergence :

where

and $D_{h}(\cdot,\cdot)$ is the Bregman divergence induced by 1-stongly convex continuously differentiable distantce generating function $h(\cdot)$ w.r.t $\|\cdot\|_{1}$, i.e.,

The residual function $r_{1}({\mathbf{x}})$ is a natural generalization from the one used in davis2018stochastic to $\ell_{1}$-norm. In fact, $r_{2}({\mathbf{x}})$ is stronger than $r_{1}({\mathbf{x}})$ since

We point out that $r({\mathbf{x}})$ is stronger than $r_{2}({\mathbf{x}})$. In fact, if $f$ is $\rho$-relatively weakly convex ($f({\mathbf{x}})+\rho h({\mathbf{x}})$ is convex for such $h(\cdot)$ and some $\rho>0$) and $\lambda\in(0,\rho^{-1})$ so that (11) is well-defined, we have the following statement.

We start from a general framework of a stochastic first-order method as follows.

Now we discuss the details of the proximal projection operator $P_{X}^{k}(\cdot)$ and gradient estimate $G^{k}$ that can achieve the dimension-insensitive property. In particular, the proximal projection operator $P_{X}^{k}(\cdot)$ is defined as

where $\phi:{\mathbb{R}}^{d}\to{\mathbb{R}}$ is a convex function and we suppose that calculation of (19) is easy and exact when $X={\mathbb{R}}^{d}$ (we will discuss the specific form of $\phi$ later) or when $\phi\equiv 0$. Moreover, suppose that $\phi(0)=0$. We could employ ADMM (see Appendix B) to resolve (18)(19) to accuracy level ${\hat{\epsilon}}>0$ so that the following conditions hold: $\exists\xi^{k+1},\Gamma^{k+1},{\mathbf{y}}^{k+1}$, s.t.,

In particular, when $X={\mathbb{R}}^{d}$, by assumption we have that (19) can be solved easily and exactly, and (20a)-(20c) hold true with ${\hat{\epsilon}}=0$. In fact, when $X={\mathbb{R}}^{d}$ we have

If $X\neq{\mathbb{R}}^{d}$, we point out that ADMM is also fast in resolving (19) under the assumption that $X$ is a polyhedron and $\partial\phi({\mathbf{x}})$ is a piecewise linear multifunctions ($Gr(\partial\phi({\mathbf{x}}))\triangleq\{({\mathbf{x}},{\mathbf{y}})\mid{\mathbf{y}}\in\partial\phi({\mathbf{x}})\}$ is the union of finitely many polyhedra). Then by yang2016linear , ADMM converges in linear rate in 2-norm. That means to obtain (20a)-(20c), the iterations of ADMM is at most $\mathcal{O}\left(\log(\sqrt{d}/{\hat{\epsilon}})\right)$. Considering that each step of ADMM is cheap, additional computation introduced by $P_{X}^{k}(\cdot)$ is incremental.