The $\mathsf{SMART}$ Approach to Instance-Optimal Online Learning

We consider a general online learning setting where at the beginning of each round $t\in[n]$, a policy $\mathsf{ALG}$ first plays an action $a_{t}\in\mathcal{A}$, following which, a loss function $\ell_{t}\mathrel{\mathop{:}}\mathcal{A}\to[0,1]$ is revealed, resulting in a loss of $\ell_{t}(a_{t})$. The regret is defined according to:

More generally, as in with bit prediction, we allow $\mathsf{ALG}$ to play in round $t$ a measure $w_{t}\in\Delta_{\mathcal{A}}$ (i.e., play $\{w_{t}\mathrel{\mathop{:}}\mathcal{A}\mapsto[0,1]|\sum_{a\in\mathcal{A}}w_{t}(a)=1\}$), resulting in an expected loss of $\sum_{a\in\mathcal{A}}w_{t}(a)\ell_{t}(a)$. For notational convenience, we henceforth use $(a_{t},\ell_{t}(a_{t}))$ for the action/loss, and reserve use of expectations for randomness in the algorithm and/or sequence.

We want to understand when is it possible to attain instance optimality as in Eq. (1) with respect to a given pair of algorithms. Ideally, we want the first to be optimal for stochastic instances, and the second to be minimax optimal; unfortunately however exact optimal policies are unknown except in simple settings. To this end, we make two amendments to our goal: First, for the stochastic optimal policy, we use $\mathsf{FTL}$; this is well defined in any online learning setting, and moreover, known to be optimal or near-optimal for a wide range of settings under minimal assumptions (Kotłowski, 2018). Second, instead of the minimax policy, we use as reference any policy $\mathsf{ALG}_{\mathsf{WC}}$ which has a known worst case regret bound $g(n)$. With these modifications in place, we have the following objective.

While the above guarantee is not truly instance-optimal in that we are comparing against a worst-case regret bound $g(n)$ for $\mathsf{ALG}_{\mathsf{WC}}$ rather than its performance on the instance $\ell^{n}$, the two are the same if $\mathsf{ALG}_{\mathsf{WC}}$ is minimax optimal and hence attaining equal regret on all loss sequences; recall this is true of $\mathsf{Cover}$ for binary prediction.

To realize the above goal, we propose the Switch via Monotone Adapted Regret Traces ($\mathsf{SMART}$) approach, which at a high level is a black-box way to convert design of instance-optimal policies into a simple optimal stopping problem. Our approach depends on just two ingredients: first, owing to the additive structure of online learning problems, we have that the minimax guarantee $g(k)$ above holds over any $k\in\mathbb{Z}$ and any (sub)sequence of $n$ loss functions; second, we show that $\mathsf{FTL}$ admits simple anytime regret estimator $\Sigma^{\mathsf{FTL}}_{\tau}$ (see Lemma 1) which is monotone and adapted (i.e., a function only of historical data). Using these two observations, we can reduce the task of minimizing regret to a version of the ‘ski-rental’ problem (Karlin et al., 1994; Borodin and El-Yaniv, 2005), as follows: we play $\mathsf{FTL}$ up to some stopping time $\tau$, and then switch to $\mathsf{ALG}_{\mathsf{WC}}$ for the remaining $n-\tau$ periods, resetting all losses to zero. This algorithm incurs a total regret bounded by $\Sigma^{\mathsf{FTL}}_{\tau}+g(n-\tau)$, and using ideas from competitive analysis, we get that there is a simple way to choose the stopping time $\tau$ to achieve an $e/(e-1)\approx 1.58$-competitive ratio guarantee with respect to the minimum between the regret of $\mathsf{FTL}$ and the worst case guarantee $g(n)$.

A highlight of our approach is the surprising simplicity of the algorithm and analysis, despite the strength of the instance optimality guarantee. In particular, our approach is modular, allowing one to plug in any $\mathsf{ALG}_{\mathsf{WC}}$ and corresponding worst case bound $g(n)$, thus letting us handle any online learning setting with known minimax bounds. This results in an entire family of instance optimal policies for settings such as predictions with experts and online convex optimization. Moreover, the approach is easy to extend to get more complex guarantees; as an example, if $\mathsf{ALG}_{\mathsf{WC}}$ is designed to get low regret for benign (i.e., ‘small-loss’) sequences $\ell^{n}$, then we show how to use $\mathsf{SMART}$ as a subroutine and achieve an instance optimal guarantee with respect to the regret of $\mathsf{FTL}$ and a small loss regret bound.

Finally, studying instance optimality also lets us understand the fundamental limits of best-of-both worlds algorithms. To this end, we provide a lower bound that shows our algorithm is nearly optimal in the competitive ratio. To the best of our knowledge, this is the first hardness result for best-of-both-worlds guarantees in online learning.

Note again that in binary prediction, $\mathsf{FTL}$ achieves the optimal pseudoregret under i.i.d. inputs, while $\mathsf{Cover}$ is the true minimax policy; thus, this is a fundamental lower bound on best-of-both-worlds guarantee in any online learning setting.

There have been many approaches towards combining stochastic and worst-case guarantees. As we discussed before, there is a large literature on best-of-both-worlds algorithms for both full and partial information settings (Wei and Luo, 2018; Bubeck et al., 2019; Zimmert and Seldin, 2019; Dann et al., 2023), and also more complex settings such as metrical task systems (Bhuyan et al., 2023) and control (Sabag et al., 2021; Goel et al., 2023). Another line of work (Rakhlin et al., 2011; Haghtalab et al., 2022; Block et al., 2022; Bhatt et al., 2023) considers smoothed analysis, where the worst-case actions of the adversary are perturbed by nature. A third approach (Bubeck and Slivkins, 2012; Lykouris et al., 2018; Amir et al., 2020; Zimmert and Seldin, 2019) interpolates between the stochastic and adversarial settings by considering most $\ell_{t}$ to be i.i.d., interspersed with a few adversarially chosen instances (corruptions). Finally, another line considers the data-generating distribution to come from a ball of specified radius around i.i.d. distributions (Mourtada and Gaïffas, 2019; Bilodeau et al., 2023). While all these approaches provide useful insights into the gap between average and worst-case guarantees, one can argue they are all imprecisely specified – given an instance $\{\ell_{t}\}_{t\in[n]}$ in hindsight, there is no clear sense as to which model best ‘explains’ the instance.

Our focus on instance optimality instead follows the approach of better understanding and shaping the per-sequence regret landscape. The origins of this approach arguably come from the seminal work of Cover (1966) for binary prediction (we discuss this in more detail in Section 3), with a later focus on better bounds for benign instances in general online learning (Auer et al., 2002b; Cesa-Bianchi et al., 2005; Hazan and Kale, 2010). More recently, a line of work (Koolen et al., 2014; Van Erven et al., 2015; Van Erven and Koolen, 2016; Gaillard et al., 2014) have studied designing policies that can adapt to different types of data sequences and achieve multiple performance guarantees simultaneously. The main idea is to use multiple learning rates that are weighted according to their empirical performance on the data. While the focus is still primarily on classifying instances based on when they are easier/harder to learn, some of the resulting guarantees have an instance-optimality flavor; for example, Van Erven and Koolen (2016) show how to simultaneously match the performance guarantee (in terms of certain variance bounds) attained by different learning rates in Hedge. Such approaches however need to understand their baseline algorithms in great detail, and use them in a ‘white-box’ way to get their guarantees.

In contrast, our approach fundamentally focuses on combining policies in a black-box way to get instance optimal outcomes. As we mention, this is similar in spirit to corralling bandit algorithms Agarwal et al. (2017); Pacchiano et al. (2020); Dann et al. (2023), as well as more recent work on online algorithms with predictions Bamas et al. (2020); Dinitz et al. (2022); Anand et al. (2022); however, as we mention above, these all get guarantees with respect to the loss of the reference algorithms, which is much weaker than our regret guarantees (though they do so in much more complex settings with partial information and/or state). To our knowledge, the only previous result which attains a comparable instance-optimality guarantee to ours is that of De Rooij et al. (2014) for the experts problem, where the authors propose the $\mathsf{FlipFlop}$ policy which interleaves $\mathsf{Hedge}$ (with varying learning rates) and $\mathsf{FTL}$ to obtain a regret guarantee similar to that of Corollary 1. In fact, their guarantee is stronger as $\mathsf{FlipFlop}$ is shown to be 5.64-competitive with respect to $\min\{\mathrm{Reg}(\mathsf{FTL},\ell^{n}),g(L^{*})\}$ where $g(L^{*})\leq\sqrt{L^{*}\log m}$ (De Rooij et al., 2014, Corollary 16). However, while $\mathsf{FlipFlop}$ depends on a clever choice of learning rates in $\mathsf{Hedge}$, $\mathsf{SMART}$ can black-box interleave $\mathsf{FTL}$ with any worst-case/small-loss algorithm without knowing the inner workings of said algorithm, which we see as a significant engineering strength. More importantly, our approach to this problem is distinct as we focus on the fundamental limits (upper and lower bounds) on the competitive ratio that must be incurred when combining $\mathsf{FTL}$ with a worst-case policy; to the best of our knowledge this viewpoint, and the corresponding reduction to an optimal stopping problem, has not been previously explored.

Before proving Theorems 1 and 2, we first illustrate the basic idea of our approach and reduction for the binary prediction setting. This is aided by the observation that the regret of $\mathsf{FTL}$ in this setting admits a simple geometric interpretation: for any sequence, and any time $t$, we have that $\Sigma^{\mathsf{FTL}}$ is equal to 1/2 times the number of ‘lead changes’ up to time $t$; where a lead change is a time step $i$ where the count of $1$s and $0$s in the (sub)sequence $y^{i-1}$ is equal (see Figure 2(a)).

Corollary 2 follows from the regret decomposition in Lemma 1. Furthermore, since the losses of 0s and 1s are equal at a lead change, it also follows that at a lead change $t$, the anytime regret $\Sigma^{\mathsf{FTL}}_{t}$ is also equal to the hindsight regret incurred by $\mathsf{FTL}$ up to time $t$. Since $\Sigma^{\mathsf{FTL}}_{t}$ only increases in value at lead changes, if the $\mathsf{SMART}$ algorithm switches to $\mathsf{ALG}_{\mathsf{WC}}$, it will only happen after a lead change, and thus $\mathsf{SMART}$ behaves as if it had oracle knowledge of the regret of $\mathsf{FTL}$ from just the history up to the current time.

Consider the instantiation of $\mathsf{SMART}$ where $\mathsf{ALG}_{\mathsf{WC}}=\mathsf{Cover}$ and $g(n)=\sqrt{\frac{n}{2\pi}}$. As mentioned in Section 1, a remarkable property of $\mathsf{Cover}$ is that it is the true minimax optimal algorithm, where $\mathrm{Reg}(\mathsf{Cover},y^{n})=\sqrt{\frac{n}{2\pi}}(1+o(1))$ for all sequences $y^{n}$, such that $g(n)$ is nearly equal to $\mathrm{Reg}(\mathsf{Cover},y^{n})$ regardless of the sequence (Cover, 1966).

It follows that $\mathsf{SMART}$ is equivalent to an algorithm which starts with $\mathsf{FTL}$, plays it until the regret of $\mathsf{FTL}$ matches the minimax regret guaranteed by $\mathsf{Cover}$ for the remaining sequence, and then switches to $\mathsf{Cover}$ until the end. Let $t_{\mathrm{sw}}$ denote the last round $\mathsf{SMART}$ plays $\mathsf{FTL}$ before switching to $\mathsf{ALG}_{\mathsf{WC}}$ (with $t_{\mathrm{sw}}=n$ if it never switches). For a single switch algorithm, the sequence which maximizes the regret is one that maximizes the $\mathsf{FTL}$ regret before the switch at $t_{\mathrm{sw}}$, and minimizes the $\mathsf{FTL}$ after the switch, as depicted in Figure 2(a). For such a sequence, $\Sigma^{\mathsf{FTL}}_{t}=t/4$ at lead changes $t$, such that the regret incurred by the algorithm is linear before the switch, matching the linear cost of renting skis in the ski rental problem. Note that $t_{\mathrm{sw}}$ will necessarily be $o(n)$ in such sequences as the time it takes until $\Sigma^{\mathsf{FTL}}_{t}\geq g(n)$ is linear in $g(n)=o(n)$. After the switch, $\mathsf{Cover}$ will incur regret $\sqrt{\frac{n-t_{\mathrm{sw}}}{2\pi}}(1+o(1))=g(n)(1+o(1))$, matching the fixed cost of buying skis at the switching point; Corollary 3 follows as a result of this analysis. Furthermore, in the binary prediction setting, $\mathsf{SMART}$ achieves the stronger notion of instance optimality stated in Definition 1.