1 Introduction
Our work aims to develop algorithms for online learning that are instance optimal (Fagin et al., 2001 ) ,(Roughgarden, 2021 , Chapter 3 3 3 3 ) with respect to the stochastic and minimax optimal algorithms for a given setting. This is best motivated via
a concrete example:
Example 1 (Binary Prediction).
We are given bit stream y n : = y 1 , y 2 , … , y n ∈ { 0 , 1 } n y^{n}\mathrel{\mathop{\mathrel{\mathop{:}}}}=y_{1},y_{2},\ldots,y_{n}\in\{0,1%
\}^{n} italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT : = italic_y start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , italic_y start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT , … , italic_y start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ∈ { 0 , 1 } start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT .
At the start of day t 𝑡 t italic_t , before seeing y t subscript 𝑦 𝑡 y_{t} italic_y start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT , we choose (possibly randomized) prediction Y ^ t ∼ Bernoulli ( a t ) similar-to subscript normal-^ 𝑌 𝑡 normal-Bernoulli subscript 𝑎 𝑡 \widehat{Y}_{t}\sim\mathrm{Bernoulli}(a_{t}) over^ start_ARG italic_Y end_ARG start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ∼ roman_Bernoulli ( italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) (for a t ∈ [ 0 , 1 ] subscript 𝑎 𝑡 0 1 a_{t}\in[0,1] italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ∈ [ 0 , 1 ] ) for the upcoming bit y t subscript 𝑦 𝑡 y_{t} italic_y start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT , given the history y t − 1 superscript 𝑦 𝑡 1 y^{t-1} italic_y start_POSTSUPERSCRIPT italic_t - 1 end_POSTSUPERSCRIPT . Our resulting loss on day t 𝑡 t italic_t is ℓ t ( a t ) = ℙ ( Y ^ t ≠ y t ) = | a t − y t | subscript normal-ℓ 𝑡 subscript 𝑎 𝑡 ℙ subscript normal-^ 𝑌 𝑡 subscript 𝑦 𝑡 subscript 𝑎 𝑡 subscript 𝑦 𝑡 \ell_{t}(a_{t})=\mathbb{P}(\widehat{Y}_{t}\neq y_{t})=|a_{t}-y_{t}| roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) = blackboard_P ( over^ start_ARG italic_Y end_ARG start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ≠ italic_y start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) = | italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT - italic_y start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT | , and our total loss is L n ( 𝖠𝖫𝖦 , y n ) : = ∑ t = 1 n ℓ t ( a t ) L_{n}(\mathsf{ALG},y^{n})\mathrel{\mathop{\mathrel{\mathop{:}}}}=\sum_{t=1}^{n%
}\ell_{t}(a_{t}) italic_L start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ( sansserif_ALG , italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) : = ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) . The objective is to achieve low regret (i.e., additive loss) compared to the loss L n ( a , y n ) = ∑ t = 1 n ℓ t ( a ) subscript 𝐿 𝑛 𝑎 superscript 𝑦 𝑛 superscript subscript 𝑡 1 𝑛 subscript normal-ℓ 𝑡 𝑎 L_{n}(a,y^{n})=\sum_{t=1}^{n}\ell_{t}(a) italic_L start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ( italic_a , italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) = ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a ) of the best fixed action a * ∈ [ 0 , 1 ] superscript 𝑎 0 1 a^{*}\in[0,1] italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ∈ [ 0 , 1 ] in hindsight. As a * superscript 𝑎 a^{*} italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT is the majority in y n superscript 𝑦 𝑛 y^{n} italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT between 0 and 1, it follows that L n ( a * , y n ) = min { ∑ t = 1 n y t , n − ∑ t = 1 n y t } subscript 𝐿 𝑛 superscript 𝑎 superscript 𝑦 𝑛 superscript subscript 𝑡 1 𝑛 subscript 𝑦 𝑡 𝑛 superscript subscript 𝑡 1 𝑛 subscript 𝑦 𝑡 L_{n}(a^{*},y^{n})=\min\left\{\sum_{t=1}^{n}y_{t},n-\sum_{t=1}^{n}y_{t}\right\} italic_L start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT , italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) = roman_min { ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT italic_y start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT , italic_n - ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT italic_y start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT } . Formally, for sequence y n ∈ { 0 , 1 } n superscript 𝑦 𝑛 superscript 0 1 𝑛 y^{n}\in\{0,1\}^{n} italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ∈ { 0 , 1 } start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT , policy 𝖠𝖫𝖦 𝖠𝖫𝖦 \mathsf{ALG} sansserif_ALG incurs regret
Reg ( 𝖠𝖫𝖦 , y n ) Reg 𝖠𝖫𝖦 superscript 𝑦 𝑛 \displaystyle\mathrm{Reg}(\mathsf{ALG},y^{n}) roman_Reg ( sansserif_ALG , italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT )
: = L n ( 𝖠𝖫𝖦 , y n ) − L n ( a * , y n ) = ∑ t = 1 n | a t − y t | − min a ∈ [ 0 , 1 ] ∑ t = 1 T | a − y t | . \displaystyle\mathrel{\mathop{\mathrel{\mathop{:}}}}=L_{n}(\mathsf{ALG},y^{n})%
-L_{n}(a^{*},y^{n})=\textstyle\sum_{t=1}^{n}|a_{t}-y_{t}|-\min_{a\in[0,1]}%
\textstyle\sum_{t=1}^{T}|a-y_{t}|. : = italic_L start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ( sansserif_ALG , italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) - italic_L start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT , italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) = ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT | italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT - italic_y start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT | - roman_min start_POSTSUBSCRIPT italic_a ∈ [ 0 , 1 ] end_POSTSUBSCRIPT ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_T end_POSTSUPERSCRIPT | italic_a - italic_y start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT | .
(1)
Binary prediction goes back to the seminal works of Blackwell (1956 ) and Hannan (1957 ) . The definition of regret is motivated by the case where y t subscript 𝑦 𝑡 y_{t} italic_y start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT is randomly generated as i.i.d. Bernoulli( p ) 𝑝 (p) ( italic_p ) . If p 𝑝 p italic_p is known, then the optimal policy is the ‘Bayes predictor’ a 𝖡𝖺𝗒𝖾𝗌 = ⌊ 2 p ⌋ superscript 𝑎 𝖡𝖺𝗒𝖾𝗌 2 𝑝 a^{\textsf{Bayes}}=\lfloor 2p\rfloor italic_a start_POSTSUPERSCRIPT Bayes end_POSTSUPERSCRIPT = ⌊ 2 italic_p ⌋ (i.e., nearest integer to p 𝑝 p italic_p ),
which coincides with hindsight optimal a * superscript 𝑎 a^{*} italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT with high probability when p 𝑝 p italic_p is away from 1 / 2 1 2 1/2 1 / 2 . When p 𝑝 p italic_p is unknown, the stochastic optimal policy is the Follow The Leader oder 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL policy, which sets a t = 𝖬𝖺𝗃𝗈𝗋𝗂𝗍𝗒 ( y t − 1 ) subscript 𝑎 𝑡 𝖬𝖺𝗃𝗈𝗋𝗂𝗍𝗒 superscript 𝑦 𝑡 1 a_{t}=\mathsf{Majority}(y^{t-1}) italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT = sansserif_Majority ( italic_y start_POSTSUPERSCRIPT italic_t - 1 end_POSTSUPERSCRIPT ) , i.e. the majority bit amongst the first t − 1 𝑡 1 t-1 italic_t - 1 bits (a t = 1 / 2 subscript 𝑎 𝑡 1 2 a_{t}=1/2 italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT = 1 / 2 if both are equal).
A starting point for online learning is the observation that it is easy to construct a sequence y n superscript 𝑦 𝑛 y^{n} italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT such that 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL has poor regret: For example, if y n = ( 1 , 0 , 1 , 0 , 1 , 0 , … ) superscript 𝑦 𝑛 1 0 1 0 1 0 … y^{n}=(1,0,1,0,1,0,\ldots) italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT = ( 1 , 0 , 1 , 0 , 1 , 0 , … ) , i.e., alternate 1 1 1 1 s and 0 0 s, then the regret of 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL grows linearly with n 𝑛 n italic_n . In contrast, worst-case optimal online learning policies such as those of Blackwell and Hannan, or more modern versions like Multiplicative Weights or Follow The Perturbed Leader (see Cesa-Bianchi and Lugosi (2006 ); Slivkins (2019 ) ) guarantee regret of Θ ( n ) Θ 𝑛 \Theta(\sqrt{n}) roman_Θ ( square-root start_ARG italic_n end_ARG ) over all sequences. Indeed, for bit prediction, the exact minimax optimal policy was established by Cover (1966 ) , and this policy (which we refer to as 𝖢𝗈𝗏𝖾𝗋 𝖢𝗈𝗏𝖾𝗋 \mathsf{Cover} sansserif_Cover ) achieves Reg ( 𝖢𝗈𝗏𝖾𝗋 , y n ) = n 2 π ( 1 + o ( 1 ) ) Reg 𝖢𝗈𝗏𝖾𝗋 superscript 𝑦 𝑛 𝑛 2 𝜋 1 𝑜 1 \mathrm{Reg}(\mathsf{Cover},y^{n})=\sqrt{\frac{n}{2\pi}}(1+o(1)) roman_Reg ( sansserif_Cover , italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) = square-root start_ARG divide start_ARG italic_n end_ARG start_ARG 2 italic_π end_ARG end_ARG ( 1 + italic_o ( 1 ) ) under any y n ∈ { 0 , 1 } n superscript 𝑦 𝑛 superscript 0 1 𝑛 y^{n}\in\{0,1\}^{n} italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ∈ { 0 , 1 } start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT , implying it is an equalizer (achieves same regret over all sequences).
While the above discussion seems a convincing endorsement of worst-case online learning algorithms, the situation is more complicated. One problem is that while 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL has bad regret on certain pathological sequences, on more ‘realistic’ sequences 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL performs orders of magnitude better than the minimax regret. As an example, with i.i.d. Bernoulli( p ) 𝑝 (p) ( italic_p ) input, Reg ( 𝖥𝖳𝖫 , y n ) Reg 𝖥𝖳𝖫 superscript 𝑦 𝑛 \mathrm{Reg}(\mathsf{FTL},y^{n}) roman_Reg ( sansserif_FTL , italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) is actually independent of n 𝑛 n italic_n (i.e., O ( 1 ) 𝑂 1 O(1) italic_O ( 1 ) ) as long as p 𝑝 p italic_p is away from 1 / 2 1 2 1/2 1 / 2 with high probability. We demonstrate this in Figure 1 ( a ) 𝑎 (a) ( italic_a ) , where we see Reg ( 𝖥𝖳𝖫 ) Reg 𝖥𝖳𝖫 \mathrm{Reg}(\mathsf{FTL}) roman_Reg ( sansserif_FTL ) is much lower than Reg ( 𝖢𝗈𝗏𝖾𝗋 ) ≈ 0.39 n Reg 𝖢𝗈𝗏𝖾𝗋 0.39 𝑛 \mathrm{Reg}(\mathsf{Cover})\approx 0.39\sqrt{n} roman_Reg ( sansserif_Cover ) ≈ 0.39 square-root start_ARG italic_n end_ARG unless p 𝑝 p italic_p is very close to 1 / 2 1 2 1/2 1 / 2 . This phenomena is known in more general settings (Huang et al., 2016 ) , suggesting that in practice one may be better off just using 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL . On the other hand, as Figure 1 ( b , c ) 𝑏 𝑐 (b,c) ( italic_b , italic_c ) indicates, we know how to generate sequences y n superscript 𝑦 𝑛 y^{n} italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT (Feder et al., 1992 ) for which Reg ( 𝖥𝖳𝖫 , y n ) Reg 𝖥𝖳𝖫 superscript 𝑦 𝑛 \mathrm{Reg}(\mathsf{FTL},y^{n}) roman_Reg ( sansserif_FTL , italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) grows linearly with n 𝑛 n italic_n , and so the n 𝑛 \sqrt{n} square-root start_ARG italic_n end_ARG regret of 𝖢𝗈𝗏𝖾𝗋 𝖢𝗈𝗏𝖾𝗋 \mathsf{Cover} sansserif_Cover becomes appealing.
Now suppose instead that a fictitious oracle is told beforehand which of 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL oder 𝖢𝗈𝗏𝖾𝗋 𝖢𝗈𝗏𝖾𝗋 \mathsf{Cover} sansserif_Cover is better suited for the upcoming sequence y n superscript 𝑦 𝑛 y^{n} italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ; the demand made by instance optimality is that we try to be competitive against such an oracle on every sequence y n superscript 𝑦 𝑛 y^{n} italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT .
Definition 1 (Instance Optimality).
A binary prediction policy 𝖠𝖫𝖦 𝖠𝖫𝖦 \mathsf{ALG} sansserif_ALG is instance optimal with respect to the regret of 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL and 𝖢𝗈𝗏𝖾𝗋 𝖢𝗈𝗏𝖾𝗋 \mathsf{Cover} sansserif_Cover if there exists some universal γ n ≥ 1 subscript 𝛾 𝑛 1 \gamma_{n}\geq 1 italic_γ start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ≥ 1 such that for all y n ∈ { 0 , 1 } n superscript 𝑦 𝑛 superscript 0 1 𝑛 y^{n}\in\{0,1\}^{n} italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ∈ { 0 , 1 } start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT :
Reg ( 𝖠𝖫𝖦 , y n ) ≤ γ n min { Reg ( 𝖥𝖳𝖫 , y n ) , Reg ( 𝖢𝗈𝗏𝖾𝗋 , y n ) } Reg 𝖠𝖫𝖦 superscript 𝑦 𝑛 subscript 𝛾 𝑛 Reg 𝖥𝖳𝖫 superscript 𝑦 𝑛 Reg 𝖢𝗈𝗏𝖾𝗋 superscript 𝑦 𝑛 \small\mathrm{Reg}(\mathsf{ALG},y^{n})\leq\gamma_{n}\min\{\mathrm{Reg}(\mathsf%
{FTL},y^{n}),\mathrm{Reg}(\mathsf{Cover},y^{n})\} roman_Reg ( sansserif_ALG , italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) ≤ italic_γ start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT roman_min { roman_Reg ( sansserif_FTL , italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) , roman_Reg ( sansserif_Cover , italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) }
We henceforth refer to γ n subscript 𝛾 𝑛 \gamma_{n} italic_γ start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT as the competitive ratio achieved by 𝖠𝖫𝖦 𝖠𝖫𝖦 \mathsf{ALG} sansserif_ALG ; ideally we want this ratio to be a constant, i.e., γ n = O ( 1 ) subscript 𝛾 𝑛 𝑂 1 \gamma_{n}=O(1) italic_γ start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT = italic_O ( 1 ) .
This necessitates that on sequences where 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL gets a constant regret, then 𝖠𝖫𝖦 𝖠𝖫𝖦 \mathsf{ALG} sansserif_ALG basically follows 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL throughout, while on sequences where 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL has high (in particular, ω ( n ) 𝜔 𝑛 \omega(\sqrt{n}) italic_ω ( square-root start_ARG italic_n end_ARG ) ) regret, then 𝖠𝖫𝖦 𝖠𝖫𝖦 \mathsf{ALG} sansserif_ALG follows 𝖢𝗈𝗏𝖾𝗋 𝖢𝗈𝗏𝖾𝗋 \mathsf{Cover} sansserif_Cover in most rounds.
The challenge in designing instance optimal algorithms is that the regret of any algorithm is a quantity that is not adapted to the natural filtration, i.e. it may not be possible to track Reg ( 𝖠𝖫𝖦 , y n ) Reg 𝖠𝖫𝖦 superscript 𝑦 𝑛 \mathrm{Reg}(\mathsf{ALG},y^{n}) roman_Reg ( sansserif_ALG , italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) for any 𝖠𝖫𝖦 𝖠𝖫𝖦 \mathsf{ALG} sansserif_ALG from just the history ( y 1 , y 2 , … , y t − 1 ) subscript 𝑦 1 subscript 𝑦 2 … subscript 𝑦 𝑡 1 (y_{1},y_{2},\ldots,y_{t-1}) ( italic_y start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , italic_y start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT , … , italic_y start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT ) , since the hindsight optimal action a * superscript 𝑎 a^{*} italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT depends on the entire sequence y n superscript 𝑦 𝑛 y^{n} italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT .
One proxy is to track an algorithm’s loss instead, leading to the idea of ‘corralling’ policies (Agarwal et al., 2017 ; Pacchiano et al., 2020 ; Dann et al., 2023 ) , that run online learning over the reference algorithms to get within O ( poly ( n ) ) 𝑂 poly 𝑛 O(\text{poly}(n)) italic_O ( poly ( italic_n ) ) of the smaller of the two losses.
Such an approach can not ensure γ n = O ( 1 ) subscript 𝛾 𝑛 𝑂 1 \gamma_{n}=O(1) italic_γ start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT = italic_O ( 1 ) :
for example, consider an i.i.d. sequence of Bernoulli( 0.1 ) 0.1 (0.1) ( 0.1 ) bits, where 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL has lower regret than 𝖢𝗈𝗏𝖾𝗋 𝖢𝗈𝗏𝖾𝗋 \mathsf{Cover} sansserif_Cover . With high probability on any such sequence we have small Reg ( 𝖥𝖳𝖫 , y n ) = O ( 1 ) Reg 𝖥𝖳𝖫 superscript 𝑦 𝑛 𝑂 1 \mathrm{Reg}(\mathsf{FTL},y^{n})=O(1) roman_Reg ( sansserif_FTL , italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) = italic_O ( 1 ) and yet high loss L n ( a * , y n ) = Θ ( n ) subscript 𝐿 𝑛 superscript 𝑎 superscript 𝑦 𝑛 Θ 𝑛 L_{n}(a^{*},y^{n})=\Theta(n) italic_L start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT , italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) = roman_Θ ( italic_n ) ; now any corralling algorithm (even a small loss one) must suffer O ( poly ( n ) ) 𝑂 poly 𝑛 O(\text{poly}(n)) italic_O ( poly ( italic_n ) ) regret, and hence ω ( 1 ) 𝜔 1 \omega(1) italic_ω ( 1 ) competitive ratio. This example also shows that achieving a constant factor guarantee with respect to the minimum of the two losses does not translate to a constant factor guarantee with respect to the minimum of the two regrets .
The instance optimal guarantee is closely related to best-of-both-worlds guarantees, which aim for algorithms that simultaneously achieve (up to constant factors) both the low pseudoregret guarantee of policies designed for stochastic inputs (as with 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL in our setting, or the Upper Confidence Bound (𝖴𝖢𝖡 𝖴𝖢𝖡 \mathsf{UCB} sansserif_UCB ) algorithm in bandits), as well as a per-sequence regret guarantee comparable to a worst-case optimal algorithm 𝖠𝖫𝖦 𝖶𝖢 subscript 𝖠𝖫𝖦 𝖶𝖢 \mathsf{ALG}_{\mathsf{WC}} sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT (Eg. 𝖢𝗈𝗏𝖾𝗋 𝖢𝗈𝗏𝖾𝗋 \mathsf{Cover} sansserif_Cover or Hedge in online learning; 𝖤𝖷𝖯 3 𝖤𝖷𝖯 3 \mathsf{EXP}3 sansserif_EXP 3 in bandits Auer et al. (2002a ) ).
Such guarantees have been shown in a variety of settings, including online learning (De Rooij et al., 2014 ; Orabona and Pál, 2015 ; Mourtada and Gaïffas, 2019 ; Bilodeau et al., 2023 ) and bandit settings (Bubeck and Slivkins, 2012 ; Zimmert and Seldin, 2019 ; Lykouris et al., 2018 ; Dann et al., 2023 ) .
One problem though is that since pseudoregret and worst-case regret are very different quantities, the above results tend to be hard to interpret, and less predictive of good performance.
Note though that given a pair of stochastic/worst-case optimal algorithms, a policy that is γ 𝛾 \gamma italic_γ -instance-optimal w.r.t. these immediately satisfies a best-of-both-worlds guarantee with constant factor γ 𝛾 \gamma italic_γ .
In this regard, instance optimality provides a stronger guarantee as it holds on every sequence y n superscript 𝑦 𝑛 y^{n} italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT regardless of how it is generated.
Moreover, the parameter γ 𝛾 \gamma italic_γ can also provide sharper comparisons between algorithms, as well as admit hardness results on the limits of such guarantees.
1.1 Our Contributions
Figure 1: Comparing regret of 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL , 𝖢𝗈𝗏𝖾𝗋 𝖢𝗈𝗏𝖾𝗋 \mathsf{Cover} sansserif_Cover and 𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART on a collection of input sequences (for fixed n 𝑛 n italic_n ).
∙ normal-∙ \bullet ∙ In Fig. ( a ) 𝑎 (a) ( italic_a ) , we consider i.i.d. Bernoulli( p ) 𝑝 (p) ( italic_p ) inputs for varying p 𝑝 p italic_p . The regret of 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL is much lower than 𝖢𝗈𝗏𝖾𝗋 𝖢𝗈𝗏𝖾𝗋 \mathsf{Cover} sansserif_Cover for p < 1 / 2 𝑝 1 2 p<1/2 italic_p < 1 / 2 ; the regret of 𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART tracks 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL closely (better than 2 R e g ( 𝖥𝖳𝖫 ) 2 normal-R normal-e normal-g 𝖥𝖳𝖫 2\mathrm{Reg}(\mathsf{FTL}) 2 roman_R roman_e roman_g ( sansserif_FTL ) , indicated by dotted line).
∙ normal-∙ \bullet ∙ In Fig. ( b ) 𝑏 (b) ( italic_b ) and ( c ) 𝑐 (c) ( italic_c ) , we consider ‘worst-case’ binary sequences (as per (Feder et al., 1992 ) ) parameterized by the number of ‘lead-changes’: the sequence with parameter c 𝑐 c italic_c comprises of c 𝑐 c italic_c pairs ‘0 , 1 0 1
0,1 0 , 1 ’ or ‘1 , 0 1 0
1,0 1 , 0 ’, followed by n − 2 c 𝑛 2 𝑐 n-2c italic_n - 2 italic_c ‘1 1 1 1 ’s.
In Fig. ( b ) 𝑏 (b) ( italic_b ) , we consider 𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART with a deterministic switching threshold (Theorem 1 ) and compare Reg ( 𝖲𝖬𝖠𝖱𝖳 ) normal-Reg 𝖲𝖬𝖠𝖱𝖳 \mathrm{Reg}(\mathsf{SMART}) roman_Reg ( sansserif_SMART ) with 2 R e g ( 𝖥𝖳𝖫 ) 2 normal-R normal-e normal-g 𝖥𝖳𝖫 2\mathrm{Reg}(\mathsf{FTL}) 2 roman_R roman_e roman_g ( sansserif_FTL ) and 2 R e g ( 𝖢𝗈𝗏𝖾𝗋 ) 2 normal-R normal-e normal-g 𝖢𝗈𝗏𝖾𝗋 2\mathrm{Reg}(\mathsf{Cover}) 2 roman_R roman_e roman_g ( sansserif_Cover ) (dotted lines); in Fig. ( c ) 𝑐 (c) ( italic_c ) , we use a randomized threshold (Theorem 2 ), and show the average regret over the randomized threshold, as well as sample paths (plotted in green), and compare with e e − 1 𝑒 𝑒 1 \frac{e}{e-1} divide start_ARG italic_e end_ARG start_ARG italic_e - 1 end_ARG times Reg ( 𝖥𝖳𝖫 ) normal-Reg 𝖥𝖳𝖫 \mathrm{Reg}(\mathsf{FTL}) roman_Reg ( sansserif_FTL ) and Reg ( 𝖢𝗈𝗏𝖾𝗋 ) normal-Reg 𝖢𝗈𝗏𝖾𝗋 \mathrm{Reg}(\mathsf{Cover}) roman_Reg ( sansserif_Cover ) (dotted lines).
We consider a general online learning setting where at the beginning of each round t ∈ [ n ] 𝑡 delimited-[] 𝑛 t\in[n] italic_t ∈ [ italic_n ] , a policy 𝖠𝖫𝖦 𝖠𝖫𝖦 \mathsf{ALG} sansserif_ALG first plays an action a t ∈ 𝒜 subscript 𝑎 𝑡 𝒜 a_{t}\in\mathcal{A} italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ∈ caligraphic_A , following which, a loss function ℓ t : 𝒜 → [ 0 , 1 ] : subscript ℓ 𝑡 𝒜 → 0 1 \ell_{t}\mathrel{\mathop{:}}\mathcal{A}\to[0,1] roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT : caligraphic_A → [ 0 , 1 ] is revealed, resulting in a loss of ℓ t ( a t ) subscript ℓ 𝑡 subscript 𝑎 𝑡 \ell_{t}(a_{t}) roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) . The regret is defined according to:
Reg ( 𝖠𝖫𝖦 , ℓ n ) = ∑ t = 1 n ℓ t ( a t ) − inf a ∈ 𝒜 ∑ t = 1 n ℓ t ( a ) . Reg 𝖠𝖫𝖦 superscript ℓ 𝑛 superscript subscript 𝑡 1 𝑛 subscript ℓ 𝑡 subscript 𝑎 𝑡 subscript infimum 𝑎 𝒜 superscript subscript 𝑡 1 𝑛 subscript ℓ 𝑡 𝑎 \displaystyle\mathrm{Reg}(\mathsf{ALG},\ell^{n})=\textstyle\sum_{t=1}^{n}\ell_%
{t}(a_{t})-\inf_{a\in\mathcal{A}}\textstyle\sum_{t=1}^{n}\ell_{t}(a). roman_Reg ( sansserif_ALG , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) = ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) - roman_inf start_POSTSUBSCRIPT italic_a ∈ caligraphic_A end_POSTSUBSCRIPT ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a ) .
(2)
More generally, as in with bit prediction, we allow 𝖠𝖫𝖦 𝖠𝖫𝖦 \mathsf{ALG} sansserif_ALG to play in round t 𝑡 t italic_t a measure w t ∈ Δ 𝒜 subscript 𝑤 𝑡 subscript Δ 𝒜 w_{t}\in\Delta_{\mathcal{A}} italic_w start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ∈ roman_Δ start_POSTSUBSCRIPT caligraphic_A end_POSTSUBSCRIPT (i.e., play { w t : 𝒜 ↦ [ 0 , 1 ] | ∑ a ∈ 𝒜 w t ( a ) = 1 } conditional-set : subscript 𝑤 𝑡 𝒜 maps-to 0 1 subscript 𝑎 𝒜 subscript 𝑤 𝑡 𝑎 1 \{w_{t}\mathrel{\mathop{:}}\mathcal{A}\mapsto[0,1]|\sum_{a\in\mathcal{A}}w_{t}%
(a)=1\} { italic_w start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT : caligraphic_A ↦ [ 0 , 1 ] | ∑ start_POSTSUBSCRIPT italic_a ∈ caligraphic_A end_POSTSUBSCRIPT italic_w start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a ) = 1 } ), resulting in an expected loss of ∑ a ∈ 𝒜 w t ( a ) ℓ t ( a ) subscript 𝑎 𝒜 subscript 𝑤 𝑡 𝑎 subscript ℓ 𝑡 𝑎 \sum_{a\in\mathcal{A}}w_{t}(a)\ell_{t}(a) ∑ start_POSTSUBSCRIPT italic_a ∈ caligraphic_A end_POSTSUBSCRIPT italic_w start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a ) roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a ) . For notational convenience, we henceforth use ( a t , ℓ t ( a t ) ) subscript 𝑎 𝑡 subscript ℓ 𝑡 subscript 𝑎 𝑡 (a_{t},\ell_{t}(a_{t})) ( italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT , roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) ) for the action/loss, and reserve use of expectations for randomness in the algorithm and/or sequence.
We want to understand when is it possible to attain instance optimality as in Eq. (1 ) with respect to a given pair of algorithms. Ideally, we want the first to be optimal for stochastic instances, and the second to be minimax optimal; unfortunately however exact optimal policies are unknown except in simple settings. To this end, we make two amendments to our goal: First, for the stochastic optimal policy, we use 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL ; this is well defined in any online learning setting, and moreover, known to be optimal or near-optimal for a wide range of settings under minimal assumptions (Kotłowski, 2018 ) . Second, instead of the minimax policy, we use as reference any policy 𝖠𝖫𝖦 𝖶𝖢 subscript 𝖠𝖫𝖦 𝖶𝖢 \mathsf{ALG}_{\mathsf{WC}} sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT which has a known worst case regret bound g ( n ) 𝑔 𝑛 g(n) italic_g ( italic_n ) . With these modifications in place, we have the following objective.
Definition 2 .
Given 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL and any algorithm 𝖠𝖫𝖦 𝖶𝖢 subscript 𝖠𝖫𝖦 𝖶𝖢 \mathsf{ALG}_{\mathsf{WC}} sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT with sup ℓ n Reg ( 𝖠𝖫𝖦 𝖶𝖢 , ℓ n ) ≤ g ( n ) subscript supremum superscript normal-ℓ 𝑛 normal-Reg subscript 𝖠𝖫𝖦 𝖶𝖢 superscript normal-ℓ 𝑛 𝑔 𝑛 \sup_{\ell^{n}}\mathrm{Reg}(\mathsf{ALG}_{\mathsf{WC}},\ell^{n})\leq g(n) roman_sup start_POSTSUBSCRIPT roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT end_POSTSUBSCRIPT roman_Reg ( sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) ≤ italic_g ( italic_n ) , we say a policy 𝖠𝖫𝖦 𝖠𝖫𝖦 \mathsf{ALG} sansserif_ALG is instance optimal with respect to the pair if there exists some universal γ n ≥ 1 subscript 𝛾 𝑛 1 \gamma_{n}\geq 1 italic_γ start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ≥ 1 (i.e. not depending on y n superscript 𝑦 𝑛 y^{n} italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) such that for every sequence of losses ℓ n superscript normal-ℓ 𝑛 \ell^{n} roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT :
Reg ( 𝖠𝖫𝖦 , ℓ n ) ≤ γ n min { Reg ( 𝖥𝖳𝖫 , ℓ n ) , g ( n ) } Reg 𝖠𝖫𝖦 superscript ℓ 𝑛 subscript 𝛾 𝑛 Reg 𝖥𝖳𝖫 superscript ℓ 𝑛 𝑔 𝑛 \small\mathrm{Reg}(\mathsf{ALG},\ell^{n})\leq\gamma_{n}\min\{\mathrm{Reg}(%
\mathsf{FTL},\ell^{n}),g(n)\} roman_Reg ( sansserif_ALG , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) ≤ italic_γ start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT roman_min { roman_Reg ( sansserif_FTL , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) , italic_g ( italic_n ) }
While the above guarantee is not truly instance-optimal in that we are comparing against a worst-case regret bound g ( n ) 𝑔 𝑛 g(n) italic_g ( italic_n ) for 𝖠𝖫𝖦 𝖶𝖢 subscript 𝖠𝖫𝖦 𝖶𝖢 \mathsf{ALG}_{\mathsf{WC}} sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT rather than its performance on the instance ℓ n superscript ℓ 𝑛 \ell^{n} roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT , the two are the same if 𝖠𝖫𝖦 𝖶𝖢 subscript 𝖠𝖫𝖦 𝖶𝖢 \mathsf{ALG}_{\mathsf{WC}} sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT is minimax optimal and hence attaining equal regret on all loss sequences; recall this is true of 𝖢𝗈𝗏𝖾𝗋 𝖢𝗈𝗏𝖾𝗋 \mathsf{Cover} sansserif_Cover for binary prediction.
To realize the above goal, we propose the Switch via Monotone Adapted Regret Traces (𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART ) approach, which at a high level is a black-box way to convert design of instance-optimal policies into a simple optimal stopping problem.
Our approach depends on just two ingredients: first, owing to the additive structure of online learning problems, we have that the minimax guarantee g ( k ) 𝑔 𝑘 g(k) italic_g ( italic_k ) above holds over any k ∈ ℤ 𝑘 ℤ k\in\mathbb{Z} italic_k ∈ blackboard_Z and any (sub)sequence of n 𝑛 n italic_n loss functions; second, we show that 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL admits simple anytime regret estimator Σ τ 𝖥𝖳𝖫 subscript superscript Σ 𝖥𝖳𝖫 𝜏 \Sigma^{\mathsf{FTL}}_{\tau} roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_τ end_POSTSUBSCRIPT (see Lemma 1 ) which is monotone and adapted (i.e., a function only of historical data). Using these two observations, we can reduce the task of minimizing regret to a version of the ‘ski-rental’ problem (Karlin et al., 1994 ; Borodin and El-Yaniv, 2005 ) , as follows:
we play 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL up to some stopping time τ 𝜏 \tau italic_τ , and then switch to 𝖠𝖫𝖦 𝖶𝖢 subscript 𝖠𝖫𝖦 𝖶𝖢 \mathsf{ALG}_{\mathsf{WC}} sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT for the remaining n − τ 𝑛 𝜏 n-\tau italic_n - italic_τ periods, resetting all losses to zero. This algorithm incurs a total regret bounded by Σ τ 𝖥𝖳𝖫 + g ( n − τ ) subscript superscript Σ 𝖥𝖳𝖫 𝜏 𝑔 𝑛 𝜏 \Sigma^{\mathsf{FTL}}_{\tau}+g(n-\tau) roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_τ end_POSTSUBSCRIPT + italic_g ( italic_n - italic_τ ) , and using ideas from competitive analysis, we get that there is a simple way to choose the stopping time τ 𝜏 \tau italic_τ to achieve an e / ( e − 1 ) ≈ 1.58 𝑒 𝑒 1 1.58 e/(e-1)\approx 1.58 italic_e / ( italic_e - 1 ) ≈ 1.58 -competitive ratio guarantee with respect to the minimum between the regret of 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL and the worst case guarantee g ( n ) 𝑔 𝑛 g(n) italic_g ( italic_n ) .
Theorem .
(See Theorem 2 )
Let 𝖠𝖫𝖦 𝖶𝖢 subscript 𝖠𝖫𝖦 𝖶𝖢 \mathsf{ALG}_{\mathsf{WC}} sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT have worst-case regret sup ℓ n Reg ( 𝖠𝖫𝖦 𝖶𝖢 , ℓ n ) ≤ g ( n ) subscript supremum superscript normal-ℓ 𝑛 normal-Reg subscript 𝖠𝖫𝖦 𝖶𝖢 superscript normal-ℓ 𝑛 𝑔 𝑛 \sup_{\ell^{n}}\mathrm{Reg}(\mathsf{ALG}_{\mathsf{WC}},\ell^{n})\leq g(n) roman_sup start_POSTSUBSCRIPT roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT end_POSTSUBSCRIPT roman_Reg ( sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) ≤ italic_g ( italic_n )
where g ( n ) 𝑔 𝑛 g(n) italic_g ( italic_n ) is some monotonic function of n 𝑛 n italic_n . An instantiation of 𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART achieves
Reg ( 𝖲𝖬𝖠𝖱𝖳 , ℓ n ) ≤ e e − 1 min { Reg ( 𝖥𝖳𝖫 , ℓ n ) , g ( n ) } + 1 . Reg 𝖲𝖬𝖠𝖱𝖳 superscript ℓ 𝑛 𝑒 𝑒 1 Reg 𝖥𝖳𝖫 superscript ℓ 𝑛 𝑔 𝑛 1 \displaystyle\small\mathrm{Reg}(\mathsf{SMART},\ell^{n})\leq\frac{e}{e-1}\min%
\{\mathrm{Reg}(\mathsf{FTL},\ell^{n}),g(n)\}+1. roman_Reg ( sansserif_SMART , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) ≤ divide start_ARG italic_e end_ARG start_ARG italic_e - 1 end_ARG roman_min { roman_Reg ( sansserif_FTL , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) , italic_g ( italic_n ) } + 1 .
(3)
A highlight of our approach is the surprising simplicity of the algorithm and analysis, despite the strength of the instance optimality guarantee. In particular, our approach is modular, allowing one to plug in any 𝖠𝖫𝖦 𝖶𝖢 subscript 𝖠𝖫𝖦 𝖶𝖢 \mathsf{ALG}_{\mathsf{WC}} sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT and corresponding worst case bound g ( n ) 𝑔 𝑛 g(n) italic_g ( italic_n ) , thus letting us handle any online learning setting with known minimax bounds. This results in an entire family of instance optimal policies for settings such as predictions with experts and online convex optimization. Moreover, the approach is easy to extend to get more complex guarantees; as an example, if 𝖠𝖫𝖦 𝖶𝖢 subscript 𝖠𝖫𝖦 𝖶𝖢 \mathsf{ALG}_{\mathsf{WC}} sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT is designed to get low regret for benign (i.e., ‘small-loss’) sequences ℓ n superscript ℓ 𝑛 \ell^{n} roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT , then we show how to use 𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART as a subroutine and achieve an instance optimal guarantee with respect to the regret of 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL and a small loss regret bound.
Corollary 1 (Following Theorem 5 ).
Consider the prediction with expert advice setting (Cesa-Bianchi et al., 1997 ) , where 𝒜 = Δ m − 1 𝒜 superscript normal-Δ 𝑚 1 \mathcal{A}=\Delta^{m-1} caligraphic_A = roman_Δ start_POSTSUPERSCRIPT italic_m - 1 end_POSTSUPERSCRIPT , the m − limit-from 𝑚 m- italic_m - simplex for m ≥ 2 𝑚 2 m\geq 2 italic_m ≥ 2 , and ℓ t ( a ) = ⟨ a , ℓ t ⟩ subscript normal-ℓ 𝑡 𝑎 𝑎 subscript normal-ℓ 𝑡
\ell_{t}(a)=\langle a,\ell_{t}\rangle roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a ) = ⟨ italic_a , roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ⟩ for ℓ t ∈ [ 0 , 1 ] m subscript normal-ℓ 𝑡 superscript 0 1 𝑚 \ell_{t}\in[0,1]^{m} roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ∈ [ 0 , 1 ] start_POSTSUPERSCRIPT italic_m end_POSTSUPERSCRIPT . Let L * : = min j ∑ t = 1 n ℓ t j L^{*}\mathrel{\mathop{\mathrel{\mathop{:}}}}=\min_{j}\sum_{t=1}^{n}\ell_{tj} italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT : = roman_min start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT roman_ℓ start_POSTSUBSCRIPT italic_t italic_j end_POSTSUBSCRIPT . An instantiation of 𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART achieves
Reg ( 𝖲𝖬𝖠𝖱𝖳 , ℓ n ) Reg 𝖲𝖬𝖠𝖱𝖳 superscript ℓ 𝑛 \displaystyle\mathrm{Reg}(\mathsf{SMART},\ell^{n}) roman_Reg ( sansserif_SMART , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT )
≤ 2 min { Reg ( 𝖥𝖳𝖫 , ℓ n ) , 10 2 L * log m } + O ( log L * log m ) . absent 2 Reg 𝖥𝖳𝖫 superscript ℓ 𝑛 10 2 superscript 𝐿 𝑚 𝑂 superscript 𝐿 𝑚 \displaystyle\leq 2\min\left\{\mathrm{Reg}(\mathsf{FTL},\ell^{n}),10\sqrt{2L^{%
*}\log m}\right\}+O(\log L^{*}\log m). ≤ 2 roman_min { roman_Reg ( sansserif_FTL , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) , 10 square-root start_ARG 2 italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT roman_log italic_m end_ARG } + italic_O ( roman_log italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT roman_log italic_m ) .
Finally, studying instance optimality also lets us understand the fundamental limits of best-of-both worlds algorithms.
To this end, we provide a lower bound that shows our algorithm is nearly optimal in the competitive ratio.
To the best of our knowledge, this is the first hardness result for best-of-both-worlds guarantees in online learning.
Theorem .
(See Theorem 3 )
In the binary prediction setting, given any online algorithm 𝖠𝖫𝖦 𝖠𝖫𝖦 \mathsf{ALG} sansserif_ALG , there exist sequences y n ∈ { 0 , 1 } n superscript 𝑦 𝑛 superscript 0 1 𝑛 y^{n}\in\{0,1\}^{n} italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ∈ { 0 , 1 } start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT such that:
Reg ( 𝖠𝖫𝖦 , y n ) Reg 𝖠𝖫𝖦 superscript 𝑦 𝑛 \displaystyle\mathrm{Reg}(\mathsf{ALG},y^{n}) roman_Reg ( sansserif_ALG , italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT )
≥ 1.43 min { Reg ( 𝖥𝖳𝖫 , y n ) , Reg ( 𝖢𝗈𝗏𝖾𝗋 , y n ) } absent 1.43 Reg 𝖥𝖳𝖫 superscript 𝑦 𝑛 Reg 𝖢𝗈𝗏𝖾𝗋 superscript 𝑦 𝑛 \displaystyle\geq 1.43\min\left\{\mathrm{Reg}(\mathsf{FTL},y^{n}),\mathrm{Reg}%
(\mathsf{Cover},y^{n})\right\} ≥ 1.43 roman_min { roman_Reg ( sansserif_FTL , italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) , roman_Reg ( sansserif_Cover , italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) }
Note again that in binary prediction, 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL achieves the optimal pseudoregret under i.i.d. inputs, while 𝖢𝗈𝗏𝖾𝗋 𝖢𝗈𝗏𝖾𝗋 \mathsf{Cover} sansserif_Cover is the true minimax policy; thus, this is a fundamental lower bound on best-of-both-worlds guarantee in any online learning setting.
1.2 Related work
There have been many approaches towards combining stochastic and worst-case guarantees. As we discussed before, there is a large literature on best-of-both-worlds algorithms for both full and partial information settings (Wei and Luo, 2018 ; Bubeck et al., 2019 ; Zimmert and Seldin, 2019 ; Dann et al., 2023 ) , and also more complex settings such as metrical task systems (Bhuyan et al., 2023 ) and control (Sabag et al., 2021 ; Goel et al., 2023 ) . Another line of work (Rakhlin et al., 2011 ; Haghtalab et al., 2022 ; Block et al., 2022 ; Bhatt et al., 2023 ) considers smoothed analysis , where the worst-case actions of the adversary are perturbed by nature. A third approach (Bubeck and Slivkins, 2012 ; Lykouris et al., 2018 ; Amir et al., 2020 ; Zimmert and Seldin, 2019 ) interpolates between the stochastic and adversarial settings by considering most ℓ t subscript ℓ 𝑡 \ell_{t} roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT to be i.i.d., interspersed with a few adversarially chosen instances (corruptions). Finally, another line considers the data-generating distribution to come from a ball of specified radius around i.i.d. distributions (Mourtada and Gaïffas, 2019 ; Bilodeau et al., 2023 ) . While all these approaches provide useful insights into the gap between average and worst-case guarantees, one can argue they are all imprecisely specified – given an instance { ℓ t } t ∈ [ n ] subscript subscript ℓ 𝑡 𝑡 delimited-[] 𝑛 \{\ell_{t}\}_{t\in[n]} { roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT } start_POSTSUBSCRIPT italic_t ∈ [ italic_n ] end_POSTSUBSCRIPT in hindsight, there is no clear sense as to which model best ‘explains’ the instance.
Our focus on instance optimality instead follows the approach of better understanding and shaping the per-sequence regret landscape. The origins of this approach arguably come from the seminal work of Cover (1966 ) for binary prediction (we discuss this in more detail in Section 3 ), with a later focus on better bounds for benign instances in general online learning (Auer et al., 2002b ; Cesa-Bianchi et al., 2005 ; Hazan and Kale, 2010 ) . More recently,
a line of work (Koolen et al., 2014 ; Van Erven et al., 2015 ; Van Erven and Koolen, 2016 ; Gaillard et al., 2014 ) have studied designing policies that can adapt to different types of data sequences and achieve multiple performance guarantees simultaneously. The main idea is to use multiple learning rates that are weighted according to their empirical performance on the data. While the focus is still primarily on classifying instances based on when they are easier/harder to learn, some of the resulting guarantees have an instance-optimality flavor; for example, Van Erven and Koolen (2016 ) show how to simultaneously match the performance guarantee (in terms of certain variance bounds) attained by different learning rates in Hedge. Such approaches however need to understand their baseline algorithms in great detail, and use them in a ‘white-box’ way to get their guarantees.
In contrast, our approach fundamentally focuses on combining policies in a black-box way to get instance optimal outcomes. As we mention, this is similar in spirit to corralling bandit algorithms Agarwal et al. (2017 ); Pacchiano et al. (2020 ); Dann et al. (2023 ) , as well as more recent work on online algorithms with predictions Bamas et al. (2020 ); Dinitz et al. (2022 ); Anand et al. (2022 ) ; however, as we mention above, these all get guarantees with respect to the loss of the reference algorithms, which is much weaker than our regret guarantees (though they do so in much more complex settings with partial information and/or state).
To our knowledge, the only previous result which attains a comparable instance-optimality guarantee to ours is that of De Rooij et al. (2014 ) for the experts problem, where the authors propose the 𝖥𝗅𝗂𝗉𝖥𝗅𝗈𝗉 𝖥𝗅𝗂𝗉𝖥𝗅𝗈𝗉 \mathsf{FlipFlop} sansserif_FlipFlop policy which interleaves 𝖧𝖾𝖽𝗀𝖾 𝖧𝖾𝖽𝗀𝖾 \mathsf{Hedge} sansserif_Hedge (with varying learning rates) and 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL to obtain a regret guarantee similar to that of Corollary 1 . In fact, their guarantee is stronger as 𝖥𝗅𝗂𝗉𝖥𝗅𝗈𝗉 𝖥𝗅𝗂𝗉𝖥𝗅𝗈𝗉 \mathsf{FlipFlop} sansserif_FlipFlop is shown to be 5.64-competitive with respect to min { Reg ( 𝖥𝖳𝖫 , ℓ n ) , g ( L * ) } Reg 𝖥𝖳𝖫 superscript ℓ 𝑛 𝑔 superscript 𝐿 \min\{\mathrm{Reg}(\mathsf{FTL},\ell^{n}),g(L^{*})\} roman_min { roman_Reg ( sansserif_FTL , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) , italic_g ( italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) } where g ( L * ) ≤ L * log m 𝑔 superscript 𝐿 superscript 𝐿 𝑚 g(L^{*})\leq\sqrt{L^{*}\log m} italic_g ( italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) ≤ square-root start_ARG italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT roman_log italic_m end_ARG (De Rooij et al., 2014 , Corollary 16) . However,
while 𝖥𝗅𝗂𝗉𝖥𝗅𝗈𝗉 𝖥𝗅𝗂𝗉𝖥𝗅𝗈𝗉 \mathsf{FlipFlop} sansserif_FlipFlop depends on a clever choice of learning rates in 𝖧𝖾𝖽𝗀𝖾 𝖧𝖾𝖽𝗀𝖾 \mathsf{Hedge} sansserif_Hedge , 𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART can black-box interleave 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL with any worst-case/small-loss algorithm without knowing the inner workings of said algorithm, which we see as a significant engineering strength. More importantly, our approach to this problem is distinct as we focus on the fundamental limits (upper and lower bounds) on the competitive ratio that must be incurred when combining 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL with a worst-case policy; to the best of our knowledge this viewpoint, and the corresponding reduction to an optimal stopping problem, has not been previously explored.
2 Instance Optimal Online Learning: Achievability via 𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART
Given the setting and problem statement above, we can now present the 𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART policy.
We do this for a general online learning problem, wherein we want to combine 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL with any given algorithm 𝖠𝖫𝖦 𝖶𝖢 subscript 𝖠𝖫𝖦 𝖶𝖢 \mathsf{ALG}_{\mathsf{WC}} sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT with a worst-case regret guarantee sup ℓ n Reg ( 𝖠𝖫𝖦 𝖶𝖢 , ℓ n ) ≤ g ( n ) subscript supremum superscript ℓ 𝑛 Reg subscript 𝖠𝖫𝖦 𝖶𝖢 superscript ℓ 𝑛 𝑔 𝑛 \sup_{\ell^{n}}\mathrm{Reg}(\mathsf{ALG}_{\mathsf{WC}},\ell^{n})\leq g(n) roman_sup start_POSTSUBSCRIPT roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT end_POSTSUBSCRIPT roman_Reg ( sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) ≤ italic_g ( italic_n ) .
Before presenting the policy, we first need the following regret decomposition for 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL .
Lemma 1 (Regret of 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL ).
If L t ( ⋅ ) : = ∑ i = 1 t ℓ t ( ⋅ ) L_{t}(\cdot)\mathrel{\mathop{\mathrel{\mathop{:}}}}=\sum_{i=1}^{t}\ell_{t}(\cdot) italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( ⋅ ) : = ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_t end_POSTSUPERSCRIPT roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( ⋅ ) , i.e. the cumulative loss function, then
Reg ( 𝖥𝖳𝖫 , ℓ n ) = ∑ t = 1 n ( L t ( a t − 1 * ) − L t ( a t * ) ) . Reg 𝖥𝖳𝖫 superscript ℓ 𝑛 superscript subscript 𝑡 1 𝑛 subscript 𝐿 𝑡 subscript superscript 𝑎 𝑡 1 subscript 𝐿 𝑡 subscript superscript 𝑎 𝑡 \displaystyle\mathrm{Reg}(\mathsf{FTL},\ell^{n})=\textstyle\sum_{t=1}^{n}(L_{t%
}(a^{*}_{t-1})-L_{t}(a^{*}_{t})). roman_Reg ( sansserif_FTL , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) = ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ( italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT ) - italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) ) .
This is reminiscent of the ‘be-the-leader’ lemma (Kalai and Vempala, 2005 ; Slivkins, 2019 ) , although never stated explicitly as an exact decomposition.
Proof.
Recall we define a t 𝖥𝖳𝖫 = a t − 1 * subscript superscript 𝑎 𝖥𝖳𝖫 𝑡 subscript superscript 𝑎 𝑡 1 a^{\mathsf{FTL}}_{t}=a^{*}_{t-1} italic_a start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT = italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT , and hence inf a ∈ 𝒜 ∑ t = 1 n ℓ t ( a ) = L n ( a n * ) subscript infimum 𝑎 𝒜 superscript subscript 𝑡 1 𝑛 subscript ℓ 𝑡 𝑎 subscript 𝐿 𝑛 subscript superscript 𝑎 𝑛 \inf_{a\in\mathcal{A}}\sum_{t=1}^{n}\ell_{t}(a)=L_{n}(a^{*}_{n}) roman_inf start_POSTSUBSCRIPT italic_a ∈ caligraphic_A end_POSTSUBSCRIPT ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a ) = italic_L start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ) . Now we have
Reg ( 𝖥𝖳𝖫 , ℓ n ) Reg 𝖥𝖳𝖫 superscript ℓ 𝑛 \displaystyle\mathrm{Reg}(\mathsf{FTL},\ell^{n}) roman_Reg ( sansserif_FTL , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT )
= ∑ t = 1 n ℓ t ( a t − 1 * ) − L n ( a n * ) absent superscript subscript 𝑡 1 𝑛 subscript ℓ 𝑡 subscript superscript 𝑎 𝑡 1 subscript 𝐿 𝑛 subscript superscript 𝑎 𝑛 \displaystyle=\textstyle\sum_{t=1}^{n}\ell_{t}(a^{*}_{t-1})-L_{n}(a^{*}_{n}) = ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT ) - italic_L start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT )
= ∑ t = 1 n ( L t ( a t − 1 * ) − L t − 1 ( a t − 1 * ) ) − L n ( a n * ) absent superscript subscript 𝑡 1 𝑛 subscript 𝐿 𝑡 subscript superscript 𝑎 𝑡 1 subscript 𝐿 𝑡 1 subscript superscript 𝑎 𝑡 1 subscript 𝐿 𝑛 subscript superscript 𝑎 𝑛 \displaystyle=\textstyle\sum_{t=1}^{n}(L_{t}(a^{*}_{t-1})-L_{t-1}(a^{*}_{t-1})%
)-L_{n}(a^{*}_{n}) = ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ( italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT ) - italic_L start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT ) ) - italic_L start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT )
= ∑ t = 1 n ( L t ( a t − 1 * ) − L t ( a t * ) ) + ∑ t = 1 n ( L t ( a t * ) − L t − 1 ( a t − 1 * ) ) − L n ( a n * ) absent superscript subscript 𝑡 1 𝑛 subscript 𝐿 𝑡 subscript superscript 𝑎 𝑡 1 subscript 𝐿 𝑡 subscript superscript 𝑎 𝑡 superscript subscript 𝑡 1 𝑛 subscript 𝐿 𝑡 subscript superscript 𝑎 𝑡 subscript 𝐿 𝑡 1 subscript superscript 𝑎 𝑡 1 subscript 𝐿 𝑛 subscript superscript 𝑎 𝑛 \displaystyle=\textstyle\sum_{t=1}^{n}(L_{t}(a^{*}_{t-1})-L_{t}(a^{*}_{t}))+%
\textstyle\sum_{t=1}^{n}(L_{t}(a^{*}_{t})-L_{t-1}(a^{*}_{t-1}))-L_{n}(a^{*}_{n}) = ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ( italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT ) - italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) ) + ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ( italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) - italic_L start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT ) ) - italic_L start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT )
= ( a ) ∑ t = 1 n ( L t ( a t − 1 * ) − L t ( a t * ) ) superscript 𝑎 absent superscript subscript 𝑡 1 𝑛 subscript 𝐿 𝑡 subscript superscript 𝑎 𝑡 1 subscript 𝐿 𝑡 subscript superscript 𝑎 𝑡 \displaystyle\stackrel{{\scriptstyle(a)}}{{=}}\textstyle\sum_{t=1}^{n}(L_{t}(a%
^{*}_{t-1})-L_{t}(a^{*}_{t})) start_RELOP SUPERSCRIPTOP start_ARG = end_ARG start_ARG ( italic_a ) end_ARG end_RELOP ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ( italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT ) - italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) )
where ( a ) 𝑎 (a) ( italic_a ) follows since ∑ t = 1 n ( L t ( a t * ) − L t − 1 ( a t − 1 * ) ) = L n ( a n * ) superscript subscript 𝑡 1 𝑛 subscript 𝐿 𝑡 subscript superscript 𝑎 𝑡 subscript 𝐿 𝑡 1 subscript superscript 𝑎 𝑡 1 subscript 𝐿 𝑛 subscript superscript 𝑎 𝑛 \sum_{t=1}^{n}(L_{t}(a^{*}_{t})-L_{t-1}(a^{*}_{t-1}))=L_{n}(a^{*}_{n}) ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ( italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) - italic_L start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT ) ) = italic_L start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ) by telescoping.
∎
Next, let Σ t 𝖥𝖳𝖫 subscript superscript Σ 𝖥𝖳𝖫 𝑡 \Sigma^{\mathsf{FTL}}_{t} roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT denote the anytime regret that 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL incurs up to round t 𝑡 t italic_t (i.e., assuming the game ends after round t 𝑡 t italic_t ). From Lemma 1 we have
Σ t 𝖥𝖳𝖫 : = Reg ( 𝖥𝖳𝖫 , ℓ t ) = ∑ i = 1 t ( L i ( a i − 1 * ) − L i ( a i * ) ) \displaystyle\Sigma^{\mathsf{FTL}}_{t}\mathrel{\mathop{\mathrel{\mathop{:}}}}=%
\mathrm{Reg}(\mathsf{FTL},\ell^{t})=\textstyle\sum_{i=1}^{t}(L_{i}(a^{*}_{i-1}%
)-L_{i}(a^{*}_{i})) roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT : = roman_Reg ( sansserif_FTL , roman_ℓ start_POSTSUPERSCRIPT italic_t end_POSTSUPERSCRIPT ) = ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_t end_POSTSUPERSCRIPT ( italic_L start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_i - 1 end_POSTSUBSCRIPT ) - italic_L start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) )
(4)
Now we make three critical observations:
•
Σ t 𝖥𝖳𝖫 subscript superscript Σ 𝖥𝖳𝖫 𝑡 \Sigma^{\mathsf{FTL}}_{t} roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT is adapted : it can be computed at the end of the t t h superscript 𝑡 𝑡 ℎ t^{th} italic_t start_POSTSUPERSCRIPT italic_t italic_h end_POSTSUPERSCRIPT round
•
Σ t 𝖥𝖳𝖫 subscript superscript Σ 𝖥𝖳𝖫 𝑡 \Sigma^{\mathsf{FTL}}_{t} roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT is monotone non-decreasing in t 𝑡 t italic_t (since by definition L i ( a i − 1 * ) − L i ( a i * ) ≥ 0 subscript 𝐿 𝑖 subscript superscript 𝑎 𝑖 1 subscript 𝐿 𝑖 subscript superscript 𝑎 𝑖 0 L_{i}(a^{*}_{i-1})-L_{i}(a^{*}_{i})\geq 0 italic_L start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_i - 1 end_POSTSUBSCRIPT ) - italic_L start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) ≥ 0 )
•
Σ t 𝖥𝖳𝖫 subscript superscript Σ 𝖥𝖳𝖫 𝑡 \Sigma^{\mathsf{FTL}}_{t} roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT is an anytime lower bound for Reg ( 𝖥𝖳𝖫 , ℓ n ) Reg 𝖥𝖳𝖫 superscript ℓ 𝑛 \mathrm{Reg}(\mathsf{FTL},\ell^{n}) roman_Reg ( sansserif_FTL , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) , with Σ n 𝖥𝖳𝖫 = Reg ( 𝖥𝖳𝖫 , ℓ n ) subscript superscript Σ 𝖥𝖳𝖫 𝑛 Reg 𝖥𝖳𝖫 superscript ℓ 𝑛 \Sigma^{\mathsf{FTL}}_{n}=\mathrm{Reg}(\mathsf{FTL},\ell^{n}) roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT = roman_Reg ( sansserif_FTL , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT )
For an input threshold θ ≥ 0 𝜃 0 \theta\geq 0 italic_θ ≥ 0 and an algorithm 𝖠𝖫𝖦 𝖶𝖢 subscript 𝖠𝖫𝖦 𝖶𝖢 \mathsf{ALG}_{\mathsf{WC}} sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT , we get the following (meta)algorithm.
Input: Policies
𝖥𝖳𝖫 , 𝖠𝖫𝖦 𝖶𝖢 𝖥𝖳𝖫 subscript 𝖠𝖫𝖦 𝖶𝖢
\mathsf{FTL},\mathsf{ALG}_{\mathsf{WC}} sansserif_FTL , sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT , threshold
θ 𝜃 \theta italic_θ
Initialize Σ 0 𝖥𝖳𝖫 = 0 subscript superscript Σ 𝖥𝖳𝖫 0 0 \Sigma^{\mathsf{FTL}}_{0}=0 roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT = 0 ,
t = 1 𝑡 1 t=1 italic_t = 1 ;
while Σ t − 1 𝖥𝖳𝖫 ≤ θ subscript superscript normal-Σ 𝖥𝖳𝖫 𝑡 1 𝜃 \Sigma^{\mathsf{FTL}}_{t-1}\leq\theta roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT ≤ italic_θ do
Set
a t = a t − 1 * subscript 𝑎 𝑡 superscript subscript 𝑎 𝑡 1 a_{t}=a_{t-1}^{*} italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT = italic_a start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ;
// Play 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL
Observe
ℓ t ( ⋅ ) subscript ℓ 𝑡 ⋅ \ell_{t}(\cdot) roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( ⋅ ) ;
Update
L t ( ⋅ ) = L t − 1 ( ⋅ ) + ℓ t ( ⋅ ) subscript 𝐿 𝑡 ⋅ subscript 𝐿 𝑡 1 ⋅ subscript ℓ 𝑡 ⋅ L_{t}(\cdot)=L_{t-1}(\cdot)+\ell_{t}(\cdot) italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( ⋅ ) = italic_L start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT ( ⋅ ) + roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( ⋅ ) and
Σ t 𝖥𝖳𝖫 = Σ t − 1 𝖥𝖳𝖫 + ( L t ( a t − 1 * ) − L t ( a t * ) ) subscript superscript Σ 𝖥𝖳𝖫 𝑡 subscript superscript Σ 𝖥𝖳𝖫 𝑡 1 subscript 𝐿 𝑡 superscript subscript 𝑎 𝑡 1 subscript 𝐿 𝑡 superscript subscript 𝑎 𝑡 \Sigma^{\mathsf{FTL}}_{t}=\Sigma^{\mathsf{FTL}}_{t-1}+(L_{t}(a_{t-1}^{*})-L_{t%
}(a_{t}^{*})) roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT = roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT + ( italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) - italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) ) and
t = t + 1 𝑡 𝑡 1 t=t+1 italic_t = italic_t + 1 ;
end while
Reset losses to
0 0 and play
𝖠𝖫𝖦 𝖶𝖢 subscript 𝖠𝖫𝖦 𝖶𝖢 \mathsf{ALG}_{\mathsf{WC}} sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT for remaining rounds (See
Figure 2 (b)) ;
Algorithm 1 Switching via Monotone Adapted Regret Traces (𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART )
We now have the following performance guarantee for Algorithm 1 for θ = g ( n ) 𝜃 𝑔 𝑛 \theta=g(n) italic_θ = italic_g ( italic_n ) .
Theorem 1 (Regret of 𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART with deterministic threshold).
We are given 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL , and any other policy 𝖠𝖫𝖦 𝖶𝖢 subscript 𝖠𝖫𝖦 𝖶𝖢 \mathsf{ALG}_{\mathsf{WC}} sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT with worst-case regret sup ℓ n Reg ( 𝖠𝖫𝖦 𝖶𝖢 , ℓ n ) ≤ g ( n ) subscript supremum superscript normal-ℓ 𝑛 normal-Reg subscript 𝖠𝖫𝖦 𝖶𝖢 superscript normal-ℓ 𝑛 𝑔 𝑛 \sup_{\ell^{n}}\mathrm{Reg}(\mathsf{ALG}_{\mathsf{WC}},\ell^{n})\leq g(n) roman_sup start_POSTSUBSCRIPT roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT end_POSTSUBSCRIPT roman_Reg ( sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) ≤ italic_g ( italic_n )
for some monotone function g 𝑔 g italic_g . Then, playing 𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART with threshold θ = g ( n ) 𝜃 𝑔 𝑛 \theta=g(n) italic_θ = italic_g ( italic_n ) ensures
Reg ( 𝖲𝖬𝖠𝖱𝖳 , ℓ n ) ≤ 2 min { Reg ( 𝖥𝖳𝖫 , ℓ n ) , g ( n ) } + 1 . Reg 𝖲𝖬𝖠𝖱𝖳 superscript ℓ 𝑛 2 Reg 𝖥𝖳𝖫 superscript ℓ 𝑛 𝑔 𝑛 1 \displaystyle\mathrm{Reg}(\mathsf{SMART},\ell^{n})\leq 2\min\{\mathrm{Reg}(%
\mathsf{FTL},\ell^{n}),g(n)\}+1. roman_Reg ( sansserif_SMART , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) ≤ 2 roman_min { roman_Reg ( sansserif_FTL , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) , italic_g ( italic_n ) } + 1 .
(5)
As we mention before, the structure of the 𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART algorithm (and the resulting guarantee) parallels the standard 2-competitive guarantee for the ski-rental problem (Karlin et al., 1994 ) .
This is a classical optimal stopping problem, where a principal faces an unknown horizon, and in each period must decide whether to rent a pair of skis for the period (for fixed cost $1) or buy the skis for the remaining horizon (for fixed cost $B 𝐵 B italic_B ). The aim is to design a policy which is minimax optimal (over the unknown horizon) with regards to the ratio of the cost paid by the principal, and the optimal cost in hindsight.
We further expand on this connection in Section 2.1 for the case of binary prediction. However, the connection suggests a natural follow-up question of whether randomized switching can help (as is the case for ski-rental); the following result answers this in the affirmative.
Theorem 2 (Regret of 𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART with Randomized Thresholds).
We are given 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL and any other policy 𝖠𝖫𝖦 𝖶𝖢 subscript 𝖠𝖫𝖦 𝖶𝖢 \mathsf{ALG}_{\mathsf{WC}} sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT with worst-case regret sup ℓ n Reg ( 𝖠𝖫𝖦 𝖶𝖢 , ℓ n ) ≤ g ( n ) subscript supremum superscript normal-ℓ 𝑛 normal-Reg subscript 𝖠𝖫𝖦 𝖶𝖢 superscript normal-ℓ 𝑛 𝑔 𝑛 \sup_{\ell^{n}}\mathrm{Reg}(\mathsf{ALG}_{\mathsf{WC}},\ell^{n})\leq g(n) roman_sup start_POSTSUBSCRIPT roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT end_POSTSUBSCRIPT roman_Reg ( sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) ≤ italic_g ( italic_n )
for some monotone function g 𝑔 g italic_g .
Moreover, given random sample U ∼ 𝑈𝑛𝑖𝑓 [ 0 , 1 ] similar-to 𝑈 𝑈𝑛𝑖𝑓 0 1 U\sim\text{Unif}[0,1] italic_U ∼ Unif [ 0 , 1 ] , suppose we set
θ = g ( n ) ln ( 1 + ( e − 1 ) U ) 𝜃 𝑔 𝑛 1 𝑒 1 𝑈 \displaystyle\theta=g(n)\ln(1+(e-1)U) italic_θ = italic_g ( italic_n ) roman_ln ( 1 + ( italic_e - 1 ) italic_U )
Then playing the 𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART policy (Algorithm 1 ) with random threshold θ 𝜃 \theta italic_θ ensures
𝔼 θ subscript 𝔼 𝜃 \displaystyle\operatorname{\mathbb{E}}_{\theta} blackboard_E start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT
[ Reg ( 𝖲𝖬𝖠𝖱𝖳 , ℓ n ) ] ≤ e e − 1 min { Reg ( 𝖥𝖳𝖫 , ℓ n ) , g ( n ) } + 1 . delimited-[] Reg 𝖲𝖬𝖠𝖱𝖳 superscript ℓ 𝑛 𝑒 𝑒 1 Reg 𝖥𝖳𝖫 superscript ℓ 𝑛 𝑔 𝑛 1 \displaystyle\left[\mathrm{Reg}(\mathsf{SMART},\ell^{n})\right]\leq\frac{e}{e-%
1}\min\{\mathrm{Reg}(\mathsf{FTL},\ell^{n}),g(n)\}+1. [ roman_Reg ( sansserif_SMART , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) ] ≤ divide start_ARG italic_e end_ARG start_ARG italic_e - 1 end_ARG roman_min { roman_Reg ( sansserif_FTL , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) , italic_g ( italic_n ) } + 1 .
While we state the above as a randomized switching policy, this is more for interpretability – it is easier to view our policy as switching between two black-box algorithms rather than playing a convex combination of the two. However, since we define the loss incurred by any 𝖠𝖫𝖦 𝖠𝖫𝖦 \mathsf{ALG} sansserif_ALG in round t 𝑡 t italic_t to be the expected loss when 𝖠𝖫𝖦 𝖠𝖫𝖦 \mathsf{ALG} sansserif_ALG plays a distribution w t subscript 𝑤 𝑡 w_{t} italic_w start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT over actions 𝒜 𝒜 \mathcal{A} caligraphic_A (see Section 1.1 ), therefore we can alternately implement the above by mixing between the actions of 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL and 𝖠𝖫𝖦 𝖶𝖢 subscript 𝖠𝖫𝖦 𝖶𝖢 \mathsf{ALG}_{\mathsf{WC}} sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT . More specifically, the above policy induces a monotone mixing rule, where over t 𝑡 t italic_t , the weight on the action suggested by 𝖠𝖫𝖦 𝖶𝖢 subscript 𝖠𝖫𝖦 𝖶𝖢 \mathsf{ALG}_{\mathsf{WC}} sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT is non-decreasing.
2.1 Illustrating the Reduction to Ski Rental in Binary Prediction
Before proving Theorems 1 and 2 , we first illustrate the basic idea of our approach and reduction for the binary prediction setting. This is aided by the observation that the regret of 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL in this setting admits a simple geometric interpretation: for any sequence, and any time t 𝑡 t italic_t , we have that Σ 𝖥𝖳𝖫 superscript Σ 𝖥𝖳𝖫 \Sigma^{\mathsf{FTL}} roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT is equal to 1/2 times the number of ‘lead changes’ up to time t 𝑡 t italic_t ; where a lead change is a time step i 𝑖 i italic_i where the count of 1 1 1 1 s and 0 0 s in the (sub)sequence y i − 1 superscript 𝑦 𝑖 1 y^{i-1} italic_y start_POSTSUPERSCRIPT italic_i - 1 end_POSTSUPERSCRIPT is equal (see Figure 2 (a)).
Corollary 2 (𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL for binary prediction).
In binary prediction, for any sequence y ∈ { 0 , 1 } n 𝑦 superscript 0 1 𝑛 y\in\{0,1\}^{n} italic_y ∈ { 0 , 1 } start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT and any time t ≤ n 𝑡 𝑛 t\leq n italic_t ≤ italic_n , define the lead-change counter
c ( y t ) : = | { 0 ≤ j ≤ t s . t . ∑ i = 1 j y i = ∑ i = 1 j ( 1 − y i ) } | . c(y^{t})\mathrel{\mathop{\mathrel{\mathop{:}}}}=|\{0\leq j\leq t~{}s.t.~{}\sum%
_{i=1}^{j}y_{i}=\sum_{i=1}^{j}(1-y_{i})\}|. italic_c ( italic_y start_POSTSUPERSCRIPT italic_t end_POSTSUPERSCRIPT ) : = | { 0 ≤ italic_j ≤ italic_t italic_s . italic_t . ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_j end_POSTSUPERSCRIPT italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT = ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_j end_POSTSUPERSCRIPT ( 1 - italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) } | .
Then we have Σ t 𝖥𝖳𝖫 = 1 2 c ( y t − 1 ) subscript superscript normal-Σ 𝖥𝖳𝖫 𝑡 1 2 𝑐 superscript 𝑦 𝑡 1 \Sigma^{\mathsf{FTL}}_{t}=\frac{1}{2}c(y^{t-1}) roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT = divide start_ARG 1 end_ARG start_ARG 2 end_ARG italic_c ( italic_y start_POSTSUPERSCRIPT italic_t - 1 end_POSTSUPERSCRIPT ) and thus Reg ( 𝖥𝖳𝖫 , y n ) = 1 2 c ( y n − 1 ) normal-Reg 𝖥𝖳𝖫 superscript 𝑦 𝑛 1 2 𝑐 superscript 𝑦 𝑛 1 \mathrm{Reg}(\mathsf{FTL},y^{n})=\frac{1}{2}c(y^{n-1}) roman_Reg ( sansserif_FTL , italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) = divide start_ARG 1 end_ARG start_ARG 2 end_ARG italic_c ( italic_y start_POSTSUPERSCRIPT italic_n - 1 end_POSTSUPERSCRIPT ) .
Corollary 2 follows from the regret decomposition in Lemma 1 . Furthermore, since the losses of 0s and 1s are equal at a lead change, it also follows that at a lead change t 𝑡 t italic_t , the anytime regret Σ t 𝖥𝖳𝖫 subscript superscript Σ 𝖥𝖳𝖫 𝑡 \Sigma^{\mathsf{FTL}}_{t} roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT is also equal to the hindsight regret incurred by 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL up to time t 𝑡 t italic_t . Since Σ t 𝖥𝖳𝖫 subscript superscript Σ 𝖥𝖳𝖫 𝑡 \Sigma^{\mathsf{FTL}}_{t} roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT only increases in value at lead changes, if the 𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART algorithm switches to 𝖠𝖫𝖦 𝖶𝖢 subscript 𝖠𝖫𝖦 𝖶𝖢 \mathsf{ALG}_{\mathsf{WC}} sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT , it will only happen after a lead change, and thus 𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART behaves as if it had oracle knowledge of the regret of 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL from just the history up to the current time.
Consider the instantiation of 𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART where 𝖠𝖫𝖦 𝖶𝖢 = 𝖢𝗈𝗏𝖾𝗋 subscript 𝖠𝖫𝖦 𝖶𝖢 𝖢𝗈𝗏𝖾𝗋 \mathsf{ALG}_{\mathsf{WC}}=\mathsf{Cover} sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT = sansserif_Cover and g ( n ) = n 2 π 𝑔 𝑛 𝑛 2 𝜋 g(n)=\sqrt{\frac{n}{2\pi}} italic_g ( italic_n ) = square-root start_ARG divide start_ARG italic_n end_ARG start_ARG 2 italic_π end_ARG end_ARG . As mentioned in Section 1 , a remarkable property of 𝖢𝗈𝗏𝖾𝗋 𝖢𝗈𝗏𝖾𝗋 \mathsf{Cover} sansserif_Cover is that it is the true minimax optimal algorithm, where Reg ( 𝖢𝗈𝗏𝖾𝗋 , y n ) = n 2 π ( 1 + o ( 1 ) ) Reg 𝖢𝗈𝗏𝖾𝗋 superscript 𝑦 𝑛 𝑛 2 𝜋 1 𝑜 1 \mathrm{Reg}(\mathsf{Cover},y^{n})=\sqrt{\frac{n}{2\pi}}(1+o(1)) roman_Reg ( sansserif_Cover , italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) = square-root start_ARG divide start_ARG italic_n end_ARG start_ARG 2 italic_π end_ARG end_ARG ( 1 + italic_o ( 1 ) ) for all sequences y n superscript 𝑦 𝑛 y^{n} italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT , such that g ( n ) 𝑔 𝑛 g(n) italic_g ( italic_n ) is nearly equal to Reg ( 𝖢𝗈𝗏𝖾𝗋 , y n ) Reg 𝖢𝗈𝗏𝖾𝗋 superscript 𝑦 𝑛 \mathrm{Reg}(\mathsf{Cover},y^{n}) roman_Reg ( sansserif_Cover , italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) regardless of the sequence (Cover, 1966 ) .
It follows that 𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART is equivalent to an algorithm which starts with 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL , plays it until the regret of 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL matches the minimax regret guaranteed by 𝖢𝗈𝗏𝖾𝗋 𝖢𝗈𝗏𝖾𝗋 \mathsf{Cover} sansserif_Cover for the remaining sequence, and then switches to 𝖢𝗈𝗏𝖾𝗋 𝖢𝗈𝗏𝖾𝗋 \mathsf{Cover} sansserif_Cover until the end.
Let t sw subscript 𝑡 sw t_{\mathrm{sw}} italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT denote the last round 𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART plays 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL before switching to 𝖠𝖫𝖦 𝖶𝖢 subscript 𝖠𝖫𝖦 𝖶𝖢 \mathsf{ALG}_{\mathsf{WC}} sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT (with t sw = n subscript 𝑡 sw 𝑛 t_{\mathrm{sw}}=n italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT = italic_n if it never switches).
For a single switch algorithm, the sequence which maximizes the regret is one that maximizes the 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL regret before the switch at t sw subscript 𝑡 sw t_{\mathrm{sw}} italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT , and minimizes the 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL after the switch, as depicted in Figure 2 (a). For such a sequence, Σ t 𝖥𝖳𝖫 = t / 4 subscript superscript Σ 𝖥𝖳𝖫 𝑡 𝑡 4 \Sigma^{\mathsf{FTL}}_{t}=t/4 roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT = italic_t / 4 at lead changes t 𝑡 t italic_t , such that the regret incurred by the algorithm is linear before the switch, matching the linear cost of renting skis in the ski rental problem. Note that t sw subscript 𝑡 sw t_{\mathrm{sw}} italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT will necessarily be o ( n ) 𝑜 𝑛 o(n) italic_o ( italic_n ) in such sequences as the time it takes until Σ t 𝖥𝖳𝖫 ≥ g ( n ) subscript superscript Σ 𝖥𝖳𝖫 𝑡 𝑔 𝑛 \Sigma^{\mathsf{FTL}}_{t}\geq g(n) roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ≥ italic_g ( italic_n ) is linear in g ( n ) = o ( n ) 𝑔 𝑛 𝑜 𝑛 g(n)=o(n) italic_g ( italic_n ) = italic_o ( italic_n ) . After the switch, 𝖢𝗈𝗏𝖾𝗋 𝖢𝗈𝗏𝖾𝗋 \mathsf{Cover} sansserif_Cover will incur regret n − t sw 2 π ( 1 + o ( 1 ) ) = g ( n ) ( 1 + o ( 1 ) ) 𝑛 subscript 𝑡 sw 2 𝜋 1 𝑜 1 𝑔 𝑛 1 𝑜 1 \sqrt{\frac{n-t_{\mathrm{sw}}}{2\pi}}(1+o(1))=g(n)(1+o(1)) square-root start_ARG divide start_ARG italic_n - italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT end_ARG start_ARG 2 italic_π end_ARG end_ARG ( 1 + italic_o ( 1 ) ) = italic_g ( italic_n ) ( 1 + italic_o ( 1 ) ) , matching the fixed cost of buying skis at the switching point; Corollary 3 follows as a result of this analysis. Furthermore, in the binary prediction setting, 𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART achieves the stronger notion of instance optimality stated in Definition 1 .
Corollary 3 .
For all y n ∈ { 0 , 1 } n superscript 𝑦 𝑛 superscript 0 1 𝑛 y^{n}\in\{0,1\}^{n} italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ∈ { 0 , 1 } start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT , 𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART with 𝖠𝖫𝖦 𝖶𝖢 = 𝖢𝗈𝗏𝖾𝗋 subscript 𝖠𝖫𝖦 𝖶𝖢 𝖢𝗈𝗏𝖾𝗋 \mathsf{ALG}_{\mathsf{WC}}=\mathsf{Cover} sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT = sansserif_Cover and θ = n 2 π 𝜃 𝑛 2 𝜋 \theta=\sqrt{\frac{n}{2\pi}} italic_θ = square-root start_ARG divide start_ARG italic_n end_ARG start_ARG 2 italic_π end_ARG end_ARG satisfies
Reg ( 𝖲𝖬𝖠𝖱𝖳 , y n ) Reg 𝖲𝖬𝖠𝖱𝖳 superscript 𝑦 𝑛 \displaystyle\mathrm{Reg}(\mathsf{SMART},y^{n}) roman_Reg ( sansserif_SMART , italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT )
≤ 2 min { Reg ( 𝖥𝖳𝖫 , y n ) , Reg ( 𝖢𝗈𝗏𝖾𝗋 , y n ) } + 1 . absent 2 Reg 𝖥𝖳𝖫 superscript 𝑦 𝑛 Reg 𝖢𝗈𝗏𝖾𝗋 superscript 𝑦 𝑛 1 \displaystyle\leq 2\min\{\mathrm{Reg}(\mathsf{FTL},y^{n}),\mathrm{Reg}(\mathsf%
{Cover},y^{n})\}+1. ≤ 2 roman_min { roman_Reg ( sansserif_FTL , italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) , roman_Reg ( sansserif_Cover , italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) } + 1 .
𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART with 𝖠𝖫𝖦 𝖶𝖢 = 𝖢𝗈𝗏𝖾𝗋 subscript 𝖠𝖫𝖦 𝖶𝖢 𝖢𝗈𝗏𝖾𝗋 \mathsf{ALG}_{\mathsf{WC}}=\mathsf{Cover} sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT = sansserif_Cover and θ = n 2 π ln ( 1 + ( e − 1 ) U ) 𝜃 𝑛 2 𝜋 1 𝑒 1 𝑈 \theta=\sqrt{\frac{n}{2\pi}}\ln(1+(e-1)U) italic_θ = square-root start_ARG divide start_ARG italic_n end_ARG start_ARG 2 italic_π end_ARG end_ARG roman_ln ( 1 + ( italic_e - 1 ) italic_U ) for U ∼ 𝑈𝑛𝑖𝑓𝑜𝑟𝑚 [ 0 , 1 ] similar-to 𝑈 𝑈𝑛𝑖𝑓𝑜𝑟𝑚 0 1 U\sim\text{Uniform}[0,1] italic_U ∼ Uniform [ 0 , 1 ] satisfies
Reg ( 𝖲𝖬𝖠𝖱𝖳 , y n ) Reg 𝖲𝖬𝖠𝖱𝖳 superscript 𝑦 𝑛 \displaystyle\mathrm{Reg}(\mathsf{SMART},y^{n}) roman_Reg ( sansserif_SMART , italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT )
≤ 1.58 min { Reg ( 𝖥𝖳𝖫 , y n ) , Reg ( 𝖢𝗈𝗏𝖾𝗋 , y n ) } + 1 . absent 1.58 Reg 𝖥𝖳𝖫 superscript 𝑦 𝑛 Reg 𝖢𝗈𝗏𝖾𝗋 superscript 𝑦 𝑛 1 \displaystyle\leq 1.58\min\{\mathrm{Reg}(\mathsf{FTL},y^{n}),\mathrm{Reg}(%
\mathsf{Cover},y^{n})\}+1. ≤ 1.58 roman_min { roman_Reg ( sansserif_FTL , italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) , roman_Reg ( sansserif_Cover , italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) } + 1 .
Figure 2: Figure 2 (a) on the left shows the worst case instance in binary prediction for an algorithm which starts with 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL and switches at most once during the time horizon to 𝖢𝗈𝗏𝖾𝗋 𝖢𝗈𝗏𝖾𝗋 \mathsf{Cover} sansserif_Cover . Figure2 (b) on the right depicts in a prediction with experts setting how 𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART resets the losses after the switch from 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL to 𝖠𝖫𝖦 𝖶𝖢 subscript 𝖠𝖫𝖦 𝖶𝖢 \mathsf{ALG}_{\mathsf{WC}} sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT .
2.2 Proofs for General Online Learning
In the general online learning setting, the proof is nearly the same, with the reduction to the ski-rental problem captured by Lemma 2 .
Lemma 2 .
Let t sw : = min 1 ≤ t ≤ n − 1 Σ t 𝖥𝖳𝖫 > θ t_{\mathrm{sw}}\mathrel{\mathop{\mathrel{\mathop{:}}}}=\min_{1\leq t\leq n-1}%
\Sigma^{\mathsf{FTL}}_{t}>\theta italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT : = roman_min start_POSTSUBSCRIPT 1 ≤ italic_t ≤ italic_n - 1 end_POSTSUBSCRIPT roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT > italic_θ denote the last round 𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART plays 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL before switching to 𝖠𝖫𝖦 𝖶𝖢 subscript 𝖠𝖫𝖦 𝖶𝖢 \mathsf{ALG}_{\mathsf{WC}} sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT (with t sw = n subscript 𝑡 normal-sw 𝑛 t_{\mathrm{sw}}=n italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT = italic_n if it never switches). 𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART incurs regret bounded by
Reg ( 𝖲𝖬𝖠𝖱𝖳 , ℓ n ) ≤ Reg ( 𝖥𝖳𝖫 , ℓ t sw ) + Reg ( 𝖠𝖫𝖦 𝖶𝖢 , ℓ t sw + 1 n ) ≤ θ + Reg ( 𝖠𝖫𝖦 𝖶𝖢 , ℓ t sw + 1 n ) + 1 . Reg 𝖲𝖬𝖠𝖱𝖳 superscript ℓ 𝑛 Reg 𝖥𝖳𝖫 superscript ℓ subscript 𝑡 sw Reg subscript 𝖠𝖫𝖦 𝖶𝖢 superscript subscript ℓ subscript 𝑡 sw 1 𝑛 𝜃 Reg subscript 𝖠𝖫𝖦 𝖶𝖢 superscript subscript ℓ subscript 𝑡 sw 1 𝑛 1 \mathrm{Reg}(\mathsf{SMART},\ell^{n})\leq\mathrm{Reg}(\mathsf{FTL},\ell^{t_{%
\mathrm{sw}}})+\mathrm{Reg}(\mathsf{ALG}_{\mathsf{WC}},\ell_{t_{\mathrm{sw}}+1%
}^{n})\leq\theta+\mathrm{Reg}(\mathsf{ALG}_{\mathsf{WC}},\ell_{t_{\mathrm{sw}}%
+1}^{n})+1. roman_Reg ( sansserif_SMART , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) ≤ roman_Reg ( sansserif_FTL , roman_ℓ start_POSTSUPERSCRIPT italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT end_POSTSUPERSCRIPT ) + roman_Reg ( sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT , roman_ℓ start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) ≤ italic_θ + roman_Reg ( sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT , roman_ℓ start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) + 1 .
Proof.
We separately bound the regret of 𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART before the switch and after the switch,
Reg ( 𝖲𝖬𝖠𝖱𝖳 , ℓ n ) Reg 𝖲𝖬𝖠𝖱𝖳 superscript ℓ 𝑛 \displaystyle\mathrm{Reg}(\mathsf{SMART},\ell^{n}) roman_Reg ( sansserif_SMART , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT )
= ( ∑ t = 1 t sw ℓ t ( a t ) − ∑ t = 1 t sw ℓ t ( a n * ) ) + ( ∑ t = t sw + 1 n ℓ t ( a t ) − ∑ t = t sw + 1 n ℓ t ( a n * ) ) absent superscript subscript 𝑡 1 subscript 𝑡 sw subscript ℓ 𝑡 subscript 𝑎 𝑡 superscript subscript 𝑡 1 subscript 𝑡 sw subscript ℓ 𝑡 subscript superscript 𝑎 𝑛 superscript subscript 𝑡 subscript 𝑡 sw 1 𝑛 subscript ℓ 𝑡 subscript 𝑎 𝑡 superscript subscript 𝑡 subscript 𝑡 sw 1 𝑛 subscript ℓ 𝑡 subscript superscript 𝑎 𝑛 \displaystyle=\big{(}\textstyle\sum_{t=1}^{t_{\mathrm{sw}}}\ell_{t}(a_{t})-%
\textstyle\sum_{t=1}^{t_{\mathrm{sw}}}\ell_{t}(a^{*}_{n})\big{)}+\big{(}%
\textstyle\sum_{t=t_{\mathrm{sw}}+1}^{n}\ell_{t}(a_{t})-\textstyle\sum_{t=t_{%
\mathrm{sw}}+1}^{n}\ell_{t}(a^{*}_{n})\big{)} = ( ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT end_POSTSUPERSCRIPT roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) - ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT end_POSTSUPERSCRIPT roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ) ) + ( ∑ start_POSTSUBSCRIPT italic_t = italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) - ∑ start_POSTSUBSCRIPT italic_t = italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ) )
≤ ( a ) ( ∑ t = 1 t sw ℓ t ( a t ) − ∑ t = 1 t sw ℓ t ( a t sw * ) ) + ( ∑ t = t sw + 1 n ℓ t ( a t ) − ∑ t = t sw + 1 n ℓ t ( a t sw + 1 : n * ) ) superscript 𝑎 absent superscript subscript 𝑡 1 subscript 𝑡 sw subscript ℓ 𝑡 subscript 𝑎 𝑡 superscript subscript 𝑡 1 subscript 𝑡 sw subscript ℓ 𝑡 subscript superscript 𝑎 subscript 𝑡 sw superscript subscript 𝑡 subscript 𝑡 sw 1 𝑛 subscript ℓ 𝑡 subscript 𝑎 𝑡 superscript subscript 𝑡 subscript 𝑡 sw 1 𝑛 subscript ℓ 𝑡 subscript superscript 𝑎 : subscript 𝑡 sw 1 𝑛 \displaystyle\stackrel{{\scriptstyle(a)}}{{\leq}}\big{(}\textstyle\sum_{t=1}^{%
t_{\mathrm{sw}}}\ell_{t}(a_{t})-\textstyle\sum_{t=1}^{t_{\mathrm{sw}}}\ell_{t}%
(a^{*}_{t_{\mathrm{sw}}})\big{)}+\big{(}\textstyle\sum_{t=t_{\mathrm{sw}}+1}^{%
n}\ell_{t}(a_{t})-\textstyle\sum_{t=t_{\mathrm{sw}}+1}^{n}\ell_{t}(a^{*}_{t_{%
\mathrm{sw}}+1\mathrel{\mathop{:}}n})\big{)} start_RELOP SUPERSCRIPTOP start_ARG ≤ end_ARG start_ARG ( italic_a ) end_ARG end_RELOP ( ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT end_POSTSUPERSCRIPT roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) - ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT end_POSTSUPERSCRIPT roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT end_POSTSUBSCRIPT ) ) + ( ∑ start_POSTSUBSCRIPT italic_t = italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) - ∑ start_POSTSUBSCRIPT italic_t = italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT + 1 : italic_n end_POSTSUBSCRIPT ) )
= Reg ( 𝖥𝖳𝖫 , ℓ t sw ) + Reg ( 𝖠𝖫𝖦 𝖶𝖢 , ℓ t sw + 1 n ) . absent Reg 𝖥𝖳𝖫 superscript ℓ subscript 𝑡 sw Reg subscript 𝖠𝖫𝖦 𝖶𝖢 superscript subscript ℓ subscript 𝑡 sw 1 𝑛 \displaystyle=\mathrm{Reg}(\mathsf{FTL},\ell^{t_{\mathrm{sw}}})+\mathrm{Reg}(%
\mathsf{ALG}_{\mathsf{WC}},\ell_{t_{\mathrm{sw}}+1}^{n}). = roman_Reg ( sansserif_FTL , roman_ℓ start_POSTSUPERSCRIPT italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT end_POSTSUPERSCRIPT ) + roman_Reg ( sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT , roman_ℓ start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) .
The first term is upper bounded by Reg ( 𝖥𝖳𝖫 , ℓ t sw ) Reg 𝖥𝖳𝖫 superscript ℓ subscript 𝑡 sw \mathrm{Reg}(\mathsf{FTL},\ell^{t_{\mathrm{sw}}}) roman_Reg ( sansserif_FTL , roman_ℓ start_POSTSUPERSCRIPT italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT end_POSTSUPERSCRIPT ) since L t sw ( a n * ) ≥ L t sw ( a t sw * ) subscript 𝐿 subscript 𝑡 sw subscript superscript 𝑎 𝑛 subscript 𝐿 subscript 𝑡 sw subscript superscript 𝑎 subscript 𝑡 sw L_{t_{\mathrm{sw}}}(a^{*}_{n})\geq L_{t_{\mathrm{sw}}}(a^{*}_{t_{\mathrm{sw}}}) italic_L start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ) ≥ italic_L start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT end_POSTSUBSCRIPT ) by definition, as a t sw * subscript superscript 𝑎 subscript 𝑡 sw a^{*}_{t_{\mathrm{sw}}} italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT end_POSTSUBSCRIPT is the minimizer of L t sw subscript 𝐿 subscript 𝑡 sw L_{t_{\mathrm{sw}}} italic_L start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT end_POSTSUBSCRIPT , and furthermore 𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART always plays according to 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL in rounds up to t sw subscript 𝑡 sw t_{\mathrm{sw}} italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT . The second term is upper bounded by Reg ( 𝖠𝖫𝖦 𝖶𝖢 , ℓ t sw + 1 n ) Reg subscript 𝖠𝖫𝖦 𝖶𝖢 superscript subscript ℓ subscript 𝑡 sw 1 𝑛 \mathrm{Reg}(\mathsf{ALG}_{\mathsf{WC}},\ell_{t_{\mathrm{sw}}+1}^{n}) roman_Reg ( sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT , roman_ℓ start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) because ∑ t = t sw + 1 n ℓ t ( a t sw + 1 : n * ) ≤ ∑ t = t sw + 1 n ℓ t ( a n * ) superscript subscript 𝑡 subscript 𝑡 sw 1 𝑛 subscript ℓ 𝑡 subscript superscript 𝑎 : subscript 𝑡 sw 1 𝑛 superscript subscript 𝑡 subscript 𝑡 sw 1 𝑛 subscript ℓ 𝑡 subscript superscript 𝑎 𝑛 \sum_{t=t_{\mathrm{sw}}+1}^{n}\ell_{t}(a^{*}_{t_{\mathrm{sw}}+1\mathrel{%
\mathop{:}}n})\leq\sum_{t=t_{\mathrm{sw}}+1}^{n}\ell_{t}(a^{*}_{n}) ∑ start_POSTSUBSCRIPT italic_t = italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT + 1 : italic_n end_POSTSUBSCRIPT ) ≤ ∑ start_POSTSUBSCRIPT italic_t = italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ) as a t sw + 1 : n * subscript superscript 𝑎 : subscript 𝑡 sw 1 𝑛 a^{*}_{t_{\mathrm{sw}}+1\mathrel{\mathop{:}}n} italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT + 1 : italic_n end_POSTSUBSCRIPT is the minimizer of the losses after t sw subscript 𝑡 sw t_{\mathrm{sw}} italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT , and at time t > t sw 𝑡 subscript 𝑡 sw t>t_{\mathrm{sw}} italic_t > italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT , 𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART plays according to 𝖠𝖫𝖦 𝖶𝖢 subscript 𝖠𝖫𝖦 𝖶𝖢 \mathsf{ALG}_{\mathsf{WC}} sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT on the sequence of losses limited to ℓ t + 1 n superscript subscript ℓ 𝑡 1 𝑛 \ell_{t+1}^{n} roman_ℓ start_POSTSUBSCRIPT italic_t + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT . This illustrates the important role of resetting the losses after the switch as depicted in Figure 2 (b).
Using the fact that Σ t sw − 1 𝖥𝖳𝖫 ≤ θ subscript superscript Σ 𝖥𝖳𝖫 subscript 𝑡 sw 1 𝜃 \Sigma^{\mathsf{FTL}}_{t_{\mathrm{sw}}-1}\leq\theta roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT - 1 end_POSTSUBSCRIPT ≤ italic_θ and L t sw − 1 ( x t sw − 1 * ) ≤ L t sw − 1 ( x t sw * ) subscript 𝐿 subscript 𝑡 sw 1 subscript superscript 𝑥 subscript 𝑡 sw 1 subscript 𝐿 subscript 𝑡 sw 1 subscript superscript 𝑥 subscript 𝑡 sw L_{t_{\mathrm{sw}}-1}(x^{*}_{t_{\mathrm{sw}}-1})\leq L_{t_{\mathrm{sw}}-1}(x^{%
*}_{t_{\mathrm{sw}}}) italic_L start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT - 1 end_POSTSUBSCRIPT ( italic_x start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT - 1 end_POSTSUBSCRIPT ) ≤ italic_L start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT - 1 end_POSTSUBSCRIPT ( italic_x start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT end_POSTSUBSCRIPT ) , it follows that
Reg ( 𝖥𝖳𝖫 , ℓ t sw ) = Σ t sw − 1 𝖥𝖳𝖫 + L t sw − 1 ( a t sw − 1 * ) − L t sw − 1 ( x t sw * ) + ℓ t sw ( a t sw − 1 * ) − ℓ t sw ( x t sw * ) ≤ θ + 1 . Reg 𝖥𝖳𝖫 superscript ℓ subscript 𝑡 sw subscript superscript Σ 𝖥𝖳𝖫 subscript 𝑡 sw 1 subscript 𝐿 subscript 𝑡 sw 1 subscript superscript 𝑎 subscript 𝑡 sw 1 subscript 𝐿 subscript 𝑡 sw 1 subscript superscript 𝑥 subscript 𝑡 sw subscript ℓ subscript 𝑡 sw subscript superscript 𝑎 subscript 𝑡 sw 1 subscript ℓ subscript 𝑡 sw subscript superscript 𝑥 subscript 𝑡 sw 𝜃 1 ~{}~{}~{}~{}~{}~{}\mathrm{Reg}(\mathsf{FTL},\ell^{t_{\mathrm{sw}}})=\Sigma^{%
\mathsf{FTL}}_{t_{\mathrm{sw}}-1}+L_{t_{\mathrm{sw}}-1}(a^{*}_{t_{\mathrm{sw}}%
-1})-L_{t_{\mathrm{sw}}-1}(x^{*}_{t_{\mathrm{sw}}})+\ell_{t_{\mathrm{sw}}}(a^{%
*}_{t_{\mathrm{sw}}-1})-\ell_{t_{\mathrm{sw}}}(x^{*}_{t_{\mathrm{sw}}})\leq%
\theta+1. roman_Reg ( sansserif_FTL , roman_ℓ start_POSTSUPERSCRIPT italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT end_POSTSUPERSCRIPT ) = roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT - 1 end_POSTSUBSCRIPT + italic_L start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT - 1 end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT - 1 end_POSTSUBSCRIPT ) - italic_L start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT - 1 end_POSTSUBSCRIPT ( italic_x start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT end_POSTSUBSCRIPT ) + roman_ℓ start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT - 1 end_POSTSUBSCRIPT ) - roman_ℓ start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( italic_x start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT end_POSTSUBSCRIPT ) ≤ italic_θ + 1 .
∎
The reduction to ski-rental is again immediate due to the properties that Σ t 𝖥𝖳𝖫 : = Reg ( 𝖥𝖳𝖫 , ℓ t sw ) \Sigma^{\mathsf{FTL}}_{t}\mathrel{\mathop{:}}=\mathrm{Reg}(\mathsf{FTL},\ell^{%
t_{\mathrm{sw}}}) roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT : = roman_Reg ( sansserif_FTL , roman_ℓ start_POSTSUPERSCRIPT italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT end_POSTSUPERSCRIPT ) is adapted, monotone, and is an anytime lower bound for Reg ( 𝖥𝖳𝖫 , ℓ n ) Reg 𝖥𝖳𝖫 superscript ℓ 𝑛 \mathrm{Reg}(\mathsf{FTL},\ell^{n}) roman_Reg ( sansserif_FTL , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) , while remaining an upper bound on the true regret incurred by 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL up to time t 𝑡 t italic_t . As a result, the algorithm can pretend that it truly observes the regret it incurs at each time up to the switching time t sw subscript 𝑡 sw t_{\mathrm{sw}} italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT . After the switch, 𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART incurs regret Reg ( 𝖠𝖫𝖦 𝖶𝖢 , ℓ t sw + 1 n ) Reg subscript 𝖠𝖫𝖦 𝖶𝖢 superscript subscript ℓ subscript 𝑡 sw 1 𝑛 \mathrm{Reg}(\mathsf{ALG}_{\mathsf{WC}},\ell_{t_{\mathrm{sw}}+1}^{n}) roman_Reg ( sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT , roman_ℓ start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) which is upper bounded by g ( n − t sw ) ≤ g ( n ) 𝑔 𝑛 subscript 𝑡 sw 𝑔 𝑛 g(n-t_{\mathrm{sw}})\leq g(n) italic_g ( italic_n - italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT ) ≤ italic_g ( italic_n ) by assumption.
Proof of Theorem 1 .
This follows immediately from Lemma 2 . For ℓ n superscript ℓ 𝑛 \ell^{n} roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT such that Reg ( 𝖥𝖳𝖫 , ℓ n ) ≤ g ( n ) Reg 𝖥𝖳𝖫 superscript ℓ 𝑛 𝑔 𝑛 \mathrm{Reg}(\mathsf{FTL},\ell^{n})\leq g(n) roman_Reg ( sansserif_FTL , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) ≤ italic_g ( italic_n ) , 𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART will never switch to 𝖢𝗈𝗏𝖾𝗋 𝖢𝗈𝗏𝖾𝗋 \mathsf{Cover} sansserif_Cover as Σ t 𝖥𝖳𝖫 ≤ Reg ( 𝖥𝖳𝖫 , ℓ n ) ≤ g ( n ) subscript superscript Σ 𝖥𝖳𝖫 𝑡 Reg 𝖥𝖳𝖫 superscript ℓ 𝑛 𝑔 𝑛 \Sigma^{\mathsf{FTL}}_{t}\leq\mathrm{Reg}(\mathsf{FTL},\ell^{n})\leq g(n) roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ≤ roman_Reg ( sansserif_FTL , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) ≤ italic_g ( italic_n ) such that Reg ( 𝖲𝖬𝖠𝖱𝖳 , ℓ n ) = min { Reg ( 𝖥𝖳𝖫 , ℓ n ) , g ( n ) } Reg 𝖲𝖬𝖠𝖱𝖳 superscript ℓ 𝑛 Reg 𝖥𝖳𝖫 superscript ℓ 𝑛 𝑔 𝑛 \mathrm{Reg}(\mathsf{SMART},\ell^{n})=\min\{\mathrm{Reg}(\mathsf{FTL},\ell^{n}%
),g(n)\} roman_Reg ( sansserif_SMART , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) = roman_min { roman_Reg ( sansserif_FTL , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) , italic_g ( italic_n ) } . For ℓ n superscript ℓ 𝑛 \ell^{n} roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT such that Reg ( 𝖥𝖳𝖫 , ℓ n ) > g ( n ) Reg 𝖥𝖳𝖫 superscript ℓ 𝑛 𝑔 𝑛 \mathrm{Reg}(\mathsf{FTL},\ell^{n})>g(n) roman_Reg ( sansserif_FTL , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) > italic_g ( italic_n ) , by Lemma 2 , Reg ( 𝖲𝖬𝖠𝖱𝖳 , ℓ n ) ≤ g ( n ) + g ( n − t sw ) + 1 ≤ 2 g ( n ) + 1 Reg 𝖲𝖬𝖠𝖱𝖳 superscript ℓ 𝑛 𝑔 𝑛 𝑔 𝑛 subscript 𝑡 sw 1 2 𝑔 𝑛 1 \mathrm{Reg}(\mathsf{SMART},\ell^{n})\leq g(n)+g(n-t_{\mathrm{sw}})+1\leq 2g(n%
)+1 roman_Reg ( sansserif_SMART , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) ≤ italic_g ( italic_n ) + italic_g ( italic_n - italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT ) + 1 ≤ 2 italic_g ( italic_n ) + 1 .
∎
Proof of Theorem 2 .
The proof uses a primal-dual approach, similar to that of Karlin et al. (1994 ) for ski-rental.
For a given sequence of loss functions ℓ n superscript ℓ 𝑛 \ell^{n} roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT , we use the shorthand r = Reg ( 𝖥𝖳𝖫 , ℓ n ) 𝑟 Reg 𝖥𝖳𝖫 superscript ℓ 𝑛 r=\mathrm{Reg}(\mathsf{FTL},\ell^{n}) italic_r = roman_Reg ( sansserif_FTL , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) and g = g ( n ) 𝑔 𝑔 𝑛 g=g(n) italic_g = italic_g ( italic_n ) .
Also, for our given choice of cumulative distribution function F n subscript 𝐹 𝑛 F_{n} italic_F start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT , the corresponding probability density function is given by f ( z ) = e z / g g ( e − 1 ) 𝑓 𝑧 superscript 𝑒 𝑧 𝑔 𝑔 𝑒 1 f(z)=\frac{e^{z/g}}{g(e-1)} italic_f ( italic_z ) = divide start_ARG italic_e start_POSTSUPERSCRIPT italic_z / italic_g end_POSTSUPERSCRIPT end_ARG start_ARG italic_g ( italic_e - 1 ) end_ARG for z ∈ [ 0 , g ] 𝑧 0 𝑔 z\in[0,g] italic_z ∈ [ 0 , italic_g ] .
As before, let t sw : = min 1 ≤ t ≤ n − 1 Σ t 𝖥𝖳𝖫 > θ t_{\mathrm{sw}}\mathrel{\mathop{\mathrel{\mathop{:}}}}=\min_{1\leq t\leq n-1}%
\Sigma^{\mathsf{FTL}}_{t}>\theta italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT : = roman_min start_POSTSUBSCRIPT 1 ≤ italic_t ≤ italic_n - 1 end_POSTSUBSCRIPT roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT > italic_θ be the (random) round where 𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART switches from 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL to 𝖠𝖫𝖦 𝖶𝖢 subscript 𝖠𝖫𝖦 𝖶𝖢 \mathsf{ALG}_{\mathsf{WC}} sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT (with t sw = n subscript 𝑡 sw 𝑛 t_{\mathrm{sw}}=n italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT = italic_n if it never switches). Then by Lemma 2 we have
Reg ( 𝖲𝖬𝖠𝖱𝖳 , ℓ n ) − 1 min { Reg ( 𝖥𝖳𝖫 , ℓ n ) , g ( n ) } ≤ { θ + g min { r , g } if t sw < n 1 if t sw = n Reg 𝖲𝖬𝖠𝖱𝖳 superscript ℓ 𝑛 1 Reg 𝖥𝖳𝖫 superscript ℓ 𝑛 𝑔 𝑛 cases 𝜃 𝑔 𝑟 𝑔 if subscript 𝑡 sw 𝑛 1 if subscript 𝑡 sw 𝑛 \displaystyle\frac{\mathrm{Reg}(\mathsf{SMART},\ell^{n})-1}{\min\{\mathrm{Reg}%
(\mathsf{FTL},\ell^{n}),g(n)\}}\leq\begin{cases}\frac{\theta+g}{\min\{r,g\}}&%
\text{if }t_{\mathrm{sw}}<n\\
1&\text{if }t_{\mathrm{sw}}=n\end{cases} divide start_ARG roman_Reg ( sansserif_SMART , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) - 1 end_ARG start_ARG roman_min { roman_Reg ( sansserif_FTL , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) , italic_g ( italic_n ) } end_ARG ≤ { start_ROW start_CELL divide start_ARG italic_θ + italic_g end_ARG start_ARG roman_min { italic_r , italic_g } end_ARG end_CELL start_CELL if italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT < italic_n end_CELL end_ROW start_ROW start_CELL 1 end_CELL start_CELL if italic_t start_POSTSUBSCRIPT roman_sw end_POSTSUBSCRIPT = italic_n end_CELL end_ROW
where the second case follows from the fact that θ ∈ [ 0 , g ] 𝜃 0 𝑔 \theta\in[0,g] italic_θ ∈ [ 0 , italic_g ] , and hence if we never switch, then r ≤ g ( n ) 𝑟 𝑔 𝑛 r\leq g(n) italic_r ≤ italic_g ( italic_n ) .
Taking expectation over θ 𝜃 \theta italic_θ , we have
𝔼 θ [ Reg ( 𝖲𝖬𝖠𝖱𝖳 , ℓ n ) ] − 1 min { Reg ( 𝖥𝖳𝖫 , ℓ n ) , g } ≤ { ∫ 0 r x + g r f ( x ) 𝑑 x + 1 − F ( r ) if r ≤ g ∫ 0 g x + g g f ( x ) 𝑑 x if r > g subscript 𝔼 𝜃 Reg 𝖲𝖬𝖠𝖱𝖳 superscript ℓ 𝑛 1 Reg 𝖥𝖳𝖫 superscript ℓ 𝑛 𝑔 cases superscript subscript 0 𝑟 𝑥 𝑔 𝑟 𝑓 𝑥 differential-d 𝑥 1 𝐹 𝑟 if 𝑟 𝑔 superscript subscript 0 𝑔 𝑥 𝑔 𝑔 𝑓 𝑥 differential-d 𝑥 if 𝑟 𝑔 \displaystyle\frac{\operatorname{\mathbb{E}}_{\theta}\left[\mathrm{Reg}(%
\mathsf{SMART},\ell^{n})\right]-1}{\min\{\mathrm{Reg}(\mathsf{FTL},\ell^{n}),g%
\}}\leq\begin{cases}\int_{0}^{r}\frac{x+g}{r}f(x)dx+1-F(r)&\text{if }r\leq g\\
\int_{0}^{g}\frac{x+g}{g}f(x)dx&\text{if }r>g\end{cases} divide start_ARG blackboard_E start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT [ roman_Reg ( sansserif_SMART , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) ] - 1 end_ARG start_ARG roman_min { roman_Reg ( sansserif_FTL , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) , italic_g } end_ARG ≤ { start_ROW start_CELL ∫ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_r end_POSTSUPERSCRIPT divide start_ARG italic_x + italic_g end_ARG start_ARG italic_r end_ARG italic_f ( italic_x ) italic_d italic_x + 1 - italic_F ( italic_r ) end_CELL start_CELL if italic_r ≤ italic_g end_CELL end_ROW start_ROW start_CELL ∫ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_g end_POSTSUPERSCRIPT divide start_ARG italic_x + italic_g end_ARG start_ARG italic_g end_ARG italic_f ( italic_x ) italic_d italic_x end_CELL start_CELL if italic_r > italic_g end_CELL end_ROW
Let ϕ ( z ) : = ∫ 0 z ( x + g ) z f ( x ) d x + 1 − F ( z ) \phi(z)\mathrel{\mathop{\mathrel{\mathop{:}}}}=\int_{0}^{z}\frac{(x+g)}{z}f(x)%
dx+1-F(z) italic_ϕ ( italic_z ) : = ∫ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_z end_POSTSUPERSCRIPT divide start_ARG ( italic_x + italic_g ) end_ARG start_ARG italic_z end_ARG italic_f ( italic_x ) italic_d italic_x + 1 - italic_F ( italic_z ) for z ∈ [ 0 , g ] 𝑧 0 𝑔 z\in[0,g] italic_z ∈ [ 0 , italic_g ] ; then 𝔼 θ [ Reg ( 𝖲𝖬𝖠𝖱𝖳 , ℓ n ) ] − 1 min { Reg ( 𝖥𝖳𝖫 , ℓ n ) , g } ≤ max z ∈ [ 0 , g ] ϕ ( z ) subscript 𝔼 𝜃 Reg 𝖲𝖬𝖠𝖱𝖳 superscript ℓ 𝑛 1 Reg 𝖥𝖳𝖫 superscript ℓ 𝑛 𝑔 subscript 𝑧 0 𝑔 italic-ϕ 𝑧 \frac{\operatorname{\mathbb{E}}_{\theta}\left[\mathrm{Reg}(\mathsf{SMART},\ell%
^{n})\right]-1}{\min\{\mathrm{Reg}(\mathsf{FTL},\ell^{n}),g\}}\leq\max_{z\in[0%
,g]}\phi(z) divide start_ARG blackboard_E start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT [ roman_Reg ( sansserif_SMART , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) ] - 1 end_ARG start_ARG roman_min { roman_Reg ( sansserif_FTL , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) , italic_g } end_ARG ≤ roman_max start_POSTSUBSCRIPT italic_z ∈ [ 0 , italic_g ] end_POSTSUBSCRIPT italic_ϕ ( italic_z ) . Moreover, we can differentiate to get z 2 ϕ ′ ( z ) = g z f ( z ) − ∫ 0 z ( x + g ) f ( x ) 𝑑 x superscript 𝑧 2 superscript italic-ϕ ′ 𝑧 𝑔 𝑧 𝑓 𝑧 superscript subscript 0 𝑧 𝑥 𝑔 𝑓 𝑥 differential-d 𝑥 z^{2}\phi^{\prime}(z)=gzf(z)-\int_{0}^{z}(x+g)f(x)dx italic_z start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT italic_ϕ start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ( italic_z ) = italic_g italic_z italic_f ( italic_z ) - ∫ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_z end_POSTSUPERSCRIPT ( italic_x + italic_g ) italic_f ( italic_x ) italic_d italic_x . Substituting our choice of f 𝑓 f italic_f in this expression, we get
z 2 d ϕ ( z ) d z superscript 𝑧 2 𝑑 italic-ϕ 𝑧 𝑑 𝑧 \displaystyle\frac{z^{2}d\phi(z)}{dz} divide start_ARG italic_z start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT italic_d italic_ϕ ( italic_z ) end_ARG start_ARG italic_d italic_z end_ARG
= z e z / g ( e − 1 ) − ∫ 0 z ( x + g ) e x / g g ( e − 1 ) 𝑑 x = z e z / g − g ∫ 0 z / g ( w + 1 ) e w 𝑑 w ( e − 1 ) = 0 absent 𝑧 superscript 𝑒 𝑧 𝑔 𝑒 1 superscript subscript 0 𝑧 𝑥 𝑔 superscript 𝑒 𝑥 𝑔 𝑔 𝑒 1 differential-d 𝑥 𝑧 superscript 𝑒 𝑧 𝑔 𝑔 superscript subscript 0 𝑧 𝑔 𝑤 1 superscript 𝑒 𝑤 differential-d 𝑤 𝑒 1 0 \displaystyle=\frac{ze^{z/g}}{(e-1)}-\int_{0}^{z}(x+g)\frac{e^{x/g}}{g(e-1)}dx%
=\frac{ze^{z/g}-g\int_{0}^{z/g}(w+1)e^{w}dw}{(e-1)}=0 = divide start_ARG italic_z italic_e start_POSTSUPERSCRIPT italic_z / italic_g end_POSTSUPERSCRIPT end_ARG start_ARG ( italic_e - 1 ) end_ARG - ∫ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_z end_POSTSUPERSCRIPT ( italic_x + italic_g ) divide start_ARG italic_e start_POSTSUPERSCRIPT italic_x / italic_g end_POSTSUPERSCRIPT end_ARG start_ARG italic_g ( italic_e - 1 ) end_ARG italic_d italic_x = divide start_ARG italic_z italic_e start_POSTSUPERSCRIPT italic_z / italic_g end_POSTSUPERSCRIPT - italic_g ∫ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_z / italic_g end_POSTSUPERSCRIPT ( italic_w + 1 ) italic_e start_POSTSUPERSCRIPT italic_w end_POSTSUPERSCRIPT italic_d italic_w end_ARG start_ARG ( italic_e - 1 ) end_ARG = 0
Thus, ϕ ( z ) italic-ϕ 𝑧 \phi(z) italic_ϕ ( italic_z ) is constant for all z ∈ [ 0 , g ] 𝑧 0 𝑔 z\in[0,g] italic_z ∈ [ 0 , italic_g ] and ϕ ( g ) = 1 g ( e − 1 ) ∫ 0 g ( 1 + x / g ) e x / g 𝑑 x = e e − 1 italic-ϕ 𝑔 1 𝑔 𝑒 1 superscript subscript 0 𝑔 1 𝑥 𝑔 superscript 𝑒 𝑥 𝑔 differential-d 𝑥 𝑒 𝑒 1 \phi(g)=\frac{1}{g(e-1)}\int_{0}^{g}(1+x/g)e^{x/g}dx=\frac{e}{e-1} italic_ϕ ( italic_g ) = divide start_ARG 1 end_ARG start_ARG italic_g ( italic_e - 1 ) end_ARG ∫ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_g end_POSTSUPERSCRIPT ( 1 + italic_x / italic_g ) italic_e start_POSTSUPERSCRIPT italic_x / italic_g end_POSTSUPERSCRIPT italic_d italic_x = divide start_ARG italic_e end_ARG start_ARG italic_e - 1 end_ARG .
∎
3 Instance Optimal Online Learning: Converse
In this section, we investigate fundamental limits on the instance-optimal regret guarantees achievable by any algorithm. More precisely, in the setting of binary prediction, we ask what is the smallest value of γ n subscript 𝛾 𝑛 \gamma_{n} italic_γ start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT satisfying
Reg ( 𝖠𝖫𝖦 , y n ) Reg 𝖠𝖫𝖦 superscript 𝑦 𝑛 \displaystyle\mathrm{Reg}(\mathsf{ALG},y^{n}) roman_Reg ( sansserif_ALG , italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT )
≤ γ n min { Reg ( 𝖥𝖳𝖫 , y n ) , Reg ( 𝖢𝗈𝗏𝖾𝗋 , y n ) } = γ n min { 1 2 c ( y n − 1 ) , f n } absent subscript 𝛾 𝑛 Reg 𝖥𝖳𝖫 superscript 𝑦 𝑛 Reg 𝖢𝗈𝗏𝖾𝗋 superscript 𝑦 𝑛 subscript 𝛾 𝑛 1 2 𝑐 superscript 𝑦 𝑛 1 subscript 𝑓 𝑛 \displaystyle\leq\gamma_{n}\min\{\mathrm{Reg}(\mathsf{FTL},y^{n}),\mathrm{Reg}%
(\mathsf{Cover},y^{n})\}=\gamma_{n}\min\left\{\tfrac{1}{2}c(y^{n-1}),f_{n}\right\} ≤ italic_γ start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT roman_min { roman_Reg ( sansserif_FTL , italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) , roman_Reg ( sansserif_Cover , italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) } = italic_γ start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT roman_min { divide start_ARG 1 end_ARG start_ARG 2 end_ARG italic_c ( italic_y start_POSTSUPERSCRIPT italic_n - 1 end_POSTSUPERSCRIPT ) , italic_f start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT }
(6)
for all y n superscript 𝑦 𝑛 y^{n} italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT , where f n : = Reg ( 𝖢𝗈𝗏𝖾𝗋 , y n ) = n 2 π ( 1 + o ( 1 ) ) f_{n}\mathrel{\mathop{\mathrel{\mathop{:}}}}=\mathrm{Reg}(\mathsf{Cover},y^{n}%
)=\sqrt{\frac{n}{2\pi}}(1+o(1)) italic_f start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT : = roman_Reg ( sansserif_Cover , italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) = square-root start_ARG divide start_ARG italic_n end_ARG start_ARG 2 italic_π end_ARG end_ARG ( 1 + italic_o ( 1 ) ) .
We show the following lower bound.
Theorem 3 (Lower bound on the competitive ratio).
lim n → ∞ γ n ≥ ( 1 − e − 1 / π + 2 Q ( 2 / π ) ) − 1 ≈ 1.4335 subscript → 𝑛 subscript 𝛾 𝑛 superscript 1 superscript 𝑒 1 𝜋 2 𝑄 2 𝜋 1 1.4335 \lim_{n\to\infty}\gamma_{n}\geq\Big{(}1-e^{-1/\pi}+2Q\big{(}\sqrt{2/\pi}\big{)%
}\Big{)}^{-1}\approx 1.4335 roman_lim start_POSTSUBSCRIPT italic_n → ∞ end_POSTSUBSCRIPT italic_γ start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ≥ ( 1 - italic_e start_POSTSUPERSCRIPT - 1 / italic_π end_POSTSUPERSCRIPT + 2 italic_Q ( square-root start_ARG 2 / italic_π end_ARG ) ) start_POSTSUPERSCRIPT - 1 end_POSTSUPERSCRIPT ≈ 1.4335
where the Q ( ⋅ ) 𝑄 normal-⋅ Q(\cdot) italic_Q ( ⋅ ) function is Q ( x ) : = 1 2 π ∫ x ∞ e − t 2 / 2 Q(x)\mathrel{\mathop{\mathrel{\mathop{:}}}}=\frac{1}{\sqrt{2\pi}}\int_{x}^{%
\infty}e^{-t^{2}/2} italic_Q ( italic_x ) : = divide start_ARG 1 end_ARG start_ARG square-root start_ARG 2 italic_π end_ARG end_ARG ∫ start_POSTSUBSCRIPT italic_x end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ∞ end_POSTSUPERSCRIPT italic_e start_POSTSUPERSCRIPT - italic_t start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT / 2 end_POSTSUPERSCRIPT .
Since binary prediction is a specific online learning problem, this also yields a fundamental lower bound for instance-optimality for general online learning.
Note particularly that γ n > 1 subscript 𝛾 𝑛 1 \gamma_{n}>1 italic_γ start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT > 1 , implying that ( 1 + o ( 1 ) ) min { Reg ( 𝖥𝖳𝖫 ) , Reg ( 𝖠𝖫𝖦 𝖶𝖢 ) } 1 𝑜 1 Reg 𝖥𝖳𝖫 Reg subscript 𝖠𝖫𝖦 𝖶𝖢 (1+o(1))\min\{\mathrm{Reg}(\mathsf{FTL}),\mathrm{Reg}(\mathsf{ALG}_{\mathsf{WC%
}})\} ( 1 + italic_o ( 1 ) ) roman_min { roman_Reg ( sansserif_FTL ) , roman_Reg ( sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT ) } regret is not possible to achieve. Thus, there is an inevitable multiplicative factor that must be paid in order to achieve an instance-optimal regret guarantee.
An equivalent way to state 6 is to find the smallest γ n subscript 𝛾 𝑛 \gamma_{n} italic_γ start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT for which a predictor { a t ( y t − 1 ) } t = 1 n superscript subscript subscript 𝑎 𝑡 superscript 𝑦 𝑡 1 𝑡 1 𝑛 \{a_{t}(y^{t-1})\}_{t=1}^{n} { italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_y start_POSTSUPERSCRIPT italic_t - 1 end_POSTSUPERSCRIPT ) } start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT satisfies for all y ∈ { 0 , 1 } n 𝑦 superscript 0 1 𝑛 y\in\{0,1\}^{n} italic_y ∈ { 0 , 1 } start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT
∑ t = 1 n | a t ( y t − 1 ) − y t | superscript subscript 𝑡 1 𝑛 subscript 𝑎 𝑡 superscript 𝑦 𝑡 1 subscript 𝑦 𝑡 \displaystyle\textstyle\sum_{t=1}^{n}|a_{t}(y^{t-1})-y_{t}| ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT | italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_y start_POSTSUPERSCRIPT italic_t - 1 end_POSTSUPERSCRIPT ) - italic_y start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT |
≤ γ n min { 1 2 c ( y n − 1 ) , f n } + min { ∑ t = 1 n y i , n − ∑ t = 1 n y i } . absent subscript 𝛾 𝑛 1 2 𝑐 superscript 𝑦 𝑛 1 subscript 𝑓 𝑛 superscript subscript 𝑡 1 𝑛 subscript 𝑦 𝑖 𝑛 superscript subscript 𝑡 1 𝑛 subscript 𝑦 𝑖 \displaystyle\leq\gamma_{n}\min\big{\{}\tfrac{1}{2}c(y^{n-1}),f_{n}\big{\}}+%
\min\big{\{}\textstyle\sum_{t=1}^{n}y_{i},n-\textstyle\sum_{t=1}^{n}y_{i}\big{%
\}}. ≤ italic_γ start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT roman_min { divide start_ARG 1 end_ARG start_ARG 2 end_ARG italic_c ( italic_y start_POSTSUPERSCRIPT italic_n - 1 end_POSTSUPERSCRIPT ) , italic_f start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT } + roman_min { ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , italic_n - ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT } .
(7)
In order to establish the values of γ n subscript 𝛾 𝑛 \gamma_{n} italic_γ start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT for which the loss function in the right hand side of 7 are achievable, we utilize the following result of Cover (1966 ) , which provides an exact characterization of the set of all loss functions achievable in binary prediction. Formally, we say a function ϕ : { 0 , 1 } n → ℝ + : italic-ϕ superscript 0 1 𝑛 → superscript ℝ \phi\mathrel{\mathop{:}}\{0,1\}^{n}\to\mathbb{R}^{+} italic_ϕ : { 0 , 1 } start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT → blackboard_R start_POSTSUPERSCRIPT + end_POSTSUPERSCRIPT is achievable in binary prediction if there exists a predictor/strategy a t : y t − 1 ↦ [ 0 , 1 ] : subscript 𝑎 𝑡 superscript 𝑦 𝑡 1 maps-to 0 1 a_{t}\mathrel{\mathop{:}}y^{t-1}\mapsto[0,1] italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT : italic_y start_POSTSUPERSCRIPT italic_t - 1 end_POSTSUPERSCRIPT ↦ [ 0 , 1 ] that ensures ∑ t = 1 n | a t ( y t − 1 ) − y t | = ϕ ( y n ) , ∀ y n ∈ { 0 , 1 } n \sum_{t=1}^{n}|a_{t}(y^{t-1})-y_{t}|=\phi(y^{n})\quad,\,\forall\,y^{n}\in\{0,1%
\}^{n} ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT | italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_y start_POSTSUPERSCRIPT italic_t - 1 end_POSTSUPERSCRIPT ) - italic_y start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT | = italic_ϕ ( italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) , ∀ italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ∈ { 0 , 1 } start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT .
Then, we have the following characterization.
Theorem 4 (Cover (1966 ) ).
Let ϵ n ∼ Bern ( 1 2 ) similar-to superscript italic-ϵ 𝑛 normal-Bern 1 2 \epsilon^{n}\sim\mathrm{Bern}\left(\frac{1}{2}\right) italic_ϵ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ∼ roman_Bern ( divide start_ARG 1 end_ARG start_ARG 2 end_ARG ) i.i.d. For ϕ italic-ϕ \phi italic_ϕ to be achievable, it must satisfy the following:
•
Balance: 𝔼 [ ϕ ( ϵ n ) ] = n 2 𝔼 italic-ϕ superscript italic-ϵ 𝑛 𝑛 2 \operatorname{\mathbb{E}}[\phi(\epsilon^{n})]=\frac{n}{2} blackboard_E [ italic_ϕ ( italic_ϵ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) ] = divide start_ARG italic_n end_ARG start_ARG 2 end_ARG .
•
Stability: Let ϕ t ( y t ) : = 𝔼 [ ϕ ( y t ϵ t + 1 n ) ] \phi_{t}(y^{t})\mathrel{\mathop{\mathrel{\mathop{:}}}}=\operatorname{\mathbb{E%
}}[\phi(y^{t}\epsilon_{t+1}^{n})] italic_ϕ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_y start_POSTSUPERSCRIPT italic_t end_POSTSUPERSCRIPT ) : = blackboard_E [ italic_ϕ ( italic_y start_POSTSUPERSCRIPT italic_t end_POSTSUPERSCRIPT italic_ϵ start_POSTSUBSCRIPT italic_t + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) ] ; then | ϕ t ( y t − 1 0 ) − ϕ t ( y t − 1 1 ) | ≤ 1 ∀ t ∈ [ n ] , y t ∈ { 0 , 1 } t formulae-sequence subscript italic-ϕ 𝑡 superscript 𝑦 𝑡 1 0 subscript italic-ϕ 𝑡 superscript 𝑦 𝑡 1 1 1 for-all 𝑡 delimited-[] 𝑛 superscript 𝑦 𝑡 superscript 0 1 𝑡 |\phi_{t}(y^{t-1}0)-\phi_{t}(y^{t-1}1)|\leq 1\,\forall t\in[n],\,y^{t}\in\{0,1%
\}^{t} | italic_ϕ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_y start_POSTSUPERSCRIPT italic_t - 1 end_POSTSUPERSCRIPT 0 ) - italic_ϕ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_y start_POSTSUPERSCRIPT italic_t - 1 end_POSTSUPERSCRIPT 1 ) | ≤ 1 ∀ italic_t ∈ [ italic_n ] , italic_y start_POSTSUPERSCRIPT italic_t end_POSTSUPERSCRIPT ∈ { 0 , 1 } start_POSTSUPERSCRIPT italic_t end_POSTSUPERSCRIPT .
Further any ϕ italic-ϕ \phi italic_ϕ satisfying the above is realized by predictor a t ( y t − 1 ) = 1 + ϕ t ( y t − 1 0 ) − ϕ t ( y t − 1 1 ) 2 subscript 𝑎 𝑡 superscript 𝑦 𝑡 1 1 subscript italic-ϕ 𝑡 superscript 𝑦 𝑡 1 0 subscript italic-ϕ 𝑡 superscript 𝑦 𝑡 1 1 2 a_{t}(y^{t-1})=\frac{1+\phi_{t}(y^{t-1}0)-\phi_{t}(y^{t-1}1)}{2} italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_y start_POSTSUPERSCRIPT italic_t - 1 end_POSTSUPERSCRIPT ) = divide start_ARG 1 + italic_ϕ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_y start_POSTSUPERSCRIPT italic_t - 1 end_POSTSUPERSCRIPT 0 ) - italic_ϕ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_y start_POSTSUPERSCRIPT italic_t - 1 end_POSTSUPERSCRIPT 1 ) end_ARG start_ARG 2 end_ARG .
As an immediate corollary, Theorem 4 equips us with the exact minimax optimal algorithm for binary prediction alluded to in Section 1 .
Returning to our setting, from the balance condition in Theorem 4 , for ϵ n ∼ Bern ( 1 / 2 ) similar-to superscript italic-ϵ 𝑛 Bern 1 2 \epsilon^{n}\sim\mathrm{Bern}(1/2) italic_ϵ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ∼ roman_Bern ( 1 / 2 ) i.i.d.
γ n 𝔼 [ min { 1 2 c ( ϵ n − 1 ) , f n } ] + 𝔼 [ min { ∑ t = 1 n ϵ t , n − ∑ t = 1 n ϵ t } ] ≥ n 2 . subscript 𝛾 𝑛 𝔼 1 2 𝑐 superscript italic-ϵ 𝑛 1 subscript 𝑓 𝑛 𝔼 superscript subscript 𝑡 1 𝑛 subscript italic-ϵ 𝑡 𝑛 superscript subscript 𝑡 1 𝑛 subscript italic-ϵ 𝑡 𝑛 2 \displaystyle\gamma_{n}\operatorname{\mathbb{E}}\big{[}\min\big{\{}\tfrac{1}{2%
}c(\epsilon^{n-1}),f_{n}\big{\}}\big{]}+\operatorname{\mathbb{E}}\big{[}\min%
\big{\{}\textstyle\sum_{t=1}^{n}\epsilon_{t},n-\textstyle\sum_{t=1}^{n}%
\epsilon_{t}\big{\}}\big{]}\geq\frac{n}{2}. italic_γ start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT blackboard_E [ roman_min { divide start_ARG 1 end_ARG start_ARG 2 end_ARG italic_c ( italic_ϵ start_POSTSUPERSCRIPT italic_n - 1 end_POSTSUPERSCRIPT ) , italic_f start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT } ] + blackboard_E [ roman_min { ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT italic_ϵ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT , italic_n - ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT italic_ϵ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT } ] ≥ divide start_ARG italic_n end_ARG start_ARG 2 end_ARG .
for the function in (7 ) to be achievable. Using the definition of f n subscript 𝑓 𝑛 f_{n} italic_f start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ,
γ n ≥ f n 𝔼 [ min { 1 2 c ( ϵ n − 1 ) , f n } ] = 2 f n 𝔼 [ min { c ( ϵ n − 1 ) , 2 f n } ] ] . \displaystyle\gamma_{n}\geq\frac{f_{n}}{\operatorname{\mathbb{E}}\left[\min%
\left\{\tfrac{1}{2}c(\epsilon^{n-1}),f_{n}\right\}\right]}=\frac{2f_{n}}{%
\operatorname{\mathbb{E}}\left[\min\left\{c(\epsilon^{n-1}),2f_{n}\right\}]%
\right]}. italic_γ start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ≥ divide start_ARG italic_f start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT end_ARG start_ARG blackboard_E [ roman_min { divide start_ARG 1 end_ARG start_ARG 2 end_ARG italic_c ( italic_ϵ start_POSTSUPERSCRIPT italic_n - 1 end_POSTSUPERSCRIPT ) , italic_f start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT } ] end_ARG = divide start_ARG 2 italic_f start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT end_ARG start_ARG blackboard_E [ roman_min { italic_c ( italic_ϵ start_POSTSUPERSCRIPT italic_n - 1 end_POSTSUPERSCRIPT ) , 2 italic_f start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT } ] ] end_ARG .
The above bound immediately yields that γ n ≥ 1 subscript 𝛾 𝑛 1 \gamma_{n}\geq 1 italic_γ start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ≥ 1 as expected. We can further sharply characterize the asymptotics of γ n subscript 𝛾 𝑛 \gamma_{n} italic_γ start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT , resulting in the stated lower bound. The full proof of Theorem 3 is provided in Appendix B .
4 Instance-Optimal Algorithms in Small-Loss Settings
So far, we have presented specializations of 𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART that achieve instance-optimality between 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL and the worst-case regret g ( n ) 𝑔 𝑛 g(n) italic_g ( italic_n ) . However, many worst-case algorithms can still adapt to the instance ℓ n superscript ℓ 𝑛 \ell^{n} roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT and achieve regret guarantees that are a function of the ‘difficulty’ of the instance ℓ n superscript ℓ 𝑛 \ell^{n} roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT . A common way to quantify this is difficulty is via small-loss bounds , where the regret is upper bounded by g ( L * ) 𝑔 superscript 𝐿 g(L^{*}) italic_g ( italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) where g ( ⋅ ) 𝑔 ⋅ g(\cdot) italic_g ( ⋅ ) as earlier is a monotonic increasing function and L * : = min a ∈ 𝒜 ∑ t = 1 n ℓ t ( a ) L^{*}\mathrel{\mathop{\mathrel{\mathop{:}}}}=\min_{a\in\mathcal{A}}\sum_{t=1}^%
{n}\ell_{t}(a) italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT : = roman_min start_POSTSUBSCRIPT italic_a ∈ caligraphic_A end_POSTSUBSCRIPT ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a ) is the loss achieved by the best action. Such guarantees imply that for sequences where there exists an action achieving low loss, the corresponding regret achieved is also low. Thus, a natural question is whether 𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART can be specialized to yield an algorithm that is constant competitive with respect to min { Reg ( 𝖥𝖳𝖫 , ℓ n ) , g ( L * ) } Reg 𝖥𝖳𝖫 superscript ℓ 𝑛 𝑔 superscript 𝐿 \min\{\mathrm{Reg}(\mathsf{FTL},\ell^{n}),g(L^{*})\} roman_min { roman_Reg ( sansserif_FTL , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) , italic_g ( italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) } .
As a starting point, if L * superscript 𝐿 L^{*} italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT is known apriori, it is easy to achieve a e e − 1 𝑒 𝑒 1 \frac{e}{e-1} divide start_ARG italic_e end_ARG start_ARG italic_e - 1 end_ARG approximation by simply using 𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART with (random) threshold θ = g ( L * ) ln ( 1 + ( e − 1 ) U ) , U ∼ Unif [ 0 , 1 ] formulae-sequence 𝜃 𝑔 superscript 𝐿 1 𝑒 1 𝑈 similar-to 𝑈 Unif 0 1 \theta=g(L^{*})\ln(1+(e-1)U),U\sim\mathrm{Unif}[0,1] italic_θ = italic_g ( italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) roman_ln ( 1 + ( italic_e - 1 ) italic_U ) , italic_U ∼ roman_Unif [ 0 , 1 ] ; this is an immediate corollary of Theorem 2 . When L * superscript 𝐿 L^{*} italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT is not known, we use a guess-and-double argument to devise an algorithm that achieves the following instance-optimality guarantee.
Theorem 5 (Regret of 𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART for unknown small loss).
Let 𝖠𝖫𝖦 𝖶𝖢 subscript 𝖠𝖫𝖦 𝖶𝖢 \mathsf{ALG}_{\mathsf{WC}} sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT have small loss regret guarantees satisfying Reg ( 𝖠𝖫𝖦 𝖶𝖢 , ℓ n ) ≤ g ( L * ) normal-Reg subscript 𝖠𝖫𝖦 𝖶𝖢 superscript normal-ℓ 𝑛 𝑔 superscript 𝐿 \mathrm{Reg}(\mathsf{ALG}_{\mathsf{WC}},\ell^{n})\leq g(L^{*}) roman_Reg ( sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) ≤ italic_g ( italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT )
for any ℓ n superscript normal-ℓ 𝑛 \ell^{n} roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT where L * = min j ∈ [ m ] L t j superscript 𝐿 subscript 𝑗 delimited-[] 𝑚 subscript 𝐿 𝑡 𝑗 L^{*}=\min_{j\in[m]}L_{tj} italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT = roman_min start_POSTSUBSCRIPT italic_j ∈ [ italic_m ] end_POSTSUBSCRIPT italic_L start_POSTSUBSCRIPT italic_t italic_j end_POSTSUBSCRIPT , i.e. the loss achieved by the best expert in hindsight. Then, if we play 𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART for Small-Loss as stated in Algorithm 2 2 2 2 , we have
Reg ( 𝖲𝖬𝖠𝖱𝖳 , ℓ n ) Reg 𝖲𝖬𝖠𝖱𝖳 superscript ℓ 𝑛 \displaystyle\mathrm{Reg}(\mathsf{SMART},\ell^{n}) roman_Reg ( sansserif_SMART , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT )
≤ 2 min ( Reg ( 𝖥𝖳𝖫 , ℓ n ) , ∑ z = 1 log ( 1 + L * / log m ) + 1 g ( 2 z log m ) ) + O ( log L * / log m ) absent 2 Reg 𝖥𝖳𝖫 superscript ℓ 𝑛 superscript subscript 𝑧 1 1 superscript 𝐿 𝑚 1 𝑔 superscript 2 𝑧 𝑚 𝑂 superscript 𝐿 𝑚 \displaystyle\leq 2\min\big{(}\mathrm{Reg}(\mathsf{FTL},\ell^{n}),\textstyle%
\sum_{z=1}^{\log(1+L^{*}/\log m)+1}g(2^{z}\log m)\big{)}+O(\log L^{*}/\log m) ≤ 2 roman_min ( roman_Reg ( sansserif_FTL , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) , ∑ start_POSTSUBSCRIPT italic_z = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT roman_log ( 1 + italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT / roman_log italic_m ) + 1 end_POSTSUPERSCRIPT italic_g ( 2 start_POSTSUPERSCRIPT italic_z end_POSTSUPERSCRIPT roman_log italic_m ) ) + italic_O ( roman_log italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT / roman_log italic_m )
In particular, in the prediction with expert advice setting, we know that 𝖧𝖾𝖽𝗀𝖾 𝖧𝖾𝖽𝗀𝖾 \mathsf{Hedge} sansserif_Hedge with a time-varying learning rate achieves g ( L * ) ≡ 2 2 L * log m + κ log m 𝑔 superscript 𝐿 2 2 superscript 𝐿 𝑚 𝜅 𝑚 g(L^{*})\equiv 2\sqrt{2L^{*}\log m}+\kappa\log m italic_g ( italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) ≡ 2 square-root start_ARG 2 italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT roman_log italic_m end_ARG + italic_κ roman_log italic_m (where κ > 0 𝜅 0 \kappa>0 italic_κ > 0 is an absolute constant) (Auer et al., 2002b ; Cesa-Bianchi et al., 2007 ) ; this gives Corollary 1 in Section 1 .
The intuitive idea behind the algorithm is to guess the value of L * superscript 𝐿 L^{*} italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT , and play 𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART with this guessed value while simultaneously keeping track of the regret incurred. Whenever the regret incurred exceeds the guarantee established by 𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART with known L * superscript 𝐿 L^{*} italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT double the guessed value and start again. We use the notation 𝖠𝖫𝖦 𝖶𝖢 ( ℓ t 1 t 2 ) subscript 𝖠𝖫𝖦 𝖶𝖢 superscript subscript ℓ subscript 𝑡 1 subscript 𝑡 2 \mathsf{ALG}_{\mathsf{WC}}(\ell_{t_{1}}^{t_{2}}) sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT ( roman_ℓ start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_t start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_POSTSUPERSCRIPT ) to refer to the worst-case algorithm when the previously observed sequence is ℓ t 1 t 2 superscript subscript ℓ subscript 𝑡 1 subscript 𝑡 2 \ell_{t_{1}}^{t_{2}} roman_ℓ start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_t start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_POSTSUPERSCRIPT ; in particular this would be equivalent to the action recommended at time t 2 + 1 subscript 𝑡 2 1 t_{2}+1 italic_t start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT + 1 after throwing away all the observed losses before t 1 subscript 𝑡 1 t_{1} italic_t start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT .
We let Σ t 1 : t 2 𝖥𝖳𝖫 = ∑ i = t 1 t 2 ( L i ( a i − 1 * ) − L i ( a i * ) ) subscript superscript Σ 𝖥𝖳𝖫 : subscript 𝑡 1 subscript 𝑡 2 superscript subscript 𝑖 subscript 𝑡 1 subscript 𝑡 2 subscript 𝐿 𝑖 subscript superscript 𝑎 𝑖 1 subscript 𝐿 𝑖 subscript superscript 𝑎 𝑖 \Sigma^{\mathsf{FTL}}_{t_{1}\mathrel{\mathop{:}}t_{2}}=\sum_{i=t_{1}}^{t_{2}}(%
L_{i}(a^{*}_{i-1})-L_{i}(a^{*}_{i})) roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT : italic_t start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_POSTSUBSCRIPT = ∑ start_POSTSUBSCRIPT italic_i = italic_t start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_t start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_POSTSUPERSCRIPT ( italic_L start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_i - 1 end_POSTSUBSCRIPT ) - italic_L start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) ) , which grows as the number of leader changes within i ∈ [ t 1 , t 2 ] 𝑖 subscript 𝑡 1 subscript 𝑡 2 i\in[t_{1},t_{2}] italic_i ∈ [ italic_t start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , italic_t start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ] . The algorithm’s pseudocode is given in Algorithm 2 below, and a proof of Theorem 5 is provided in Appendix A .
Input: Policies
𝖥𝖳𝖫 , 𝖠𝖫𝖦 𝖶𝖢 𝖥𝖳𝖫 subscript 𝖠𝖫𝖦 𝖶𝖢
\mathsf{FTL},\mathsf{ALG}_{\mathsf{WC}} sansserif_FTL , sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT ; Small-loss bound
g ( ⋅ ) 𝑔 ⋅ g(\cdot) italic_g ( ⋅ )
for z = 0 , 1 , … 𝑧 0 1 normal-…
z=0,1,\dotsc italic_z = 0 , 1 , … (epochs) do
Let
t = t z : = 𝑡 subscript 𝑡 𝑧 : t=t_{z}\mathrel{\mathop{\mathrel{\mathop{:}}}}= italic_t = italic_t start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT : = start time of
z t h superscript 𝑧 𝑡 ℎ z^{th} italic_z start_POSTSUPERSCRIPT italic_t italic_h end_POSTSUPERSCRIPT epoch,
L z * : = 2 z log m L^{*}_{z}\mathrel{\mathop{\mathrel{\mathop{:}}}}=2^{z}\log m italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT : = 2 start_POSTSUPERSCRIPT italic_z end_POSTSUPERSCRIPT roman_log italic_m (current guess for
L * superscript 𝐿 L^{*} italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ),
Σ t z : t z − 1 𝖥𝖳𝖫 = 0 subscript superscript Σ 𝖥𝖳𝖫 : subscript 𝑡 𝑧 subscript 𝑡 𝑧 1 0 \Sigma^{\mathsf{FTL}}_{t_{z}\mathrel{\mathop{:}}t_{z}-1}=0 roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT : italic_t start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT - 1 end_POSTSUBSCRIPT = 0 ;
while Σ t z : t − 1 𝖥𝖳𝖫 ≤ g ( L z * ) subscript superscript normal-Σ 𝖥𝖳𝖫 normal-: subscript 𝑡 𝑧 𝑡 1 𝑔 subscript superscript 𝐿 𝑧 \Sigma^{\mathsf{FTL}}_{t_{z}\mathrel{\mathop{:}}t-1}\leq g(L^{*}_{z}) roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT : italic_t - 1 end_POSTSUBSCRIPT ≤ italic_g ( italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT ) do
Set
a t = a t − 1 * subscript 𝑎 𝑡 superscript subscript 𝑎 𝑡 1 a_{t}=a_{t-1}^{*} italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT = italic_a start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ;
// Play 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL
Observe
ℓ t ( ⋅ ) subscript ℓ 𝑡 ⋅ \ell_{t}(\cdot) roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( ⋅ ) ;
Update
L t ( ⋅ ) = L t − 1 ( ⋅ ) + ℓ t ( ⋅ ) subscript 𝐿 𝑡 ⋅ subscript 𝐿 𝑡 1 ⋅ subscript ℓ 𝑡 ⋅ L_{t}(\cdot)=L_{t-1}(\cdot)+\ell_{t}(\cdot) italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( ⋅ ) = italic_L start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT ( ⋅ ) + roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( ⋅ ) and
Σ t z : t 𝖥𝖳𝖫 = Σ t z : t − 1 𝖥𝖳𝖫 + ( L t ( a t − 1 * ) − L t ( a t * ) ) subscript superscript Σ 𝖥𝖳𝖫 : subscript 𝑡 𝑧 𝑡 subscript superscript Σ 𝖥𝖳𝖫 : subscript 𝑡 𝑧 𝑡 1 subscript 𝐿 𝑡 superscript subscript 𝑎 𝑡 1 subscript 𝐿 𝑡 superscript subscript 𝑎 𝑡 \Sigma^{\mathsf{FTL}}_{t_{z}\mathrel{\mathop{:}}t}=\Sigma^{\mathsf{FTL}}_{t_{z%
}\mathrel{\mathop{:}}t-1}+(L_{t}(a_{t-1}^{*})-L_{t}(a_{t}^{*})) roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT : italic_t end_POSTSUBSCRIPT = roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT : italic_t - 1 end_POSTSUBSCRIPT + ( italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) - italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) ) and
t = t + 1 𝑡 𝑡 1 t=t+1 italic_t = italic_t + 1 ;
end while
Let
τ z : = min t ≥ t z Σ t z : t 𝖥𝖳𝖫 > g ( L z * ) \tau_{z}\mathrel{\mathop{:}}=\min_{t\geq t_{z}}\Sigma^{\mathsf{FTL}}_{t_{z}%
\mathrel{\mathop{:}}t}>g(L^{*}_{z}) italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT : = roman_min start_POSTSUBSCRIPT italic_t ≥ italic_t start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT end_POSTSUBSCRIPT roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT : italic_t end_POSTSUBSCRIPT > italic_g ( italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT ) and
t = τ z + 1 𝑡 subscript 𝜏 𝑧 1 t=\tau_{z}+1 italic_t = italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT + 1 ;
// Check if loss incurred by 𝖠𝖫𝖦 𝖶𝖢 subscript 𝖠𝖫𝖦 𝖶𝖢 \mathsf{ALG}_{\mathsf{WC}} sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT in this epoch violates the upper bound from L z * superscript subscript 𝐿 𝑧 L_{z}^{*} italic_L start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT is correct
while ∑ t = t z t ⟨ a t , ℓ t ⟩ ≤ L z * + 2 min { Σ t z : t 𝖥𝖳𝖫 , g ( L z * ) } + 1 superscript subscript 𝑡 subscript 𝑡 𝑧 𝑡 subscript 𝑎 𝑡 subscript normal-ℓ 𝑡
superscript subscript 𝐿 𝑧 2 subscript superscript normal-Σ 𝖥𝖳𝖫 normal-: subscript 𝑡 𝑧 𝑡 𝑔 superscript subscript 𝐿 𝑧 1 \sum_{t=t_{z}}^{t}\langle a_{t},\ell_{t}\rangle\leq L_{z}^{*}+2\min\{\Sigma^{%
\mathsf{FTL}}_{t_{z}\mathrel{\mathop{:}}t},g(L_{z}^{*})\}+1 ∑ start_POSTSUBSCRIPT italic_t = italic_t start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_t end_POSTSUPERSCRIPT ⟨ italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT , roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ⟩ ≤ italic_L start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT + 2 roman_min { roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT : italic_t end_POSTSUBSCRIPT , italic_g ( italic_L start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) } + 1 do
Set
a t = 𝖠𝖫𝖦 𝖶𝖢 ( ℓ τ z + 1 t − 1 ) subscript 𝑎 𝑡 subscript 𝖠𝖫𝖦 𝖶𝖢 superscript subscript ℓ subscript 𝜏 𝑧 1 𝑡 1 a_{t}=\mathsf{ALG}_{\mathsf{WC}}(\ell_{\tau_{z}+1}^{t-1}) italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT = sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT ( roman_ℓ start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_t - 1 end_POSTSUPERSCRIPT ) ;
// Play 𝖠𝖫𝖦 𝖶𝖢 subscript 𝖠𝖫𝖦 𝖶𝖢 \mathsf{ALG}_{\mathsf{WC}} sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT forgetting losses before τ z + 1 subscript 𝜏 𝑧 1 \tau_{z}+1 italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT + 1
Observe
ℓ t ( ⋅ ) subscript ℓ 𝑡 ⋅ \ell_{t}(\cdot) roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( ⋅ ) ;
Update
L t ( ⋅ ) = L t − 1 ( ⋅ ) + ℓ t ( ⋅ ) subscript 𝐿 𝑡 ⋅ subscript 𝐿 𝑡 1 ⋅ subscript ℓ 𝑡 ⋅ L_{t}(\cdot)=L_{t-1}(\cdot)+\ell_{t}(\cdot) italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( ⋅ ) = italic_L start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT ( ⋅ ) + roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( ⋅ ) and
Σ t z : t 𝖥𝖳𝖫 = Σ t z : t − 1 𝖥𝖳𝖫 + ( L t ( a t − 1 * ) − L t ( a t * ) ) subscript superscript Σ 𝖥𝖳𝖫 : subscript 𝑡 𝑧 𝑡 subscript superscript Σ 𝖥𝖳𝖫 : subscript 𝑡 𝑧 𝑡 1 subscript 𝐿 𝑡 superscript subscript 𝑎 𝑡 1 subscript 𝐿 𝑡 superscript subscript 𝑎 𝑡 \Sigma^{\mathsf{FTL}}_{t_{z}\mathrel{\mathop{:}}t}=\Sigma^{\mathsf{FTL}}_{t_{z%
}\mathrel{\mathop{:}}t-1}+(L_{t}(a_{t-1}^{*})-L_{t}(a_{t}^{*})) roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT : italic_t end_POSTSUBSCRIPT = roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT : italic_t - 1 end_POSTSUBSCRIPT + ( italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) - italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) ) and
t = t + 1 𝑡 𝑡 1 t=t+1 italic_t = italic_t + 1 ;
end while
end for
Algorithm 2 𝖲𝖬𝖠𝖱𝖳 𝖲𝖬𝖠𝖱𝖳 \mathsf{SMART} sansserif_SMART for Small-Loss
Appendix A Omitted proofs from Section 4
In this Section, we will establish the proofs of Theorem 5 and Corollary 1 .
Recall Algorithm 2, where 𝖠𝖫𝖦 𝖶𝖢 ( ℓ t 1 t 2 ) subscript 𝖠𝖫𝖦 𝖶𝖢 superscript subscript ℓ subscript 𝑡 1 subscript 𝑡 2 \mathsf{ALG}_{\mathsf{WC}}(\ell_{t_{1}}^{t_{2}}) sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT ( roman_ℓ start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_t start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_POSTSUPERSCRIPT ) refers to the worst-case algorithm when the previously observed sequence is ℓ t 1 t 2 superscript subscript ℓ subscript 𝑡 1 subscript 𝑡 2 \ell_{t_{1}}^{t_{2}} roman_ℓ start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_t start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_POSTSUPERSCRIPT ; in particular this would be equivalent to the action recommended at time t 2 + 1 subscript 𝑡 2 1 t_{2}+1 italic_t start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT + 1 after throwing away all the observed losses before t 1 subscript 𝑡 1 t_{1} italic_t start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT .
We let Σ t 1 : t 2 𝖥𝖳𝖫 = ∑ i = t 1 t 2 ( L i ( a i − 1 * ) − L i ( a i * ) ) subscript superscript Σ 𝖥𝖳𝖫 : subscript 𝑡 1 subscript 𝑡 2 superscript subscript 𝑖 subscript 𝑡 1 subscript 𝑡 2 subscript 𝐿 𝑖 subscript superscript 𝑎 𝑖 1 subscript 𝐿 𝑖 subscript superscript 𝑎 𝑖 \Sigma^{\mathsf{FTL}}_{t_{1}\mathrel{\mathop{:}}t_{2}}=\sum_{i=t_{1}}^{t_{2}}(%
L_{i}(a^{*}_{i-1})-L_{i}(a^{*}_{i})) roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT : italic_t start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_POSTSUBSCRIPT = ∑ start_POSTSUBSCRIPT italic_i = italic_t start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_t start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_POSTSUPERSCRIPT ( italic_L start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_i - 1 end_POSTSUBSCRIPT ) - italic_L start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) ) , which grows as the number of leader changes within i ∈ [ t 1 , t 2 ] 𝑖 subscript 𝑡 1 subscript 𝑡 2 i\in[t_{1},t_{2}] italic_i ∈ [ italic_t start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , italic_t start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ] .
We first have the following decomposition of the regret for any algorithm 𝖠𝖫𝖦 𝖠𝖫𝖦 \mathsf{ALG} sansserif_ALG that plays the sequence of actions a t 𝖠𝖫𝖦 subscript superscript 𝑎 𝖠𝖫𝖦 𝑡 a^{\mathsf{ALG}}_{t} italic_a start_POSTSUPERSCRIPT sansserif_ALG end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT at time t 𝑡 t italic_t .
Lemma 3 .
The regret incurred by any sequence of actions ( a t 𝖠𝖫𝖦 ) t ∈ [ n + 1 ] subscript subscript superscript 𝑎 𝖠𝖫𝖦 𝑡 𝑡 delimited-[] 𝑛 1 (a^{\mathsf{ALG}}_{t})_{t\in[n+1]} ( italic_a start_POSTSUPERSCRIPT sansserif_ALG end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) start_POSTSUBSCRIPT italic_t ∈ [ italic_n + 1 ] end_POSTSUBSCRIPT can be written as
Reg ( 𝖠𝖫𝖦 , ℓ n ) Reg 𝖠𝖫𝖦 superscript ℓ 𝑛 \displaystyle\mathrm{Reg}(\mathsf{ALG},\ell^{n}) roman_Reg ( sansserif_ALG , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT )
= ∑ t = 1 n ( L t ( a t 𝖠𝖫𝖦 ) − L t ( a t + 1 𝖠𝖫𝖦 ) ) , absent superscript subscript 𝑡 1 𝑛 subscript 𝐿 𝑡 subscript superscript 𝑎 𝖠𝖫𝖦 𝑡 subscript 𝐿 𝑡 subscript superscript 𝑎 𝖠𝖫𝖦 𝑡 1 \displaystyle=\sum_{t=1}^{n}\left(L_{t}(a^{\mathsf{ALG}}_{t})-L_{t}(a^{\mathsf%
{ALG}}_{t+1})\right), = ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ( italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT sansserif_ALG end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) - italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT sansserif_ALG end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t + 1 end_POSTSUBSCRIPT ) ) ,
(8)
where we let a n + 1 𝖠𝖫𝖦 : = a n * a_{n+1}^{\mathsf{ALG}}\mathrel{\mathop{:}}=a_{n}^{*} italic_a start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT sansserif_ALG end_POSTSUPERSCRIPT : = italic_a start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT .
Proof.
L n ( 𝖠𝖫𝖦 ) subscript 𝐿 𝑛 𝖠𝖫𝖦 \displaystyle L_{n}(\mathsf{ALG}) italic_L start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ( sansserif_ALG )
= ∑ t = 1 n ℓ t ( a t 𝖠𝖫𝖦 ) absent superscript subscript 𝑡 1 𝑛 subscript ℓ 𝑡 subscript superscript 𝑎 𝖠𝖫𝖦 𝑡 \displaystyle=\sum_{t=1}^{n}\ell_{t}(a^{\mathsf{ALG}}_{t}) = ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT sansserif_ALG end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT )
(9)
= ∑ t = 1 n ( L t ( a t 𝖠𝖫𝖦 ) − L t − 1 ( a t 𝖠𝖫𝖦 ) ) absent superscript subscript 𝑡 1 𝑛 subscript 𝐿 𝑡 subscript superscript 𝑎 𝖠𝖫𝖦 𝑡 subscript 𝐿 𝑡 1 subscript superscript 𝑎 𝖠𝖫𝖦 𝑡 \displaystyle=\sum_{t=1}^{n}\left(L_{t}(a^{\mathsf{ALG}}_{t})-L_{t-1}(a^{%
\mathsf{ALG}}_{t})\right) = ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ( italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT sansserif_ALG end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) - italic_L start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT sansserif_ALG end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) )
(10)
= L n ( a n 𝖠𝖫𝖦 ) + ∑ t = 1 n − 1 ( L t ( a t 𝖠𝖫𝖦 ) − L t ( a t + 1 𝖠𝖫𝖦 ) ) − L 0 ( a 1 𝖠𝖫𝖦 ) . absent subscript 𝐿 𝑛 subscript superscript 𝑎 𝖠𝖫𝖦 𝑛 superscript subscript 𝑡 1 𝑛 1 subscript 𝐿 𝑡 subscript superscript 𝑎 𝖠𝖫𝖦 𝑡 subscript 𝐿 𝑡 subscript superscript 𝑎 𝖠𝖫𝖦 𝑡 1 subscript 𝐿 0 subscript superscript 𝑎 𝖠𝖫𝖦 1 \displaystyle=L_{n}(a^{\mathsf{ALG}}_{n})+\sum_{t=1}^{n-1}\left(L_{t}(a^{%
\mathsf{ALG}}_{t})-L_{t}(a^{\mathsf{ALG}}_{t+1})\right)-L_{0}(a^{\mathsf{ALG}}%
_{1}). = italic_L start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT sansserif_ALG end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ) + ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n - 1 end_POSTSUPERSCRIPT ( italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT sansserif_ALG end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) - italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT sansserif_ALG end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t + 1 end_POSTSUBSCRIPT ) ) - italic_L start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT sansserif_ALG end_POSTSUPERSCRIPT start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ) .
(11)
This implies a regret decomposition of
Reg ( 𝖠𝖫𝖦 , ℓ n ) Reg 𝖠𝖫𝖦 superscript ℓ 𝑛 \displaystyle\mathrm{Reg}(\mathsf{ALG},\ell^{n}) roman_Reg ( sansserif_ALG , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT )
= L n ( 𝖠𝖫𝖦 ) − L n ( a n * ) absent subscript 𝐿 𝑛 𝖠𝖫𝖦 subscript 𝐿 𝑛 superscript subscript 𝑎 𝑛 \displaystyle=L_{n}(\mathsf{ALG})-L_{n}(a_{n}^{*}) = italic_L start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ( sansserif_ALG ) - italic_L start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT )
(12)
= L n ( a n 𝖠𝖫𝖦 ) − L n ( a n * ) + ∑ t = 1 n − 1 ( L t ( a t 𝖠𝖫𝖦 ) − L t ( a t + 1 𝖠𝖫𝖦 ) ) − L 0 ( a 1 𝖠𝖫𝖦 ) absent subscript 𝐿 𝑛 subscript superscript 𝑎 𝖠𝖫𝖦 𝑛 subscript 𝐿 𝑛 superscript subscript 𝑎 𝑛 superscript subscript 𝑡 1 𝑛 1 subscript 𝐿 𝑡 subscript superscript 𝑎 𝖠𝖫𝖦 𝑡 subscript 𝐿 𝑡 subscript superscript 𝑎 𝖠𝖫𝖦 𝑡 1 subscript 𝐿 0 subscript superscript 𝑎 𝖠𝖫𝖦 1 \displaystyle=L_{n}(a^{\mathsf{ALG}}_{n})-L_{n}(a_{n}^{*})+\sum_{t=1}^{n-1}%
\left(L_{t}(a^{\mathsf{ALG}}_{t})-L_{t}(a^{\mathsf{ALG}}_{t+1})\right)-L_{0}(a%
^{\mathsf{ALG}}_{1}) = italic_L start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT sansserif_ALG end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ) - italic_L start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) + ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n - 1 end_POSTSUPERSCRIPT ( italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT sansserif_ALG end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) - italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT sansserif_ALG end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t + 1 end_POSTSUBSCRIPT ) ) - italic_L start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT sansserif_ALG end_POSTSUPERSCRIPT start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT )
(13)
As L 0 ( a 1 𝖠𝖫𝖦 ) = 0 subscript 𝐿 0 subscript superscript 𝑎 𝖠𝖫𝖦 1 0 L_{0}(a^{\mathsf{ALG}}_{1})=0 italic_L start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT sansserif_ALG end_POSTSUPERSCRIPT start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ) = 0 , and a n + 1 𝖠𝖫𝖦 : = a n * a_{n+1}^{\mathsf{ALG}}\mathrel{\mathop{:}}=a_{n}^{*} italic_a start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT sansserif_ALG end_POSTSUPERSCRIPT : = italic_a start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT , it follows that
Reg ( 𝖠𝖫𝖦 , ℓ n ) Reg 𝖠𝖫𝖦 superscript ℓ 𝑛 \displaystyle\mathrm{Reg}(\mathsf{ALG},\ell^{n}) roman_Reg ( sansserif_ALG , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT )
= ∑ t = 1 n ( L t ( a t 𝖠𝖫𝖦 ) − L t ( a t + 1 𝖠𝖫𝖦 ) ) . absent superscript subscript 𝑡 1 𝑛 subscript 𝐿 𝑡 subscript superscript 𝑎 𝖠𝖫𝖦 𝑡 subscript 𝐿 𝑡 subscript superscript 𝑎 𝖠𝖫𝖦 𝑡 1 \displaystyle=\sum_{t=1}^{n}\left(L_{t}(a^{\mathsf{ALG}}_{t})-L_{t}(a^{\mathsf%
{ALG}}_{t+1})\right). = ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ( italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT sansserif_ALG end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) - italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUPERSCRIPT sansserif_ALG end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t + 1 end_POSTSUBSCRIPT ) ) .
(14)
∎
Next, we use this decomposition to establish the regret of any algorithm 𝖠𝖫𝖦 𝖠𝖫𝖦 \mathsf{ALG} sansserif_ALG that alternates between playing 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL and another algorithm 𝖠𝖫𝖦 𝖶𝖢 subscript 𝖠𝖫𝖦 𝖶𝖢 \mathsf{ALG}_{\mathsf{WC}} sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT .
Lemma 4 .
Consider an algorithm 𝖠𝖫𝖦 𝖠𝖫𝖦 \mathsf{ALG} sansserif_ALG which alternates between playing 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL and 𝖠𝖫𝖦 𝖶𝖢 subscript 𝖠𝖫𝖦 𝖶𝖢 \mathsf{ALG}_{\mathsf{WC}} sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT , where 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL is played in the intervals { [ t z , τ z ] } z ∈ [ z last ] subscript subscript 𝑡 𝑧 subscript 𝜏 𝑧 𝑧 delimited-[] subscript 𝑧 normal-last \{[t_{z},\tau_{z}]\}_{z\in[z_{\mathrm{last}}]} { [ italic_t start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT , italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT ] } start_POSTSUBSCRIPT italic_z ∈ [ italic_z start_POSTSUBSCRIPT roman_last end_POSTSUBSCRIPT ] end_POSTSUBSCRIPT , and 𝖠𝖫𝖦 𝖶𝖢 subscript 𝖠𝖫𝖦 𝖶𝖢 \mathsf{ALG}_{\mathsf{WC}} sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT is played in intervals { [ τ z + 1 , t z + 1 − 1 ] } z ∈ [ z last ] subscript subscript 𝜏 𝑧 1 subscript 𝑡 𝑧 1 1 𝑧 delimited-[] subscript 𝑧 normal-last \{[\tau_{z}+1,t_{z+1}-1]\}_{z\in[z_{\mathrm{last}}]} { [ italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT + 1 , italic_t start_POSTSUBSCRIPT italic_z + 1 end_POSTSUBSCRIPT - 1 ] } start_POSTSUBSCRIPT italic_z ∈ [ italic_z start_POSTSUBSCRIPT roman_last end_POSTSUBSCRIPT ] end_POSTSUBSCRIPT . The regret of 𝖠𝖫𝖦 𝖠𝖫𝖦 \mathsf{ALG} sansserif_ALG is bounded by
Reg ( 𝖠𝖫𝖦 , ℓ n ) ≤ ∑ z ( Σ t z : τ z − 1 𝖥𝖳𝖫 + Reg ( 𝖠𝖫𝖦 𝖶𝖢 , ℓ τ z + 1 t z + 1 − 1 ) + 1 ) . Reg 𝖠𝖫𝖦 superscript ℓ 𝑛 subscript 𝑧 subscript superscript Σ 𝖥𝖳𝖫 : subscript 𝑡 𝑧 subscript 𝜏 𝑧 1 Reg subscript 𝖠𝖫𝖦 𝖶𝖢 superscript subscript ℓ subscript 𝜏 𝑧 1 subscript 𝑡 𝑧 1 1 1 \displaystyle\mathrm{Reg}(\mathsf{ALG},\ell^{n})\leq\sum_{z}\left(\Sigma^{%
\mathsf{FTL}}_{t_{z}\mathrel{\mathop{:}}\tau_{z}-1}+\mathrm{Reg}(\mathsf{ALG}_%
{\mathsf{WC}},\ell_{\tau_{z}+1}^{t_{z+1}-1})+1\right). roman_Reg ( sansserif_ALG , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) ≤ ∑ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT ( roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT : italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT - 1 end_POSTSUBSCRIPT + roman_Reg ( sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT , roman_ℓ start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_t start_POSTSUBSCRIPT italic_z + 1 end_POSTSUBSCRIPT - 1 end_POSTSUPERSCRIPT ) + 1 ) .
(15)
Proof.
We let t z last = n + 1 subscript 𝑡 subscript 𝑧 last 𝑛 1 t_{z_{\mathrm{last}}}=n+1 italic_t start_POSTSUBSCRIPT italic_z start_POSTSUBSCRIPT roman_last end_POSTSUBSCRIPT end_POSTSUBSCRIPT = italic_n + 1 , and a n + 1 = a n * subscript 𝑎 𝑛 1 superscript subscript 𝑎 𝑛 a_{n+1}=a_{n}^{*} italic_a start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT = italic_a start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT . We use Lemma 3 and rearrange the terms by grouping them by the FTL periods and the 𝖠𝖫𝖦 𝖶𝖢 subscript 𝖠𝖫𝖦 𝖶𝖢 \mathsf{ALG}_{\mathsf{WC}} sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT periods.
Reg Reg \displaystyle\mathrm{Reg} roman_Reg
( 𝖠𝖫𝖦 , ℓ n ) 𝖠𝖫𝖦 superscript ℓ 𝑛 \displaystyle(\mathsf{ALG},\ell^{n}) ( sansserif_ALG , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT )
= ∑ z ( ∑ t = t z t z + 1 − 1 ( L t ( a t ) − L t ( a t + 1 ) ) ) absent subscript 𝑧 superscript subscript 𝑡 subscript 𝑡 𝑧 subscript 𝑡 𝑧 1 1 subscript 𝐿 𝑡 subscript 𝑎 𝑡 subscript 𝐿 𝑡 subscript 𝑎 𝑡 1 \displaystyle=\sum_{z}\left(\sum_{t=t_{z}}^{t_{z+1}-1}(L_{t}(a_{t})-L_{t}(a_{t%
+1}))\right) = ∑ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT ( ∑ start_POSTSUBSCRIPT italic_t = italic_t start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_t start_POSTSUBSCRIPT italic_z + 1 end_POSTSUBSCRIPT - 1 end_POSTSUPERSCRIPT ( italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) - italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t + 1 end_POSTSUBSCRIPT ) ) )
= ∑ z ( ∑ t = t z τ z − 1 ( L t ( a t − 1 * ) − L t ( a t * ) ) + L τ z ( a τ z − 1 * ) − L τ z ( a τ z + 1 ) + ∑ t = τ z + 1 t z + 1 − 1 ( L t ( a t ) − L t ( a t + 1 ) ) ) absent subscript 𝑧 superscript subscript 𝑡 subscript 𝑡 𝑧 subscript 𝜏 𝑧 1 subscript 𝐿 𝑡 superscript subscript 𝑎 𝑡 1 subscript 𝐿 𝑡 superscript subscript 𝑎 𝑡 subscript 𝐿 subscript 𝜏 𝑧 superscript subscript 𝑎 subscript 𝜏 𝑧 1 subscript 𝐿 subscript 𝜏 𝑧 subscript 𝑎 subscript 𝜏 𝑧 1 superscript subscript 𝑡 subscript 𝜏 𝑧 1 subscript 𝑡 𝑧 1 1 subscript 𝐿 𝑡 subscript 𝑎 𝑡 subscript 𝐿 𝑡 subscript 𝑎 𝑡 1 \displaystyle=\sum_{z}\left(\sum_{t=t_{z}}^{\tau_{z}-1}(L_{t}(a_{t-1}^{*})-L_{%
t}(a_{t}^{*}))+L_{\tau_{z}}(a_{\tau_{z}-1}^{*})-L_{\tau_{z}}(a_{\tau_{z}+1})+%
\sum_{t=\tau_{z}+1}^{t_{z+1}-1}(L_{t}(a_{t})-L_{t}(a_{t+1}))\right) = ∑ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT ( ∑ start_POSTSUBSCRIPT italic_t = italic_t start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT - 1 end_POSTSUPERSCRIPT ( italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) - italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) ) + italic_L start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT - 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) - italic_L start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT + 1 end_POSTSUBSCRIPT ) + ∑ start_POSTSUBSCRIPT italic_t = italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_t start_POSTSUBSCRIPT italic_z + 1 end_POSTSUBSCRIPT - 1 end_POSTSUPERSCRIPT ( italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) - italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t + 1 end_POSTSUBSCRIPT ) ) )
= ∑ z ( ∑ t = t z τ z − 1 ( L t ( a t − 1 * ) − L t ( a t * ) ) + L τ z ( a τ z − 1 * ) + ∑ t = τ z + 1 t z + 1 − 1 ℓ t ( a t ) − L t z + 1 − 1 ( a t z + 1 ) ) absent subscript 𝑧 superscript subscript 𝑡 subscript 𝑡 𝑧 subscript 𝜏 𝑧 1 subscript 𝐿 𝑡 superscript subscript 𝑎 𝑡 1 subscript 𝐿 𝑡 superscript subscript 𝑎 𝑡 subscript 𝐿 subscript 𝜏 𝑧 superscript subscript 𝑎 subscript 𝜏 𝑧 1 superscript subscript 𝑡 subscript 𝜏 𝑧 1 subscript 𝑡 𝑧 1 1 subscript ℓ 𝑡 subscript 𝑎 𝑡 subscript 𝐿 subscript 𝑡 𝑧 1 1 subscript 𝑎 subscript 𝑡 𝑧 1 \displaystyle=\sum_{z}\left(\sum_{t=t_{z}}^{\tau_{z}-1}(L_{t}(a_{t-1}^{*})-L_{%
t}(a_{t}^{*}))+L_{\tau_{z}}(a_{\tau_{z}-1}^{*})+\sum_{t=\tau_{z}+1}^{t_{z+1}-1%
}\ell_{t}(a_{t})-L_{t_{z+1}-1}(a_{t_{z+1}})\right) = ∑ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT ( ∑ start_POSTSUBSCRIPT italic_t = italic_t start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT - 1 end_POSTSUPERSCRIPT ( italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) - italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) ) + italic_L start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT - 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) + ∑ start_POSTSUBSCRIPT italic_t = italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_t start_POSTSUBSCRIPT italic_z + 1 end_POSTSUBSCRIPT - 1 end_POSTSUPERSCRIPT roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) - italic_L start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT italic_z + 1 end_POSTSUBSCRIPT - 1 end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT italic_z + 1 end_POSTSUBSCRIPT end_POSTSUBSCRIPT ) )
= ∑ z ∑ t = t z τ z − 1 ( L t ( a t − 1 * ) − L t ( a t * ) ) + ∑ z ( ∑ t = τ z + 1 t z + 1 − 1 ℓ t ( a t ) + L τ z ( a τ z − 1 * ) − L t z + 1 − 1 ( a t z + 1 − 1 * ) ) . absent subscript 𝑧 superscript subscript 𝑡 subscript 𝑡 𝑧 subscript 𝜏 𝑧 1 subscript 𝐿 𝑡 superscript subscript 𝑎 𝑡 1 subscript 𝐿 𝑡 superscript subscript 𝑎 𝑡 subscript 𝑧 superscript subscript 𝑡 subscript 𝜏 𝑧 1 subscript 𝑡 𝑧 1 1 subscript ℓ 𝑡 subscript 𝑎 𝑡 subscript 𝐿 subscript 𝜏 𝑧 superscript subscript 𝑎 subscript 𝜏 𝑧 1 subscript 𝐿 subscript 𝑡 𝑧 1 1 superscript subscript 𝑎 subscript 𝑡 𝑧 1 1 \displaystyle=\sum_{z}\sum_{t=t_{z}}^{\tau_{z}-1}(L_{t}(a_{t-1}^{*})-L_{t}(a_{%
t}^{*}))+\sum_{z}\left(\sum_{t=\tau_{z}+1}^{t_{z+1}-1}\ell_{t}(a_{t})+L_{\tau_%
{z}}(a_{\tau_{z-1}}^{*})-L_{t_{z+1}-1}(a_{t_{z+1}-1}^{*})\right). = ∑ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT ∑ start_POSTSUBSCRIPT italic_t = italic_t start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT - 1 end_POSTSUPERSCRIPT ( italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) - italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) ) + ∑ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT ( ∑ start_POSTSUBSCRIPT italic_t = italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_t start_POSTSUBSCRIPT italic_z + 1 end_POSTSUBSCRIPT - 1 end_POSTSUPERSCRIPT roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) + italic_L start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT italic_z - 1 end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) - italic_L start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT italic_z + 1 end_POSTSUBSCRIPT - 1 end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT italic_z + 1 end_POSTSUBSCRIPT - 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) ) .
We bound the first term by the FTL regret. Recall the notation
Σ t z : τ z − 1 𝖥𝖳𝖫 : = ∑ t = t z τ z − 1 ( L t ( a t − 1 * ) − L t ( a t * ) ) . \displaystyle\Sigma^{\mathsf{FTL}}_{t_{z}\mathrel{\mathop{:}}\tau_{z}-1}%
\mathrel{\mathop{:}}=\sum_{t=t_{z}}^{\tau_{z}-1}(L_{t}(a_{t-1}^{*})-L_{t}(a_{t%
}^{*})). roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT : italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT - 1 end_POSTSUBSCRIPT : = ∑ start_POSTSUBSCRIPT italic_t = italic_t start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT - 1 end_POSTSUPERSCRIPT ( italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) - italic_L start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) ) .
(16)
Because we are playing FTL at both time τ z subscript 𝜏 𝑧 \tau_{z} italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT and τ z − 1 subscript 𝜏 𝑧 1 \tau_{z}-1 italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT - 1 , it holds that
L τ z ( a τ z − 1 * ) = L τ z − 1 ( a τ z − 1 * ) + ℓ τ z ( a τ z − 1 * ) ≤ L τ z ( a τ z * ) + 1 . subscript 𝐿 subscript 𝜏 𝑧 superscript subscript 𝑎 subscript 𝜏 𝑧 1 subscript 𝐿 subscript 𝜏 𝑧 1 superscript subscript 𝑎 subscript 𝜏 𝑧 1 subscript ℓ subscript 𝜏 𝑧 superscript subscript 𝑎 subscript 𝜏 𝑧 1 subscript 𝐿 subscript 𝜏 𝑧 superscript subscript 𝑎 subscript 𝜏 𝑧 1 L_{\tau_{z}}(a_{\tau_{z}-1}^{*})=L_{\tau_{z}-1}(a_{\tau_{z}-1}^{*})+\ell_{\tau%
_{z}}(a_{\tau_{z}-1}^{*})\leq L_{\tau_{z}}(a_{\tau_{z}}^{*})+1. italic_L start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT - 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) = italic_L start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT - 1 end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT - 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) + roman_ℓ start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT - 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) ≤ italic_L start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) + 1 .
To bound the second term, we will show that
∑ t = τ z + 1 t z + 1 − 1 superscript subscript 𝑡 subscript 𝜏 𝑧 1 subscript 𝑡 𝑧 1 1 \displaystyle\sum_{t=\tau_{z}+1}^{t_{z+1}-1} ∑ start_POSTSUBSCRIPT italic_t = italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_t start_POSTSUBSCRIPT italic_z + 1 end_POSTSUBSCRIPT - 1 end_POSTSUPERSCRIPT
ℓ t ( a t ) + L τ z ( a τ z − 1 * ) − L t z + 1 − 1 ( a t z + 1 − 1 * ) subscript ℓ 𝑡 subscript 𝑎 𝑡 subscript 𝐿 subscript 𝜏 𝑧 superscript subscript 𝑎 subscript 𝜏 𝑧 1 subscript 𝐿 subscript 𝑡 𝑧 1 1 superscript subscript 𝑎 subscript 𝑡 𝑧 1 1 \displaystyle\ell_{t}(a_{t})+L_{\tau_{z}}(a_{\tau_{z-1}}^{*})-L_{t_{z+1}-1}(a_%
{t_{z+1}-1}^{*}) roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) + italic_L start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT italic_z - 1 end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) - italic_L start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT italic_z + 1 end_POSTSUBSCRIPT - 1 end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT italic_z + 1 end_POSTSUBSCRIPT - 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT )
≤ ∑ t = τ z + 1 t z + 1 − 1 ℓ t ( a t ) + L τ z ( a τ z * ) − L t z + 1 − 1 ( a t z + 1 − 1 * ) + 1 absent superscript subscript 𝑡 subscript 𝜏 𝑧 1 subscript 𝑡 𝑧 1 1 subscript ℓ 𝑡 subscript 𝑎 𝑡 subscript 𝐿 subscript 𝜏 𝑧 superscript subscript 𝑎 subscript 𝜏 𝑧 subscript 𝐿 subscript 𝑡 𝑧 1 1 superscript subscript 𝑎 subscript 𝑡 𝑧 1 1 1 \displaystyle\leq\sum_{t=\tau_{z}+1}^{t_{z+1}-1}\ell_{t}(a_{t})+L_{\tau_{z}}(a%
_{\tau_{z}}^{*})-L_{t_{z+1}-1}(a_{t_{z+1}-1}^{*})+1 ≤ ∑ start_POSTSUBSCRIPT italic_t = italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_t start_POSTSUBSCRIPT italic_z + 1 end_POSTSUBSCRIPT - 1 end_POSTSUPERSCRIPT roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) + italic_L start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) - italic_L start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT italic_z + 1 end_POSTSUBSCRIPT - 1 end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT italic_z + 1 end_POSTSUBSCRIPT - 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) + 1
(17)
= ∑ t = τ z + 1 t z + 1 − 1 ℓ t ( a t ) + min a L τ z ( a ) − min a L t z + 1 − 1 ( a ) + 1 absent superscript subscript 𝑡 subscript 𝜏 𝑧 1 subscript 𝑡 𝑧 1 1 subscript ℓ 𝑡 subscript 𝑎 𝑡 subscript 𝑎 subscript 𝐿 subscript 𝜏 𝑧 𝑎 subscript 𝑎 subscript 𝐿 subscript 𝑡 𝑧 1 1 𝑎 1 \displaystyle=\sum_{t=\tau_{z}+1}^{t_{z+1}-1}\ell_{t}(a_{t})+\min_{a}L_{\tau_{%
z}}(a)-\min_{a}L_{t_{z+1}-1}(a)+1 = ∑ start_POSTSUBSCRIPT italic_t = italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_t start_POSTSUBSCRIPT italic_z + 1 end_POSTSUBSCRIPT - 1 end_POSTSUPERSCRIPT roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) + roman_min start_POSTSUBSCRIPT italic_a end_POSTSUBSCRIPT italic_L start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( italic_a ) - roman_min start_POSTSUBSCRIPT italic_a end_POSTSUBSCRIPT italic_L start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT italic_z + 1 end_POSTSUBSCRIPT - 1 end_POSTSUBSCRIPT ( italic_a ) + 1
(18)
≤ ∑ t = τ z + 1 t z + 1 − 1 ℓ t ( a t ) − min a ( L t z + 1 − 1 ( a ) − L τ z ( a ) ) + 1 absent superscript subscript 𝑡 subscript 𝜏 𝑧 1 subscript 𝑡 𝑧 1 1 subscript ℓ 𝑡 subscript 𝑎 𝑡 subscript 𝑎 subscript 𝐿 subscript 𝑡 𝑧 1 1 𝑎 subscript 𝐿 subscript 𝜏 𝑧 𝑎 1 \displaystyle\leq\sum_{t=\tau_{z}+1}^{t_{z+1}-1}\ell_{t}(a_{t})-\min_{a}\left(%
L_{t_{z+1}-1}(a)-L_{\tau_{z}}(a)\right)+1 ≤ ∑ start_POSTSUBSCRIPT italic_t = italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_t start_POSTSUBSCRIPT italic_z + 1 end_POSTSUBSCRIPT - 1 end_POSTSUPERSCRIPT roman_ℓ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) - roman_min start_POSTSUBSCRIPT italic_a end_POSTSUBSCRIPT ( italic_L start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT italic_z + 1 end_POSTSUBSCRIPT - 1 end_POSTSUBSCRIPT ( italic_a ) - italic_L start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( italic_a ) ) + 1
(19)
= Reg ( 𝖠𝖫𝖦 𝖶𝖢 , ℓ τ z + 1 t z + 1 − 1 ) + 1 . absent Reg subscript 𝖠𝖫𝖦 𝖶𝖢 superscript subscript ℓ subscript 𝜏 𝑧 1 subscript 𝑡 𝑧 1 1 1 \displaystyle=\mathrm{Reg}(\mathsf{ALG}_{\mathsf{WC}},\ell_{\tau_{z}+1}^{t_{z+%
1}-1})+1. = roman_Reg ( sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT , roman_ℓ start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_t start_POSTSUBSCRIPT italic_z + 1 end_POSTSUBSCRIPT - 1 end_POSTSUPERSCRIPT ) + 1 .
(20)
∎
We now complete the proof of Theorem 3 by showing that the conditions that determine the switching time between 𝖥𝖳𝖫 𝖥𝖳𝖫 \mathsf{FTL} sansserif_FTL and 𝖠𝖫𝖦 𝖶𝖢 subscript 𝖠𝖫𝖦 𝖶𝖢 \mathsf{ALG}_{\mathsf{WC}} sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT are appropriately chosen to upper bound Σ t z : τ z − 1 𝖥𝖳𝖫 subscript superscript Σ 𝖥𝖳𝖫 : subscript 𝑡 𝑧 subscript 𝜏 𝑧 1 \Sigma^{\mathsf{FTL}}_{t_{z}\mathrel{\mathop{:}}\tau_{z}-1} roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT : italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT - 1 end_POSTSUBSCRIPT and Reg ( 𝖠𝖫𝖦 𝖶𝖢 , ℓ τ z + 1 t z + 1 − 1 ) Reg subscript 𝖠𝖫𝖦 𝖶𝖢 superscript subscript ℓ subscript 𝜏 𝑧 1 subscript 𝑡 𝑧 1 1 \mathrm{Reg}(\mathsf{ALG}_{\mathsf{WC}},\ell_{\tau_{z}+1}^{t_{z+1}-1}) roman_Reg ( sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT , roman_ℓ start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_t start_POSTSUBSCRIPT italic_z + 1 end_POSTSUBSCRIPT - 1 end_POSTSUPERSCRIPT ) .
Firstly, note that for any z < z last 𝑧 subscript 𝑧 last z<z_{\mathrm{last}} italic_z < italic_z start_POSTSUBSCRIPT roman_last end_POSTSUBSCRIPT , Reg ( 𝖠𝖫𝖦 𝖶𝖢 , ℓ τ z + 1 t z + 1 − 1 ) > g ( L z * ) Reg subscript 𝖠𝖫𝖦 𝖶𝖢 superscript subscript ℓ subscript 𝜏 𝑧 1 subscript 𝑡 𝑧 1 1 𝑔 superscript subscript 𝐿 𝑧 \mathrm{Reg}(\mathsf{ALG}_{\mathsf{WC}},\ell_{\tau_{z}+1}^{t_{z+1}-1})>g(L_{z}%
^{*}) roman_Reg ( sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT , roman_ℓ start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_t start_POSTSUBSCRIPT italic_z + 1 end_POSTSUBSCRIPT - 1 end_POSTSUPERSCRIPT ) > italic_g ( italic_L start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) , which implies that L * > L z * superscript 𝐿 superscript subscript 𝐿 𝑧 L^{*}>L_{z}^{*} italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT > italic_L start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT . In particular, the epoch z last − 1 subscript 𝑧 last 1 z_{\mathrm{last}}-1 italic_z start_POSTSUBSCRIPT roman_last end_POSTSUBSCRIPT - 1 was exited, which implies that
2 z last − 1 log m ≤ L * ⟹ z last ≤ log ( L * log m ) + 1 superscript 2 subscript 𝑧 last 1 𝑚 superscript 𝐿 subscript 𝑧 last superscript 𝐿 𝑚 1 2^{z_{\mathrm{last}}-1}\log m\leq L^{*}\implies z_{\mathrm{last}}\leq\log\left%
(\frac{L^{*}}{\log m}\right)+1 2 start_POSTSUPERSCRIPT italic_z start_POSTSUBSCRIPT roman_last end_POSTSUBSCRIPT - 1 end_POSTSUPERSCRIPT roman_log italic_m ≤ italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ⟹ italic_z start_POSTSUBSCRIPT roman_last end_POSTSUBSCRIPT ≤ roman_log ( divide start_ARG italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT end_ARG start_ARG roman_log italic_m end_ARG ) + 1
By the stopping condition of the epoch, Reg ( 𝖠𝖫𝖦 𝖶𝖢 , ℓ τ z + 1 t z + 1 − 1 ) ≤ Reg ( 𝖠𝖫𝖦 𝖶𝖢 , ℓ τ z + 1 t z + 1 − 2 ) + 1 ≤ g ( L z * ) + 1 Reg subscript 𝖠𝖫𝖦 𝖶𝖢 superscript subscript ℓ subscript 𝜏 𝑧 1 subscript 𝑡 𝑧 1 1 Reg subscript 𝖠𝖫𝖦 𝖶𝖢 superscript subscript ℓ subscript 𝜏 𝑧 1 subscript 𝑡 𝑧 1 2 1 𝑔 superscript subscript 𝐿 𝑧 1 \mathrm{Reg}(\mathsf{ALG}_{\mathsf{WC}},\ell_{\tau_{z}+1}^{t_{z+1}-1})\leq%
\mathrm{Reg}(\mathsf{ALG}_{\mathsf{WC}},\ell_{\tau_{z}+1}^{t_{z+1}-2})+1\leq g%
(L_{z}^{*})+1 roman_Reg ( sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT , roman_ℓ start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_t start_POSTSUBSCRIPT italic_z + 1 end_POSTSUBSCRIPT - 1 end_POSTSUPERSCRIPT ) ≤ roman_Reg ( sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT , roman_ℓ start_POSTSUBSCRIPT italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_t start_POSTSUBSCRIPT italic_z + 1 end_POSTSUBSCRIPT - 2 end_POSTSUPERSCRIPT ) + 1 ≤ italic_g ( italic_L start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) + 1 , such that substituting into Lemma 4 implies
Reg ( 𝖲𝖬𝖠𝖱𝖳 , ℓ n ) ≤ ∑ z ( Σ t z : τ z − 1 𝖥𝖳𝖫 + g ( L z * ) + 2 ) . Reg 𝖲𝖬𝖠𝖱𝖳 superscript ℓ 𝑛 subscript 𝑧 subscript superscript Σ 𝖥𝖳𝖫 : subscript 𝑡 𝑧 subscript 𝜏 𝑧 1 𝑔 superscript subscript 𝐿 𝑧 2 \displaystyle\mathrm{Reg}(\mathsf{SMART},\ell^{n})\leq\sum_{z}\left(\Sigma^{%
\mathsf{FTL}}_{t_{z}\mathrel{\mathop{:}}\tau_{z}-1}+g(L_{z}^{*})+2\right). roman_Reg ( sansserif_SMART , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) ≤ ∑ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT ( roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT : italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT - 1 end_POSTSUBSCRIPT + italic_g ( italic_L start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) + 2 ) .
(21)
Also, it always holds that Σ t z : τ z − 1 𝖥𝖳𝖫 ≤ g ( L z * ) subscript superscript Σ 𝖥𝖳𝖫 : subscript 𝑡 𝑧 subscript 𝜏 𝑧 1 𝑔 superscript subscript 𝐿 𝑧 \Sigma^{\mathsf{FTL}}_{t_{z}\mathrel{\mathop{:}}\tau_{z}-1}\leq g(L_{z}^{*}) roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT : italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT - 1 end_POSTSUBSCRIPT ≤ italic_g ( italic_L start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) and for z < z last 𝑧 subscript 𝑧 last z<z_{\mathrm{last}} italic_z < italic_z start_POSTSUBSCRIPT roman_last end_POSTSUBSCRIPT , Σ t z : τ z 𝖥𝖳𝖫 > g ( L z * ) subscript superscript Σ 𝖥𝖳𝖫 : subscript 𝑡 𝑧 subscript 𝜏 𝑧 𝑔 superscript subscript 𝐿 𝑧 \Sigma^{\mathsf{FTL}}_{t_{z}\mathrel{\mathop{:}}\tau_{z}}>g(L_{z}^{*}) roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT : italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT end_POSTSUBSCRIPT > italic_g ( italic_L start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) . Therefore
∑ z z last − 1 g ( L z * ) < ∑ z Σ t z : τ z − 1 𝖥𝖳𝖫 ≤ Reg ( 𝖥𝖳𝖫 , ℓ n ) superscript subscript 𝑧 subscript 𝑧 last 1 𝑔 superscript subscript 𝐿 𝑧 subscript 𝑧 subscript superscript Σ 𝖥𝖳𝖫 : subscript 𝑡 𝑧 subscript 𝜏 𝑧 1 Reg 𝖥𝖳𝖫 superscript ℓ 𝑛 \sum_{z}^{z_{\mathrm{last}}-1}g(L_{z}^{*})<\sum_{z}\Sigma^{\mathsf{FTL}}_{t_{z%
}\mathrel{\mathop{:}}\tau_{z}-1}\leq\mathrm{Reg}(\mathsf{FTL},\ell^{n}) ∑ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_z start_POSTSUBSCRIPT roman_last end_POSTSUBSCRIPT - 1 end_POSTSUPERSCRIPT italic_g ( italic_L start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) < ∑ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT : italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT - 1 end_POSTSUBSCRIPT ≤ roman_Reg ( sansserif_FTL , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT )
and ∑ z Σ t z : τ z − 1 𝖥𝖳𝖫 ≤ ∑ z z last g ( L z * ) subscript 𝑧 subscript superscript Σ 𝖥𝖳𝖫 : subscript 𝑡 𝑧 subscript 𝜏 𝑧 1 superscript subscript 𝑧 subscript 𝑧 last 𝑔 superscript subscript 𝐿 𝑧 \sum_{z}\Sigma^{\mathsf{FTL}}_{t_{z}\mathrel{\mathop{:}}\tau_{z}-1}\leq\sum_{z%
}^{z_{\mathrm{last}}}g(L_{z}^{*}) ∑ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT : italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT - 1 end_POSTSUBSCRIPT ≤ ∑ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_z start_POSTSUBSCRIPT roman_last end_POSTSUBSCRIPT end_POSTSUPERSCRIPT italic_g ( italic_L start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) .
To put it all together, if ∑ z z last g ( L z * ) ≤ Reg ( 𝖥𝖳𝖫 , ℓ n ) superscript subscript 𝑧 subscript 𝑧 last 𝑔 superscript subscript 𝐿 𝑧 Reg 𝖥𝖳𝖫 superscript ℓ 𝑛 \sum_{z}^{z_{\mathrm{last}}}g(L_{z}^{*})\leq\mathrm{Reg}(\mathsf{FTL},\ell^{n}) ∑ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_z start_POSTSUBSCRIPT roman_last end_POSTSUBSCRIPT end_POSTSUPERSCRIPT italic_g ( italic_L start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) ≤ roman_Reg ( sansserif_FTL , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) , then
Reg ( 𝖲𝖬𝖠𝖱𝖳 , ℓ n ) ≤ 2 ∑ z g ( L z * ) + 2 z last . Reg 𝖲𝖬𝖠𝖱𝖳 superscript ℓ 𝑛 2 subscript 𝑧 𝑔 superscript subscript 𝐿 𝑧 2 subscript 𝑧 last \displaystyle\mathrm{Reg}(\mathsf{SMART},\ell^{n})\leq 2\sum_{z}g(L_{z}^{*})+2%
z_{\mathrm{last}}. roman_Reg ( sansserif_SMART , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) ≤ 2 ∑ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT italic_g ( italic_L start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) + 2 italic_z start_POSTSUBSCRIPT roman_last end_POSTSUBSCRIPT .
(22)
If Reg ( 𝖥𝖳𝖫 , ℓ n ) < ∑ z z last g ( L z * ) Reg 𝖥𝖳𝖫 superscript ℓ 𝑛 superscript subscript 𝑧 subscript 𝑧 last 𝑔 superscript subscript 𝐿 𝑧 \mathrm{Reg}(\mathsf{FTL},\ell^{n})<\sum_{z}^{z_{\mathrm{last}}}g(L_{z}^{*}) roman_Reg ( sansserif_FTL , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) < ∑ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_z start_POSTSUBSCRIPT roman_last end_POSTSUBSCRIPT end_POSTSUPERSCRIPT italic_g ( italic_L start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) , then it must be that in the last epoch the algorithm never switches to 𝖠𝖫𝖦 𝖶𝖢 subscript 𝖠𝖫𝖦 𝖶𝖢 \mathsf{ALG}_{\mathsf{WC}} sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT . If it switched to 𝖠𝖫𝖦 𝖶𝖢 subscript 𝖠𝖫𝖦 𝖶𝖢 \mathsf{ALG}_{\mathsf{WC}} sansserif_ALG start_POSTSUBSCRIPT sansserif_WC end_POSTSUBSCRIPT it would imply that Reg ( 𝖥𝖳𝖫 , ℓ n ) ≥ ∑ z Σ t z : τ z 𝖥𝖳𝖫 > ∑ z z last g ( L z * ) Reg 𝖥𝖳𝖫 superscript ℓ 𝑛 subscript 𝑧 subscript superscript Σ 𝖥𝖳𝖫 : subscript 𝑡 𝑧 subscript 𝜏 𝑧 superscript subscript 𝑧 subscript 𝑧 last 𝑔 superscript subscript 𝐿 𝑧 \mathrm{Reg}(\mathsf{FTL},\ell^{n})\geq\sum_{z}\Sigma^{\mathsf{FTL}}_{t_{z}%
\mathrel{\mathop{:}}\tau_{z}}>\sum_{z}^{z_{\mathrm{last}}}g(L_{z}^{*}) roman_Reg ( sansserif_FTL , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) ≥ ∑ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT roman_Σ start_POSTSUPERSCRIPT sansserif_FTL end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT : italic_τ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT end_POSTSUBSCRIPT > ∑ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_z start_POSTSUBSCRIPT roman_last end_POSTSUBSCRIPT end_POSTSUPERSCRIPT italic_g ( italic_L start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) which would violate the assumption that Reg ( 𝖥𝖳𝖫 , ℓ n ) < ∑ z z last g ( L z * ) Reg 𝖥𝖳𝖫 superscript ℓ 𝑛 superscript subscript 𝑧 subscript 𝑧 last 𝑔 superscript subscript 𝐿 𝑧 \mathrm{Reg}(\mathsf{FTL},\ell^{n})<\sum_{z}^{z_{\mathrm{last}}}g(L_{z}^{*}) roman_Reg ( sansserif_FTL , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) < ∑ start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_z start_POSTSUBSCRIPT roman_last end_POSTSUBSCRIPT end_POSTSUPERSCRIPT italic_g ( italic_L start_POSTSUBSCRIPT italic_z end_POSTSUBSCRIPT start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ) . Therefore it must be that
Reg ( 𝖲𝖬𝖠𝖱𝖳 , ℓ n ) ≤ 2 R e g ( 𝖥𝖳𝖫 , ℓ n ) + 2 z last . Reg 𝖲𝖬𝖠𝖱𝖳 superscript ℓ 𝑛 2 R e g 𝖥𝖳𝖫 superscript ℓ 𝑛 2 subscript 𝑧 last \displaystyle\mathrm{Reg}(\mathsf{SMART},\ell^{n})\leq 2\mathrm{Reg}(\mathsf{%
FTL},\ell^{n})+2z_{\mathrm{last}}. roman_Reg ( sansserif_SMART , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) ≤ 2 roman_R roman_e roman_g ( sansserif_FTL , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) + 2 italic_z start_POSTSUBSCRIPT roman_last end_POSTSUBSCRIPT .
(23)
As a result, it follows that (putting the L * ≤ log m superscript 𝐿 𝑚 L^{*}\leq\log m italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT ≤ roman_log italic_m and the L * > log m superscript 𝐿 𝑚 L^{*}>\log m italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT > roman_log italic_m cases together)
Reg Reg \displaystyle\mathrm{Reg} roman_Reg
( 𝖲𝖬𝖠𝖱𝖳 , ℓ n ) 𝖲𝖬𝖠𝖱𝖳 superscript ℓ 𝑛 \displaystyle(\mathsf{SMART},\ell^{n}) ( sansserif_SMART , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT )
≤ 2 min ( Reg ( 𝖥𝖳𝖫 , ℓ n ) , ∑ z = 0 log ( 1 + L * log m ) + 1 g ( 2 z log m ) ) + 2 log ( 1 + L * log m ) + 2 . absent 2 Reg 𝖥𝖳𝖫 superscript ℓ 𝑛 superscript subscript 𝑧 0 1 superscript 𝐿 𝑚 1 𝑔 superscript 2 𝑧 𝑚 2 1 superscript 𝐿 𝑚 2 \displaystyle\leq 2\min\left(\mathrm{Reg}(\mathsf{FTL},\ell^{n}),\sum_{z=0}^{%
\log\left(1+\frac{L^{*}}{\log m}\right)+1}g(2^{z}\log m)\right)+2\log\left(1+%
\frac{L^{*}}{\log m}\right)+2. ≤ 2 roman_min ( roman_Reg ( sansserif_FTL , roman_ℓ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) , ∑ start_POSTSUBSCRIPT italic_z = 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT roman_log ( 1 + divide start_ARG italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT end_ARG start_ARG roman_log italic_m end_ARG ) + 1 end_POSTSUPERSCRIPT italic_g ( 2 start_POSTSUPERSCRIPT italic_z end_POSTSUPERSCRIPT roman_log italic_m ) ) + 2 roman_log ( 1 + divide start_ARG italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT end_ARG start_ARG roman_log italic_m end_ARG ) + 2 .
(24)
A.1 Proof of Corollary 1
This follows from Theorem 5 and by calculating
∑ z = 0 log ( 1 + L * log m ) + 1 superscript subscript 𝑧 0 1 superscript 𝐿 𝑚 1 \displaystyle\sum_{z=0}^{\log\left(1+\frac{L^{*}}{\log m}\right)+1} ∑ start_POSTSUBSCRIPT italic_z = 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT roman_log ( 1 + divide start_ARG italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT end_ARG start_ARG roman_log italic_m end_ARG ) + 1 end_POSTSUPERSCRIPT
g ( 2 z log m ) 𝑔 superscript 2 𝑧 𝑚 \displaystyle g(2^{z}\log m) italic_g ( 2 start_POSTSUPERSCRIPT italic_z end_POSTSUPERSCRIPT roman_log italic_m )
= 2 2 log m ∑ z = 0 log ( 1 + L * log m ) + 1 2 z / 2 + κ log m log ( 1 + L * log m ) + κ log m absent 2 2 𝑚 superscript subscript 𝑧 0 1 superscript 𝐿 𝑚 1 superscript 2 𝑧 2 𝜅 𝑚 1 superscript 𝐿 𝑚 𝜅 𝑚 \displaystyle=2\sqrt{2}\log m\sum_{z=0}^{\log\left(1+\frac{L^{*}}{\log m}%
\right)+1}2^{z/2}+\kappa\log m\log\left(1+\frac{L^{*}}{\log m}\right)+\kappa\log
m = 2 square-root start_ARG 2 end_ARG roman_log italic_m ∑ start_POSTSUBSCRIPT italic_z = 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT roman_log ( 1 + divide start_ARG italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT end_ARG start_ARG roman_log italic_m end_ARG ) + 1 end_POSTSUPERSCRIPT 2 start_POSTSUPERSCRIPT italic_z / 2 end_POSTSUPERSCRIPT + italic_κ roman_log italic_m roman_log ( 1 + divide start_ARG italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT end_ARG start_ARG roman_log italic_m end_ARG ) + italic_κ roman_log italic_m
≤ log m 4 2 2 − 1 1 + L * log m + κ log m log ( 1 + L * log m ) + κ log m absent 𝑚 4 2 2 1 1 superscript 𝐿 𝑚 𝜅 𝑚 1 superscript 𝐿 𝑚 𝜅 𝑚 \displaystyle\leq\log m\frac{4\sqrt{2}}{\sqrt{2}-1}\sqrt{1+\frac{L^{*}}{\log m%
}}+\kappa\log m\log\left(1+\frac{L^{*}}{\log m}\right)+\kappa\log m ≤ roman_log italic_m divide start_ARG 4 square-root start_ARG 2 end_ARG end_ARG start_ARG square-root start_ARG 2 end_ARG - 1 end_ARG square-root start_ARG 1 + divide start_ARG italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT end_ARG start_ARG roman_log italic_m end_ARG end_ARG + italic_κ roman_log italic_m roman_log ( 1 + divide start_ARG italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT end_ARG start_ARG roman_log italic_m end_ARG ) + italic_κ roman_log italic_m
≤ 10 2 log 2 m + 2 L * log m + κ log m log ( 1 + L * log m ) + κ log m absent 10 2 superscript 2 𝑚 2 superscript 𝐿 𝑚 𝜅 𝑚 1 superscript 𝐿 𝑚 𝜅 𝑚 \displaystyle\leq 10\sqrt{2\log^{2}m+2L^{*}\log m}+\kappa\log m\log\left(1+%
\frac{L^{*}}{\log m}\right)+\kappa\log m ≤ 10 square-root start_ARG 2 roman_log start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT italic_m + 2 italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT roman_log italic_m end_ARG + italic_κ roman_log italic_m roman_log ( 1 + divide start_ARG italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT end_ARG start_ARG roman_log italic_m end_ARG ) + italic_κ roman_log italic_m
≤ (a) 10 2 L * log m + κ log ( 1 + L * log m ) log m + 10 2 log m + κ log m superscript (a) absent 10 2 superscript 𝐿 𝑚 𝜅 1 superscript 𝐿 𝑚 𝑚 10 2 𝑚 𝜅 𝑚 \displaystyle\stackrel{{\scriptstyle\text{(a)}}}{{\leq}}10\sqrt{2L^{*}\log m}+%
\kappa\log\left(1+\frac{L^{*}}{\log m}\right)\log m+10\sqrt{2}\log m+\kappa\log
m start_RELOP SUPERSCRIPTOP start_ARG ≤ end_ARG start_ARG (a) end_ARG end_RELOP 10 square-root start_ARG 2 italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT roman_log italic_m end_ARG + italic_κ roman_log ( 1 + divide start_ARG italic_L start_POSTSUPERSCRIPT * end_POSTSUPERSCRIPT end_ARG start_ARG roman_log italic_m end_ARG ) roman_log italic_m + 10 square-root start_ARG 2 end_ARG roman_log italic_m + italic_κ roman_log italic_m
where (a) follows since for nonnegative a , b 𝑎 𝑏
a,b italic_a , italic_b a + b ≤ a + b 𝑎 𝑏 𝑎 𝑏 \sqrt{a+b}\leq\sqrt{a}+\sqrt{b} square-root start_ARG italic_a + italic_b end_ARG ≤ square-root start_ARG italic_a end_ARG + square-root start_ARG italic_b end_ARG .
Appendix B Proof of Theorem 3
Consider a large even n 𝑛 n italic_n . We then have for horizon size n + 1 𝑛 1 n+1 italic_n + 1 , Reg ( 𝖥𝖳𝖫 , y n + 1 ) = c ( y n ) 2 Reg 𝖥𝖳𝖫 superscript 𝑦 𝑛 1 𝑐 superscript 𝑦 𝑛 2 \mathrm{Reg}(\mathsf{FTL},y^{n+1})=\frac{c(y^{n})}{2} roman_Reg ( sansserif_FTL , italic_y start_POSTSUPERSCRIPT italic_n + 1 end_POSTSUPERSCRIPT ) = divide start_ARG italic_c ( italic_y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) end_ARG start_ARG 2 end_ARG . Moreover, Reg ( 𝖢𝗈𝗏𝖾𝗋 , y n + 1 ) = f n + 1 Reg 𝖢𝗈𝗏𝖾𝗋 superscript 𝑦 𝑛 1 subscript 𝑓 𝑛 1 \mathrm{Reg}(\mathsf{Cover},y^{n+1})=f_{n+1} roman_Reg ( sansserif_Cover , italic_y start_POSTSUPERSCRIPT italic_n + 1 end_POSTSUPERSCRIPT ) = italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT . Let
p n , k : = ℙ [ c ( ϵ n ) = k + 1 ] . \displaystyle p_{n,k}\mathrel{\mathop{\mathrel{\mathop{:}}}}=\mathbb{P}[c(%
\epsilon^{n})=k+1]. italic_p start_POSTSUBSCRIPT italic_n , italic_k end_POSTSUBSCRIPT : = blackboard_P [ italic_c ( italic_ϵ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) = italic_k + 1 ] .
(25)
We then have,
𝔼 [ min { c ( ϵ n ) , 2 f n + 1 } ] 𝔼 𝑐 superscript italic-ϵ 𝑛 2 subscript 𝑓 𝑛 1 \displaystyle\operatorname{\mathbb{E}}[\min\{c(\epsilon^{n}),2f_{n+1}\}] blackboard_E [ roman_min { italic_c ( italic_ϵ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) , 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT } ]
= ∑ k = 1 n min { k , 2 f n + 1 } Pr [ c ( ϵ n ) = k ] absent superscript subscript 𝑘 1 𝑛 𝑘 2 subscript 𝑓 𝑛 1 Pr 𝑐 superscript italic-ϵ 𝑛 𝑘 \displaystyle=\sum_{k=1}^{n}\min\{k,2f_{n+1}\}\Pr[c(\epsilon^{n})=k] = ∑ start_POSTSUBSCRIPT italic_k = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT roman_min { italic_k , 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT } roman_Pr [ italic_c ( italic_ϵ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) = italic_k ]
= ∑ k = 1 n min { k , 2 f n + 1 } p n , k − 1 absent superscript subscript 𝑘 1 𝑛 𝑘 2 subscript 𝑓 𝑛 1 subscript 𝑝 𝑛 𝑘 1
\displaystyle=\sum_{k=1}^{n}\min\{k,2f_{n+1}\}p_{n,k-1} = ∑ start_POSTSUBSCRIPT italic_k = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT roman_min { italic_k , 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT } italic_p start_POSTSUBSCRIPT italic_n , italic_k - 1 end_POSTSUBSCRIPT
= ∑ k = 0 n min { k + 1 , 2 f n + 1 } p n , k absent superscript subscript 𝑘 0 𝑛 𝑘 1 2 subscript 𝑓 𝑛 1 subscript 𝑝 𝑛 𝑘
\displaystyle=\sum_{k=0}^{n}\min\{k+1,2f_{n+1}\}p_{n,k} = ∑ start_POSTSUBSCRIPT italic_k = 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT roman_min { italic_k + 1 , 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT } italic_p start_POSTSUBSCRIPT italic_n , italic_k end_POSTSUBSCRIPT
= ∑ k = 0 ⌊ 2 f n + 1 ⌋ − 1 ( k + 1 ) p n , k + 2 f n + 1 ℙ [ c ( ϵ n ) ≥ 2 f n + 1 ] absent superscript subscript 𝑘 0 2 subscript 𝑓 𝑛 1 1 𝑘 1 subscript 𝑝 𝑛 𝑘
2 subscript 𝑓 𝑛 1 ℙ delimited-[] 𝑐 superscript italic-ϵ 𝑛 2 subscript 𝑓 𝑛 1 \displaystyle=\sum_{k=0}^{\lfloor 2f_{n+1}\rfloor-1}(k+1)p_{n,k}+2f_{n+1}%
\mathbb{P}[c(\epsilon^{n})\geq 2f_{n+1}] = ∑ start_POSTSUBSCRIPT italic_k = 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ⌊ 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT ⌋ - 1 end_POSTSUPERSCRIPT ( italic_k + 1 ) italic_p start_POSTSUBSCRIPT italic_n , italic_k end_POSTSUBSCRIPT + 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT blackboard_P [ italic_c ( italic_ϵ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) ≥ 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT ]
= ∑ k = 0 ⌊ 2 f n + 1 ⌋ − 1 ( k + 1 ) p n , k + 2 f n + 1 ℙ [ c ( ϵ n ) ≥ 2 f n + 1 ] + ℙ [ c ( ϵ n ) ≤ ⌊ 2 f n + 1 ⌋ ] absent superscript subscript 𝑘 0 2 subscript 𝑓 𝑛 1 1 𝑘 1 subscript 𝑝 𝑛 𝑘
2 subscript 𝑓 𝑛 1 ℙ delimited-[] 𝑐 superscript italic-ϵ 𝑛 2 subscript 𝑓 𝑛 1 ℙ delimited-[] 𝑐 superscript italic-ϵ 𝑛 2 subscript 𝑓 𝑛 1 \displaystyle=\sum_{k=0}^{\lfloor 2f_{n+1}\rfloor-1}(k+1)p_{n,k}+2f_{n+1}%
\mathbb{P}[c(\epsilon^{n})\geq 2f_{n+1}]+\mathbb{P}[c(\epsilon^{n})\leq\lfloor
2%
f_{n+1}\rfloor] = ∑ start_POSTSUBSCRIPT italic_k = 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ⌊ 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT ⌋ - 1 end_POSTSUPERSCRIPT ( italic_k + 1 ) italic_p start_POSTSUBSCRIPT italic_n , italic_k end_POSTSUBSCRIPT + 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT blackboard_P [ italic_c ( italic_ϵ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) ≥ 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT ] + blackboard_P [ italic_c ( italic_ϵ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) ≤ ⌊ 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT ⌋ ]
= ∑ k = 0 ⌊ 2 f n + 1 ⌋ − 1 k p n , k + 2 f n + 1 ( 1 − ∑ k = 0 ⌊ 2 f n + 1 ⌋ − 1 p n , k ) + ℙ [ c ( ϵ n ) ≤ ⌊ 2 f n + 1 ⌋ ] absent superscript subscript 𝑘 0 2 subscript 𝑓 𝑛 1 1 𝑘 subscript 𝑝 𝑛 𝑘
2 subscript 𝑓 𝑛 1 1 superscript subscript 𝑘 0 2 subscript 𝑓 𝑛 1 1 subscript 𝑝 𝑛 𝑘
ℙ delimited-[] 𝑐 superscript italic-ϵ 𝑛 2 subscript 𝑓 𝑛 1 \displaystyle=\sum_{k=0}^{\lfloor 2f_{n+1}\rfloor-1}kp_{n,k}+2f_{n+1}\left(1-%
\sum_{k=0}^{\lfloor 2f_{n+1}\rfloor-1}p_{n,k}\right)+\mathbb{P}[c(\epsilon^{n}%
)\leq\lfloor 2f_{n+1}\rfloor] = ∑ start_POSTSUBSCRIPT italic_k = 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ⌊ 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT ⌋ - 1 end_POSTSUPERSCRIPT italic_k italic_p start_POSTSUBSCRIPT italic_n , italic_k end_POSTSUBSCRIPT + 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT ( 1 - ∑ start_POSTSUBSCRIPT italic_k = 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ⌊ 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT ⌋ - 1 end_POSTSUPERSCRIPT italic_p start_POSTSUBSCRIPT italic_n , italic_k end_POSTSUBSCRIPT ) + blackboard_P [ italic_c ( italic_ϵ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) ≤ ⌊ 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT ⌋ ]
(26)
Upon dividing 26 by 2 f n + 1 2 subscript 𝑓 𝑛 1 2f_{n+1} 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT , we get
𝔼 [ min { c ( ϵ n ) , 2 f n + 1 } ] 2 f n + 1 = ∑ k = 0 ⌊ 2 f n + 1 ⌋ − 1 k p n , k 2 f n + 1 + ( 1 − ∑ k = 0 ⌊ 2 f n + 1 ⌋ − 1 p n , k ) + ℙ [ c ( ϵ n ) ≤ ⌊ 2 f n + 1 ⌋ ] 2 f n + 1 . 𝔼 𝑐 superscript italic-ϵ 𝑛 2 subscript 𝑓 𝑛 1 2 subscript 𝑓 𝑛 1 superscript subscript 𝑘 0 2 subscript 𝑓 𝑛 1 1 𝑘 subscript 𝑝 𝑛 𝑘
2 subscript 𝑓 𝑛 1 1 superscript subscript 𝑘 0 2 subscript 𝑓 𝑛 1 1 subscript 𝑝 𝑛 𝑘
ℙ delimited-[] 𝑐 superscript italic-ϵ 𝑛 2 subscript 𝑓 𝑛 1 2 subscript 𝑓 𝑛 1 \displaystyle\frac{\operatorname{\mathbb{E}}[\min\{c(\epsilon^{n}),2f_{n+1}\}]%
}{2f_{n+1}}=\frac{\sum_{k=0}^{\lfloor 2f_{n+1}\rfloor-1}kp_{n,k}}{2f_{n+1}}+%
\left(1-\sum_{k=0}^{\lfloor 2f_{n+1}\rfloor-1}p_{n,k}\right)+\frac{\mathbb{P}[%
c(\epsilon^{n})\leq\lfloor 2f_{n+1}\rfloor]}{2f_{n+1}}. divide start_ARG blackboard_E [ roman_min { italic_c ( italic_ϵ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) , 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT } ] end_ARG start_ARG 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT end_ARG = divide start_ARG ∑ start_POSTSUBSCRIPT italic_k = 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ⌊ 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT ⌋ - 1 end_POSTSUPERSCRIPT italic_k italic_p start_POSTSUBSCRIPT italic_n , italic_k end_POSTSUBSCRIPT end_ARG start_ARG 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT end_ARG + ( 1 - ∑ start_POSTSUBSCRIPT italic_k = 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ⌊ 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT ⌋ - 1 end_POSTSUPERSCRIPT italic_p start_POSTSUBSCRIPT italic_n , italic_k end_POSTSUBSCRIPT ) + divide start_ARG blackboard_P [ italic_c ( italic_ϵ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) ≤ ⌊ 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT ⌋ ] end_ARG start_ARG 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT end_ARG .
(27)
Note that the third term vanishes since f n + 1 → ∞ → subscript 𝑓 𝑛 1 f_{n+1}\to\infty italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT → ∞ and 0 ≤ ℙ [ ⋅ ] ≤ 1 0 ℙ delimited-[] ⋅ 1 0\leq\mathbb{P}[\cdot]\leq 1 0 ≤ blackboard_P [ ⋅ ] ≤ 1 . We will now separately evaluate the first two terms in 27 . To do this, we require an auxiliary lemma, the proof of which is provided later.
Lemma 5 .
If k ≤ C n 𝑘 𝐶 𝑛 k\leq C\sqrt{n} italic_k ≤ italic_C square-root start_ARG italic_n end_ARG for an absolute constant C 𝐶 C italic_C , then for large enough n 𝑛 n italic_n (n ≥ 32 C 2 𝑛 32 superscript 𝐶 2 n\geq 32C^{2} italic_n ≥ 32 italic_C start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT suffices) we have
e − 16 C 3 n ≤ p n , k 2 n π e − k 2 / 2 n ≤ 1 − C / n 1 − 2 C / n e 16 C 3 n . superscript 𝑒 16 superscript 𝐶 3 𝑛 subscript 𝑝 𝑛 𝑘
2 𝑛 𝜋 superscript 𝑒 superscript 𝑘 2 2 𝑛 1 𝐶 𝑛 1 2 𝐶 𝑛 superscript 𝑒 16 superscript 𝐶 3 𝑛 \displaystyle e^{-\frac{16C^{3}}{\sqrt{n}}}\leq\frac{p_{n,k}}{\sqrt{\frac{2}{n%
\pi}}e^{-k^{2}/2n}}\leq\sqrt{\frac{1-C/\sqrt{n}}{1-2C/\sqrt{n}}}e^{\frac{16C^{%
3}}{\sqrt{n}}}. italic_e start_POSTSUPERSCRIPT - divide start_ARG 16 italic_C start_POSTSUPERSCRIPT 3 end_POSTSUPERSCRIPT end_ARG start_ARG square-root start_ARG italic_n end_ARG end_ARG end_POSTSUPERSCRIPT ≤ divide start_ARG italic_p start_POSTSUBSCRIPT italic_n , italic_k end_POSTSUBSCRIPT end_ARG start_ARG square-root start_ARG divide start_ARG 2 end_ARG start_ARG italic_n italic_π end_ARG end_ARG italic_e start_POSTSUPERSCRIPT - italic_k start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT / 2 italic_n end_POSTSUPERSCRIPT end_ARG ≤ square-root start_ARG divide start_ARG 1 - italic_C / square-root start_ARG italic_n end_ARG end_ARG start_ARG 1 - 2 italic_C / square-root start_ARG italic_n end_ARG end_ARG end_ARG italic_e start_POSTSUPERSCRIPT divide start_ARG 16 italic_C start_POSTSUPERSCRIPT 3 end_POSTSUPERSCRIPT end_ARG start_ARG square-root start_ARG italic_n end_ARG end_ARG end_POSTSUPERSCRIPT .
(28)
That is,
p n , k = 2 n π e − k 2 / 2 n ( 1 + o ( 1 ) ) . subscript 𝑝 𝑛 𝑘
2 𝑛 𝜋 superscript 𝑒 superscript 𝑘 2 2 𝑛 1 𝑜 1 p_{n,k}=\sqrt{\frac{2}{n\pi}}e^{-k^{2}/2n}(1+o(1)). italic_p start_POSTSUBSCRIPT italic_n , italic_k end_POSTSUBSCRIPT = square-root start_ARG divide start_ARG 2 end_ARG start_ARG italic_n italic_π end_ARG end_ARG italic_e start_POSTSUPERSCRIPT - italic_k start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT / 2 italic_n end_POSTSUPERSCRIPT ( 1 + italic_o ( 1 ) ) .
We now evaluate the first term in 27 . Since ⌊ 2 f n + 1 ⌋ − 1 ≤ 2 n 2 subscript 𝑓 𝑛 1 1 2 𝑛 \lfloor 2f_{n+1}\rfloor-1\leq 2\sqrt{n} ⌊ 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT ⌋ - 1 ≤ 2 square-root start_ARG italic_n end_ARG , we invoke Lemma 5 to evaluate
∑ k = 0 ⌊ 2 f n + 1 ⌋ − 1 k p n , k superscript subscript 𝑘 0 2 subscript 𝑓 𝑛 1 1 𝑘 subscript 𝑝 𝑛 𝑘
\displaystyle\sum_{k=0}^{\lfloor 2f_{n+1}\rfloor-1}kp_{n,k} ∑ start_POSTSUBSCRIPT italic_k = 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ⌊ 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT ⌋ - 1 end_POSTSUPERSCRIPT italic_k italic_p start_POSTSUBSCRIPT italic_n , italic_k end_POSTSUBSCRIPT
= ( 1 + o ( 1 ) ) ∑ k = 0 ⌊ 2 f n + 1 ⌋ − 1 k 2 n π e − k 2 2 n absent 1 𝑜 1 superscript subscript 𝑘 0 2 subscript 𝑓 𝑛 1 1 𝑘 2 𝑛 𝜋 superscript 𝑒 superscript 𝑘 2 2 𝑛 \displaystyle=(1+o(1))\sum_{k=0}^{\lfloor 2f_{n+1}\rfloor-1}k\sqrt{\frac{2}{n%
\pi}}e^{-\frac{k^{2}}{2n}} = ( 1 + italic_o ( 1 ) ) ∑ start_POSTSUBSCRIPT italic_k = 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ⌊ 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT ⌋ - 1 end_POSTSUPERSCRIPT italic_k square-root start_ARG divide start_ARG 2 end_ARG start_ARG italic_n italic_π end_ARG end_ARG italic_e start_POSTSUPERSCRIPT - divide start_ARG italic_k start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 italic_n end_ARG end_POSTSUPERSCRIPT
(29)
= ( 1 + o ( 1 ) ) 2 n π ∑ k = 0 ⌊ 2 f n + 1 ⌋ − 1 k e − k 2 2 n absent 1 𝑜 1 2 𝑛 𝜋 superscript subscript 𝑘 0 2 subscript 𝑓 𝑛 1 1 𝑘 superscript 𝑒 superscript 𝑘 2 2 𝑛 \displaystyle=(1+o(1))\sqrt{\frac{2}{n\pi}}\sum_{k=0}^{\lfloor 2f_{n+1}\rfloor%
-1}ke^{-\frac{k^{2}}{2n}} = ( 1 + italic_o ( 1 ) ) square-root start_ARG divide start_ARG 2 end_ARG start_ARG italic_n italic_π end_ARG end_ARG ∑ start_POSTSUBSCRIPT italic_k = 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ⌊ 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT ⌋ - 1 end_POSTSUPERSCRIPT italic_k italic_e start_POSTSUPERSCRIPT - divide start_ARG italic_k start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 italic_n end_ARG end_POSTSUPERSCRIPT
(30)
Now, we note that since x ↦ x e − x 2 2 n maps-to 𝑥 𝑥 superscript 𝑒 superscript 𝑥 2 2 𝑛 x\mapsto xe^{-\frac{x^{2}}{2n}} italic_x ↦ italic_x italic_e start_POSTSUPERSCRIPT - divide start_ARG italic_x start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 italic_n end_ARG end_POSTSUPERSCRIPT is increasing on ( 0 , ⌊ 2 f n + 1 ⌋ − 1 ) 0 2 subscript 𝑓 𝑛 1 1 (0,\lfloor 2f_{n+1}\rfloor-1) ( 0 , ⌊ 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT ⌋ - 1 ) , by a Riemann approximation, we have that
∫ 0 2 f n + 1 − 2 x e − x 2 2 n 𝑑 x − 1 ≤ ∑ k = 0 ⌊ 2 f n + 1 ⌋ − 1 k e − k 2 2 n ≤ ∫ 0 2 f n + 1 x e − x 2 2 n 𝑑 x superscript subscript 0 2 subscript 𝑓 𝑛 1 2 𝑥 superscript 𝑒 superscript 𝑥 2 2 𝑛 differential-d 𝑥 1 superscript subscript 𝑘 0 2 subscript 𝑓 𝑛 1 1 𝑘 superscript 𝑒 superscript 𝑘 2 2 𝑛 superscript subscript 0 2 subscript 𝑓 𝑛 1 𝑥 superscript 𝑒 superscript 𝑥 2 2 𝑛 differential-d 𝑥 \displaystyle\int_{0}^{2f_{n+1}-2}xe^{-\frac{x^{2}}{2n}}dx-1\leq\sum_{k=0}^{%
\lfloor 2f_{n+1}\rfloor-1}ke^{-\frac{k^{2}}{2n}}\leq\int_{0}^{2f_{n+1}}xe^{-%
\frac{x^{2}}{2n}}dx ∫ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT - 2 end_POSTSUPERSCRIPT italic_x italic_e start_POSTSUPERSCRIPT - divide start_ARG italic_x start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 italic_n end_ARG end_POSTSUPERSCRIPT italic_d italic_x - 1 ≤ ∑ start_POSTSUBSCRIPT italic_k = 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ⌊ 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT ⌋ - 1 end_POSTSUPERSCRIPT italic_k italic_e start_POSTSUPERSCRIPT - divide start_ARG italic_k start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 italic_n end_ARG end_POSTSUPERSCRIPT ≤ ∫ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT end_POSTSUPERSCRIPT italic_x italic_e start_POSTSUPERSCRIPT - divide start_ARG italic_x start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 italic_n end_ARG end_POSTSUPERSCRIPT italic_d italic_x
(31)
and evaluating
∫ 0 2 f n + 1 x e − x 2 2 n 𝑑 x superscript subscript 0 2 subscript 𝑓 𝑛 1 𝑥 superscript 𝑒 superscript 𝑥 2 2 𝑛 differential-d 𝑥 \displaystyle\int_{0}^{2f_{n+1}}xe^{-\frac{x^{2}}{2n}}dx ∫ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT end_POSTSUPERSCRIPT italic_x italic_e start_POSTSUPERSCRIPT - divide start_ARG italic_x start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 italic_n end_ARG end_POSTSUPERSCRIPT italic_d italic_x
= 1 2 ∫ 0 4 f n + 1 2 e − t 2 n 𝑑 t absent 1 2 superscript subscript 0 4 superscript subscript 𝑓 𝑛 1 2 superscript 𝑒 𝑡 2 𝑛 differential-d 𝑡 \displaystyle=\frac{1}{2}\int_{0}^{4f_{n+1}^{2}}e^{-\frac{t}{2n}}dt = divide start_ARG 1 end_ARG start_ARG 2 end_ARG ∫ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 4 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_POSTSUPERSCRIPT italic_e start_POSTSUPERSCRIPT - divide start_ARG italic_t end_ARG start_ARG 2 italic_n end_ARG end_POSTSUPERSCRIPT italic_d italic_t
= n ( 1 − e − 4 f n + 1 2 2 n ) absent 𝑛 1 superscript 𝑒 4 superscript subscript 𝑓 𝑛 1 2 2 𝑛 \displaystyle=n\left(1-e^{-\frac{4f_{n+1}^{2}}{2n}}\right) = italic_n ( 1 - italic_e start_POSTSUPERSCRIPT - divide start_ARG 4 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 italic_n end_ARG end_POSTSUPERSCRIPT )
= n ( 1 − e − 1 π ( 1 + o ( 1 ) ) ) . absent 𝑛 1 superscript 𝑒 1 𝜋 1 𝑜 1 \displaystyle=n\left(1-e^{-\frac{1}{\pi}(1+o(1))}\right). = italic_n ( 1 - italic_e start_POSTSUPERSCRIPT - divide start_ARG 1 end_ARG start_ARG italic_π end_ARG ( 1 + italic_o ( 1 ) ) end_POSTSUPERSCRIPT ) .
(32)
Evaluating the lower bound in 31 analogously we have
∑ k = 0 ⌊ 2 f n + 1 ⌋ − 1 k e − k 2 2 n = ( 1 + o ( 1 ) ) n ( 1 − e − 1 π ( 1 + o ( 1 ) ) ) superscript subscript 𝑘 0 2 subscript 𝑓 𝑛 1 1 𝑘 superscript 𝑒 superscript 𝑘 2 2 𝑛 1 𝑜 1 𝑛 1 superscript 𝑒 1 𝜋 1 𝑜 1 \displaystyle\sum_{k=0}^{\lfloor 2f_{n+1}\rfloor-1}ke^{-\frac{k^{2}}{2n}}=(1+o%
(1))n\left(1-e^{-\frac{1}{\pi}(1+o(1))}\right) ∑ start_POSTSUBSCRIPT italic_k = 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ⌊ 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT ⌋ - 1 end_POSTSUPERSCRIPT italic_k italic_e start_POSTSUPERSCRIPT - divide start_ARG italic_k start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 italic_n end_ARG end_POSTSUPERSCRIPT = ( 1 + italic_o ( 1 ) ) italic_n ( 1 - italic_e start_POSTSUPERSCRIPT - divide start_ARG 1 end_ARG start_ARG italic_π end_ARG ( 1 + italic_o ( 1 ) ) end_POSTSUPERSCRIPT )
(33)
and therefore from 30 ,
∑ k = 0 ⌊ 2 f n + 1 ⌋ − 1 k p n , k 2 f n + 1 superscript subscript 𝑘 0 2 subscript 𝑓 𝑛 1 1 𝑘 subscript 𝑝 𝑛 𝑘
2 subscript 𝑓 𝑛 1 \displaystyle\frac{\sum_{k=0}^{\lfloor 2f_{n+1}\rfloor-1}kp_{n,k}}{2f_{n+1}} divide start_ARG ∑ start_POSTSUBSCRIPT italic_k = 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ⌊ 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT ⌋ - 1 end_POSTSUPERSCRIPT italic_k italic_p start_POSTSUBSCRIPT italic_n , italic_k end_POSTSUBSCRIPT end_ARG start_ARG 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT end_ARG
= ( 1 + o ( 1 ) ) ( 1 − e − 1 π ( 1 + o ( 1 ) ) ) n 2 n π 2 ( n + 1 ) π absent 1 𝑜 1 1 superscript 𝑒 1 𝜋 1 𝑜 1 𝑛 2 𝑛 𝜋 2 𝑛 1 𝜋 \displaystyle=(1+o(1))\frac{\left(1-e^{-\frac{1}{\pi}(1+o(1))}\right)n\sqrt{%
\frac{2}{n\pi}}}{\sqrt{\frac{2(n+1)}{\pi}}} = ( 1 + italic_o ( 1 ) ) divide start_ARG ( 1 - italic_e start_POSTSUPERSCRIPT - divide start_ARG 1 end_ARG start_ARG italic_π end_ARG ( 1 + italic_o ( 1 ) ) end_POSTSUPERSCRIPT ) italic_n square-root start_ARG divide start_ARG 2 end_ARG start_ARG italic_n italic_π end_ARG end_ARG end_ARG start_ARG square-root start_ARG divide start_ARG 2 ( italic_n + 1 ) end_ARG start_ARG italic_π end_ARG end_ARG end_ARG
= ( 1 + o ( 1 ) ) ( 1 − e − 1 π ( 1 + o ( 1 ) ) ) absent 1 𝑜 1 1 superscript 𝑒 1 𝜋 1 𝑜 1 \displaystyle=(1+o(1))\left(1-e^{-\frac{1}{\pi}(1+o(1))}\right) = ( 1 + italic_o ( 1 ) ) ( 1 - italic_e start_POSTSUPERSCRIPT - divide start_ARG 1 end_ARG start_ARG italic_π end_ARG ( 1 + italic_o ( 1 ) ) end_POSTSUPERSCRIPT )
(34)
and therefore, from 34 , we have that
lim n → ∞ ∑ k = 0 ⌊ 2 f n + 1 ⌋ − 1 k p n , k 2 f n + 1 = 1 − e − 1 π . subscript → 𝑛 superscript subscript 𝑘 0 2 subscript 𝑓 𝑛 1 1 𝑘 subscript 𝑝 𝑛 𝑘
2 subscript 𝑓 𝑛 1 1 superscript 𝑒 1 𝜋 \displaystyle\lim_{n\to\infty}\frac{\sum_{k=0}^{\lfloor 2f_{n+1}\rfloor-1}kp_{%
n,k}}{2f_{n+1}}=1-e^{-\frac{1}{\pi}}. roman_lim start_POSTSUBSCRIPT italic_n → ∞ end_POSTSUBSCRIPT divide start_ARG ∑ start_POSTSUBSCRIPT italic_k = 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ⌊ 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT ⌋ - 1 end_POSTSUPERSCRIPT italic_k italic_p start_POSTSUBSCRIPT italic_n , italic_k end_POSTSUBSCRIPT end_ARG start_ARG 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT end_ARG = 1 - italic_e start_POSTSUPERSCRIPT - divide start_ARG 1 end_ARG start_ARG italic_π end_ARG end_POSTSUPERSCRIPT .
(35)
We now address the second term in 27 by invoking Lemma 5 and noting that
∑ k = 0 ⌊ 2 f n + 1 ⌋ − 1 p n , k superscript subscript 𝑘 0 2 subscript 𝑓 𝑛 1 1 subscript 𝑝 𝑛 𝑘
\displaystyle\sum_{k=0}^{\lfloor 2f_{n+1}\rfloor-1}p_{n,k} ∑ start_POSTSUBSCRIPT italic_k = 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ⌊ 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT ⌋ - 1 end_POSTSUPERSCRIPT italic_p start_POSTSUBSCRIPT italic_n , italic_k end_POSTSUBSCRIPT
= ( 1 + o ( 1 ) ) 2 n π ∑ k = 0 ⌊ 2 f n + 1 ⌋ − 1 e − k 2 2 n absent 1 𝑜 1 2 𝑛 𝜋 superscript subscript 𝑘 0 2 subscript 𝑓 𝑛 1 1 superscript 𝑒 superscript 𝑘 2 2 𝑛 \displaystyle=(1+o(1))\sqrt{\frac{2}{n\pi}}\sum_{k=0}^{\lfloor 2f_{n+1}\rfloor%
-1}e^{-\frac{k^{2}}{2n}} = ( 1 + italic_o ( 1 ) ) square-root start_ARG divide start_ARG 2 end_ARG start_ARG italic_n italic_π end_ARG end_ARG ∑ start_POSTSUBSCRIPT italic_k = 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ⌊ 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT ⌋ - 1 end_POSTSUPERSCRIPT italic_e start_POSTSUPERSCRIPT - divide start_ARG italic_k start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 italic_n end_ARG end_POSTSUPERSCRIPT
= ( a ) ( 1 + o ( 1 ) ) ∫ 0 2 f n + 1 e − x 2 2 n 𝑑 x superscript 𝑎 absent 1 𝑜 1 superscript subscript 0 2 subscript 𝑓 𝑛 1 superscript 𝑒 superscript 𝑥 2 2 𝑛 differential-d 𝑥 \displaystyle\stackrel{{\scriptstyle(a)}}{{=}}(1+o(1))\int_{0}^{2f_{n+1}}e^{-%
\frac{x^{2}}{2n}}dx start_RELOP SUPERSCRIPTOP start_ARG = end_ARG start_ARG ( italic_a ) end_ARG end_RELOP ( 1 + italic_o ( 1 ) ) ∫ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT end_POSTSUPERSCRIPT italic_e start_POSTSUPERSCRIPT - divide start_ARG italic_x start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 italic_n end_ARG end_POSTSUPERSCRIPT italic_d italic_x
= ( 1 + o ( 1 ) ) 2 π ∫ 0 2 π e − t 2 2 𝑑 t absent 1 𝑜 1 2 𝜋 superscript subscript 0 2 𝜋 superscript 𝑒 superscript 𝑡 2 2 differential-d 𝑡 \displaystyle=(1+o(1))\sqrt{\frac{2}{\pi}}\int_{0}^{\sqrt{\frac{2}{\pi}}}e^{-%
\frac{t^{2}}{2}}dt = ( 1 + italic_o ( 1 ) ) square-root start_ARG divide start_ARG 2 end_ARG start_ARG italic_π end_ARG end_ARG ∫ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT square-root start_ARG divide start_ARG 2 end_ARG start_ARG italic_π end_ARG end_ARG end_POSTSUPERSCRIPT italic_e start_POSTSUPERSCRIPT - divide start_ARG italic_t start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 end_ARG end_POSTSUPERSCRIPT italic_d italic_t
= 1 2 π ∫ − 2 π 2 π e − t 2 2 𝑑 t absent 1 2 𝜋 superscript subscript 2 𝜋 2 𝜋 superscript 𝑒 superscript 𝑡 2 2 differential-d 𝑡 \displaystyle=\frac{1}{\sqrt{2\pi}}\int_{-\sqrt{\frac{2}{\pi}}}^{\sqrt{\frac{2%
}{\pi}}}e^{-\frac{t^{2}}{2}}dt = divide start_ARG 1 end_ARG start_ARG square-root start_ARG 2 italic_π end_ARG end_ARG ∫ start_POSTSUBSCRIPT - square-root start_ARG divide start_ARG 2 end_ARG start_ARG italic_π end_ARG end_ARG end_POSTSUBSCRIPT start_POSTSUPERSCRIPT square-root start_ARG divide start_ARG 2 end_ARG start_ARG italic_π end_ARG end_ARG end_POSTSUPERSCRIPT italic_e start_POSTSUPERSCRIPT - divide start_ARG italic_t start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 end_ARG end_POSTSUPERSCRIPT italic_d italic_t
= ℙ ( − 2 π ≤ X ≤ 2 π ) absent ℙ 2 𝜋 𝑋 2 𝜋 \displaystyle=\mathbb{P}\left(-\sqrt{\frac{2}{\pi}}\leq X\leq\sqrt{\frac{2}{%
\pi}}\right) = blackboard_P ( - square-root start_ARG divide start_ARG 2 end_ARG start_ARG italic_π end_ARG end_ARG ≤ italic_X ≤ square-root start_ARG divide start_ARG 2 end_ARG start_ARG italic_π end_ARG end_ARG )
(36)
where in 36 X ∼ 𝒩 ( 0 , 1 ) similar-to 𝑋 𝒩 0 1 X\sim\mathcal{N}(0,1) italic_X ∼ caligraphic_N ( 0 , 1 ) . Then,
1 − ∑ k = 0 ⌊ 2 f n + 1 ⌋ − 1 p n , k 1 superscript subscript 𝑘 0 2 subscript 𝑓 𝑛 1 1 subscript 𝑝 𝑛 𝑘
\displaystyle 1-\sum_{k=0}^{\lfloor 2f_{n+1}\rfloor-1}p_{n,k} 1 - ∑ start_POSTSUBSCRIPT italic_k = 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ⌊ 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT ⌋ - 1 end_POSTSUPERSCRIPT italic_p start_POSTSUBSCRIPT italic_n , italic_k end_POSTSUBSCRIPT
→ 1 − ℙ ( − 2 π ≤ X ≤ 2 π ) → absent 1 ℙ 2 𝜋 𝑋 2 𝜋 \displaystyle\to 1-\mathbb{P}\left(-\sqrt{\frac{2}{\pi}}\leq X\leq\sqrt{\frac{%
2}{\pi}}\right) → 1 - blackboard_P ( - square-root start_ARG divide start_ARG 2 end_ARG start_ARG italic_π end_ARG end_ARG ≤ italic_X ≤ square-root start_ARG divide start_ARG 2 end_ARG start_ARG italic_π end_ARG end_ARG )
= 2 Q ( 2 π ) . absent 2 𝑄 2 𝜋 \displaystyle=2Q\left(\sqrt{\frac{2}{\pi}}\right). = 2 italic_Q ( square-root start_ARG divide start_ARG 2 end_ARG start_ARG italic_π end_ARG end_ARG ) .
(37)
Substituting 35 and 37 in 27 yields that
1 γ n = 𝔼 [ min { c ( ϵ n ) , 2 f n + 1 } ] 2 f n + 1 → 1 − e − 1 π + 2 Q ( 2 π ) 1 subscript 𝛾 𝑛 𝔼 𝑐 superscript italic-ϵ 𝑛 2 subscript 𝑓 𝑛 1 2 subscript 𝑓 𝑛 1 → 1 superscript 𝑒 1 𝜋 2 𝑄 2 𝜋 \displaystyle\frac{1}{\gamma_{n}}=\frac{\operatorname{\mathbb{E}}[\min\{c(%
\epsilon^{n}),2f_{n+1}\}]}{2f_{n+1}}\to 1-e^{-\frac{1}{\pi}}+2Q\left(\sqrt{%
\frac{2}{\pi}}\right) divide start_ARG 1 end_ARG start_ARG italic_γ start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT end_ARG = divide start_ARG blackboard_E [ roman_min { italic_c ( italic_ϵ start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) , 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT } ] end_ARG start_ARG 2 italic_f start_POSTSUBSCRIPT italic_n + 1 end_POSTSUBSCRIPT end_ARG → 1 - italic_e start_POSTSUPERSCRIPT - divide start_ARG 1 end_ARG start_ARG italic_π end_ARG end_POSTSUPERSCRIPT + 2 italic_Q ( square-root start_ARG divide start_ARG 2 end_ARG start_ARG italic_π end_ARG end_ARG )
(38)
and therefore
γ n → 1 1 − e − 1 π + 2 Q ( 2 π ) . → subscript 𝛾 𝑛 1 1 superscript 𝑒 1 𝜋 2 𝑄 2 𝜋 \displaystyle\gamma_{n}\to\frac{1}{1-e^{-\frac{1}{\pi}}+2Q\left(\sqrt{\frac{2}%
{\pi}}\right)}. italic_γ start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT → divide start_ARG 1 end_ARG start_ARG 1 - italic_e start_POSTSUPERSCRIPT - divide start_ARG 1 end_ARG start_ARG italic_π end_ARG end_POSTSUPERSCRIPT + 2 italic_Q ( square-root start_ARG divide start_ARG 2 end_ARG start_ARG italic_π end_ARG end_ARG ) end_ARG .
(39)
B.1 Proof of Lemma 5
We first note the following.
Proposition 1 (Feller , Chapter 3, Exercise 11).
p n , k = 1 2 n − k ( n − k n / 2 ) . subscript 𝑝 𝑛 𝑘
1 superscript 2 𝑛 𝑘 binomial 𝑛 𝑘 𝑛 2 p_{n,k}=\frac{1}{2^{n-k}}\binom{n-k}{n/2}. italic_p start_POSTSUBSCRIPT italic_n , italic_k end_POSTSUBSCRIPT = divide start_ARG 1 end_ARG start_ARG 2 start_POSTSUPERSCRIPT italic_n - italic_k end_POSTSUPERSCRIPT end_ARG ( FRACOP start_ARG italic_n - italic_k end_ARG start_ARG italic_n / 2 end_ARG ) .
Therefore,
p n , k 2 n 2 k = ( n − k n / 2 ) = ( n − k ) ! n 2 ! ( n 2 − k ) ! . subscript 𝑝 𝑛 𝑘
superscript 2 𝑛 superscript 2 𝑘 binomial 𝑛 𝑘 𝑛 2 𝑛 𝑘 𝑛 2 𝑛 2 𝑘 \frac{p_{n,k}2^{n}}{2^{k}}=\binom{n-k}{n/2}=\frac{(n-k)!}{\frac{n}{2}!\left(%
\frac{n}{2}-k\right)!}. divide start_ARG italic_p start_POSTSUBSCRIPT italic_n , italic_k end_POSTSUBSCRIPT 2 start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT end_ARG start_ARG 2 start_POSTSUPERSCRIPT italic_k end_POSTSUPERSCRIPT end_ARG = ( FRACOP start_ARG italic_n - italic_k end_ARG start_ARG italic_n / 2 end_ARG ) = divide start_ARG ( italic_n - italic_k ) ! end_ARG start_ARG divide start_ARG italic_n end_ARG start_ARG 2 end_ARG ! ( divide start_ARG italic_n end_ARG start_ARG 2 end_ARG - italic_k ) ! end_ARG .
We now use the Stirling approximation:
2 π m ( m e ) m e 1 12 m + 1 ≤ m ! ≤ 2 π m ( m e ) m e 1 12 m . 2 𝜋 𝑚 superscript 𝑚 𝑒 𝑚 superscript 𝑒 1 12 𝑚 1 𝑚 2 𝜋 𝑚 superscript 𝑚 𝑒 𝑚 superscript 𝑒 1 12 𝑚 \displaystyle\sqrt{2\pi m}\left(\frac{m}{e}\right)^{m}e^{\frac{1}{12m+1}}\leq m%
!\leq\sqrt{2\pi m}\left(\frac{m}{e}\right)^{m}e^{\frac{1}{12m}}. square-root start_ARG 2 italic_π italic_m end_ARG ( divide start_ARG italic_m end_ARG start_ARG italic_e end_ARG ) start_POSTSUPERSCRIPT italic_m end_POSTSUPERSCRIPT italic_e start_POSTSUPERSCRIPT divide start_ARG 1 end_ARG start_ARG 12 italic_m + 1 end_ARG end_POSTSUPERSCRIPT ≤ italic_m ! ≤ square-root start_ARG 2 italic_π italic_m end_ARG ( divide start_ARG italic_m end_ARG start_ARG italic_e end_ARG ) start_POSTSUPERSCRIPT italic_m end_POSTSUPERSCRIPT italic_e start_POSTSUPERSCRIPT divide start_ARG 1 end_ARG start_ARG 12 italic_m end_ARG end_POSTSUPERSCRIPT .
(40)
Using 40 we have
p n , k 2 n 2 k subscript 𝑝 𝑛 𝑘
superscript 2 𝑛 superscript 2 𝑘 \displaystyle\frac{p_{n,k}2^{n}}{2^{k}} divide start_ARG italic_p start_POSTSUBSCRIPT italic_n , italic_k end_POSTSUBSCRIPT 2 start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT end_ARG start_ARG 2 start_POSTSUPERSCRIPT italic_k end_POSTSUPERSCRIPT end_ARG
≤ 2 π ( n − k ) 2 π ( n / 2 ) 2 π ( n / 2 − k ) ⋅ ( n − k ) n − k ( n / 2 ) n / 2 ( n / 2 − k ) n / 2 − k absent ⋅ 2 𝜋 𝑛 𝑘 2 𝜋 𝑛 2 2 𝜋 𝑛 2 𝑘 superscript 𝑛 𝑘 𝑛 𝑘 superscript 𝑛 2 𝑛 2 superscript 𝑛 2 𝑘 𝑛 2 𝑘 \displaystyle\leq\frac{\sqrt{2\pi(n-k)}}{\sqrt{2\pi(n/2)}\sqrt{2\pi(n/2-k)}}%
\cdot\frac{(n-k)^{n-k}}{(n/2)^{n/2}(n/2-k)^{n/2-k}} ≤ divide start_ARG square-root start_ARG 2 italic_π ( italic_n - italic_k ) end_ARG end_ARG start_ARG square-root start_ARG 2 italic_π ( italic_n / 2 ) end_ARG square-root start_ARG 2 italic_π ( italic_n / 2 - italic_k ) end_ARG end_ARG ⋅ divide start_ARG ( italic_n - italic_k ) start_POSTSUPERSCRIPT italic_n - italic_k end_POSTSUPERSCRIPT end_ARG start_ARG ( italic_n / 2 ) start_POSTSUPERSCRIPT italic_n / 2 end_POSTSUPERSCRIPT ( italic_n / 2 - italic_k ) start_POSTSUPERSCRIPT italic_n / 2 - italic_k end_POSTSUPERSCRIPT end_ARG
⋅ exp ( 1 12 ( n − k ) − 1 6 n + 1 − 1 6 n − 12 k + 1 ) ⋅ absent 1 12 𝑛 𝑘 1 6 𝑛 1 1 6 𝑛 12 𝑘 1 \displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\cdot\exp{\left(\frac{1}{12(n%
-k)}-\frac{1}{6n+1}-\frac{1}{6n-12k+1}\right)} ⋅ roman_exp ( divide start_ARG 1 end_ARG start_ARG 12 ( italic_n - italic_k ) end_ARG - divide start_ARG 1 end_ARG start_ARG 6 italic_n + 1 end_ARG - divide start_ARG 1 end_ARG start_ARG 6 italic_n - 12 italic_k + 1 end_ARG )
= 2 n π n − k n − 2 k ( n − k ) n − k 2 n − k n n / 2 ( n − 2 k ) n / 2 − k ⋅ exp ( 1 12 n − k ) absent ⋅ 2 𝑛 𝜋 𝑛 𝑘 𝑛 2 𝑘 superscript 𝑛 𝑘 𝑛 𝑘 superscript 2 𝑛 𝑘 superscript 𝑛 𝑛 2 superscript 𝑛 2 𝑘 𝑛 2 𝑘 1 12 𝑛 𝑘 \displaystyle=\sqrt{\frac{2}{n\pi}}\sqrt{\frac{n-k}{n-2k}}\frac{(n-k)^{n-k}2^{%
n-k}}{n^{n/2}(n-2k)^{n/2-k}}\cdot\exp{\left(\frac{1}{12n-k}\right)} = square-root start_ARG divide start_ARG 2 end_ARG start_ARG italic_n italic_π end_ARG end_ARG square-root start_ARG divide start_ARG italic_n - italic_k end_ARG start_ARG italic_n - 2 italic_k end_ARG end_ARG divide start_ARG ( italic_n - italic_k ) start_POSTSUPERSCRIPT italic_n - italic_k end_POSTSUPERSCRIPT 2 start_POSTSUPERSCRIPT italic_n - italic_k end_POSTSUPERSCRIPT end_ARG start_ARG italic_n start_POSTSUPERSCRIPT italic_n / 2 end_POSTSUPERSCRIPT ( italic_n - 2 italic_k ) start_POSTSUPERSCRIPT italic_n / 2 - italic_k end_POSTSUPERSCRIPT end_ARG ⋅ roman_exp ( divide start_ARG 1 end_ARG start_ARG 12 italic_n - italic_k end_ARG )
≤ ( a ) 2 n π ⋅ 2 n − k ⋅ ( n − k ) n − k n n / 2 ( n − 2 k ) n / 2 − k exp ( 1 12 n − k ) ⋅ 1 − C / n 1 − 2 C / n superscript 𝑎 absent ⋅ ⋅ 2 𝑛 𝜋 superscript 2 𝑛 𝑘 superscript 𝑛 𝑘 𝑛 𝑘 superscript 𝑛 𝑛 2 superscript 𝑛 2 𝑘 𝑛 2 𝑘 1 12 𝑛 𝑘 1 𝐶 𝑛 1 2 𝐶 𝑛 \displaystyle\stackrel{{\scriptstyle(a)}}{{\leq}}\sqrt{\frac{2}{n\pi}}\cdot 2^%
{n-k}\cdot\frac{(n-k)^{n-k}}{n^{n/2}(n-2k)^{n/2-k}}\exp\left(\frac{1}{12n-k}%
\right)\cdot\sqrt{\frac{1-C/\sqrt{n}}{1-2C/\sqrt{n}}} start_RELOP SUPERSCRIPTOP start_ARG ≤ end_ARG start_ARG ( italic_a ) end_ARG end_RELOP square-root start_ARG divide start_ARG 2 end_ARG start_ARG italic_n italic_π end_ARG end_ARG ⋅ 2 start_POSTSUPERSCRIPT italic_n - italic_k end_POSTSUPERSCRIPT ⋅ divide start_ARG ( italic_n - italic_k ) start_POSTSUPERSCRIPT italic_n - italic_k end_POSTSUPERSCRIPT end_ARG start_ARG italic_n start_POSTSUPERSCRIPT italic_n / 2 end_POSTSUPERSCRIPT ( italic_n - 2 italic_k ) start_POSTSUPERSCRIPT italic_n / 2 - italic_k end_POSTSUPERSCRIPT end_ARG roman_exp ( divide start_ARG 1 end_ARG start_ARG 12 italic_n - italic_k end_ARG ) ⋅ square-root start_ARG divide start_ARG 1 - italic_C / square-root start_ARG italic_n end_ARG end_ARG start_ARG 1 - 2 italic_C / square-root start_ARG italic_n end_ARG end_ARG end_ARG
= 2 n π ⋅ 2 n − k ( 1 − k / n ) n − k ( 1 − 2 k / n ) n / 2 − k ⏟ = : T exp ( 1 12 n − k ) ⋅ 1 − C / n 1 − 2 C / n . absent ⋅ ⋅ 2 𝑛 𝜋 superscript 2 𝑛 𝑘 subscript ⏟ superscript 1 𝑘 𝑛 𝑛 𝑘 superscript 1 2 𝑘 𝑛 𝑛 2 𝑘 : absent 𝑇
1 12 𝑛 𝑘 1 𝐶 𝑛 1 2 𝐶 𝑛 \displaystyle=\sqrt{\frac{2}{n\pi}}\cdot 2^{n-k}\underbrace{\frac{(1-k/n)^{n-k%
}}{(1-2k/n)^{n/2-k}}}_{=\mathrel{\mathop{:}}T}\exp\left(\frac{1}{12n-k}\right)%
\cdot\sqrt{\frac{1-C/\sqrt{n}}{1-2C/\sqrt{n}}}. = square-root start_ARG divide start_ARG 2 end_ARG start_ARG italic_n italic_π end_ARG end_ARG ⋅ 2 start_POSTSUPERSCRIPT italic_n - italic_k end_POSTSUPERSCRIPT under⏟ start_ARG divide start_ARG ( 1 - italic_k / italic_n ) start_POSTSUPERSCRIPT italic_n - italic_k end_POSTSUPERSCRIPT end_ARG start_ARG ( 1 - 2 italic_k / italic_n ) start_POSTSUPERSCRIPT italic_n / 2 - italic_k end_POSTSUPERSCRIPT end_ARG end_ARG start_POSTSUBSCRIPT = : italic_T end_POSTSUBSCRIPT roman_exp ( divide start_ARG 1 end_ARG start_ARG 12 italic_n - italic_k end_ARG ) ⋅ square-root start_ARG divide start_ARG 1 - italic_C / square-root start_ARG italic_n end_ARG end_ARG start_ARG 1 - 2 italic_C / square-root start_ARG italic_n end_ARG end_ARG end_ARG .
(41)
We now analyze the term T 𝑇 T italic_T in 41 in more detail.
Proposition 2 .
− k 2 2 n 2 − c k 3 n 2 ≤ ln T ≤ − k 2 2 n 2 + c k 3 n 2 superscript 𝑘 2 2 superscript 𝑛 2 𝑐 superscript 𝑘 3 superscript 𝑛 2 𝑇 superscript 𝑘 2 2 superscript 𝑛 2 𝑐 superscript 𝑘 3 superscript 𝑛 2 \displaystyle-\frac{k^{2}}{2n^{2}}-c\frac{k^{3}}{n^{2}}\leq\ln T\leq-\frac{k^{%
2}}{2n^{2}}+c\frac{k^{3}}{n^{2}} - divide start_ARG italic_k start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 italic_n start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG - italic_c divide start_ARG italic_k start_POSTSUPERSCRIPT 3 end_POSTSUPERSCRIPT end_ARG start_ARG italic_n start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG ≤ roman_ln italic_T ≤ - divide start_ARG italic_k start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 italic_n start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + italic_c divide start_ARG italic_k start_POSTSUPERSCRIPT 3 end_POSTSUPERSCRIPT end_ARG start_ARG italic_n start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG
(42)
where c ≤ 15 𝑐 15 c\leq 15 italic_c ≤ 15 .
Proof.
By Taylor theorem, we have
ln ( 1 − x ) = − x − x 2 2 − x 3 3 ( 1 − μ ) 2 for μ ∈ ( 0 , x ) . 1 𝑥 𝑥 superscript 𝑥 2 2 superscript 𝑥 3 3 superscript 1 𝜇 2 for 𝜇 0 𝑥 \displaystyle\ln(1-x)=-x-\frac{x^{2}}{2}-\frac{x^{3}}{3(1-\mu)^{2}}\text{ for %
}\mu\in(0,x). roman_ln ( 1 - italic_x ) = - italic_x - divide start_ARG italic_x start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 end_ARG - divide start_ARG italic_x start_POSTSUPERSCRIPT 3 end_POSTSUPERSCRIPT end_ARG start_ARG 3 ( 1 - italic_μ ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG for italic_μ ∈ ( 0 , italic_x ) .
(43)
Therefore, for k ≤ C n 𝑘 𝐶 𝑛 k\leq C\sqrt{n} italic_k ≤ italic_C square-root start_ARG italic_n end_ARG and n ≥ 16 C 2 𝑛 16 superscript 𝐶 2 n\geq 16C^{2} italic_n ≥ 16 italic_C start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT we have
ln ( 1 − k n ) 1 𝑘 𝑛 \displaystyle\ln\left(1-\frac{k}{n}\right) roman_ln ( 1 - divide start_ARG italic_k end_ARG start_ARG italic_n end_ARG )
= − k n − k 2 2 n 2 − α 1 k 3 n 3 absent 𝑘 𝑛 superscript 𝑘 2 2 superscript 𝑛 2 subscript 𝛼 1 superscript 𝑘 3 superscript 𝑛 3 \displaystyle=-\frac{k}{n}-\frac{k^{2}}{2n^{2}}-\frac{\alpha_{1}k^{3}}{n^{3}} = - divide start_ARG italic_k end_ARG start_ARG italic_n end_ARG - divide start_ARG italic_k start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 italic_n start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG - divide start_ARG italic_α start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT italic_k start_POSTSUPERSCRIPT 3 end_POSTSUPERSCRIPT end_ARG start_ARG italic_n start_POSTSUPERSCRIPT 3 end_POSTSUPERSCRIPT end_ARG
(44)
ln ( 1 − 2 k n ) 1 2 𝑘 𝑛 \displaystyle\ln\left(1-\frac{2k}{n}\right) roman_ln ( 1 - divide start_ARG 2 italic_k end_ARG start_ARG italic_n end_ARG )
= − 2 k n − 2 k 2 2 n 2 − α 2 k 3 n 3 absent 2 𝑘 𝑛 2 superscript 𝑘 2 2 superscript 𝑛 2 subscript 𝛼 2 superscript 𝑘 3 superscript 𝑛 3 \displaystyle=-\frac{2k}{n}-\frac{2k^{2}}{2n^{2}}-\frac{\alpha_{2}k^{3}}{n^{3}} = - divide start_ARG 2 italic_k end_ARG start_ARG italic_n end_ARG - divide start_ARG 2 italic_k start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 italic_n start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG - divide start_ARG italic_α start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT italic_k start_POSTSUPERSCRIPT 3 end_POSTSUPERSCRIPT end_ARG start_ARG italic_n start_POSTSUPERSCRIPT 3 end_POSTSUPERSCRIPT end_ARG
(45)
for α 1 , α 2 ∈ ( 1 3 , 8 3 ] subscript 𝛼 1 subscript 𝛼 2
1 3 8 3 \alpha_{1},\alpha_{2}\in\Big{(}\frac{1}{3},\frac{8}{3}\Big{]} italic_α start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , italic_α start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ∈ ( divide start_ARG 1 end_ARG start_ARG 3 end_ARG , divide start_ARG 8 end_ARG start_ARG 3 end_ARG ] . Evaluating ln T 𝑇 \ln T roman_ln italic_T we have
ln T 𝑇 \displaystyle\ln T roman_ln italic_T
= ( n − k ) ln ( 1 − k n ) − ( n − k 2 ) ln ( 1 − 2 k n ) absent 𝑛 𝑘 1 𝑘 𝑛 𝑛 𝑘 2 1 2 𝑘 𝑛 \displaystyle=(n-k)\ln\left(1-\frac{k}{n}\right)-\left(n-\frac{k}{2}\right)\ln%
\left(1-\frac{2k}{n}\right) = ( italic_n - italic_k ) roman_ln ( 1 - divide start_ARG italic_k end_ARG start_ARG italic_n end_ARG ) - ( italic_n - divide start_ARG italic_k end_ARG start_ARG 2 end_ARG ) roman_ln ( 1 - divide start_ARG 2 italic_k end_ARG start_ARG italic_n end_ARG )
= ( a ) ( n − k ) ( − k n − k 2 2 n 2 − α 1 k 3 n 3 ) − ( n − k 2 ) ( − 2 k n − 2 k 2 2 n 2 − α 2 k 3 n 3 ) superscript 𝑎 absent 𝑛 𝑘 𝑘 𝑛 superscript 𝑘 2 2 superscript 𝑛 2 subscript 𝛼 1 superscript 𝑘 3 superscript 𝑛 3 𝑛 𝑘 2 2 𝑘 𝑛 2 superscript 𝑘 2 2 superscript 𝑛 2 subscript 𝛼 2 superscript 𝑘 3 superscript 𝑛 3 \displaystyle\stackrel{{\scriptstyle(a)}}{{=}}(n-k)\left(-\frac{k}{n}-\frac{k^%
{2}}{2n^{2}}-\frac{\alpha_{1}k^{3}}{n^{3}}\right)-\left(n-\frac{k}{2}\right)%
\left(-\frac{2k}{n}-\frac{2k^{2}}{2n^{2}}-\frac{\alpha_{2}k^{3}}{n^{3}}\right) start_RELOP SUPERSCRIPTOP start_ARG = end_ARG start_ARG ( italic_a ) end_ARG end_RELOP ( italic_n - italic_k ) ( - divide start_ARG italic_k end_ARG start_ARG italic_n end_ARG - divide start_ARG italic_k start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 italic_n start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG - divide start_ARG italic_α start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT italic_k start_POSTSUPERSCRIPT 3 end_POSTSUPERSCRIPT end_ARG start_ARG italic_n start_POSTSUPERSCRIPT 3 end_POSTSUPERSCRIPT end_ARG ) - ( italic_n - divide start_ARG italic_k end_ARG start_ARG 2 end_ARG ) ( - divide start_ARG 2 italic_k end_ARG start_ARG italic_n end_ARG - divide start_ARG 2 italic_k start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 italic_n start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG - divide start_ARG italic_α start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT italic_k start_POSTSUPERSCRIPT 3 end_POSTSUPERSCRIPT end_ARG start_ARG italic_n start_POSTSUPERSCRIPT 3 end_POSTSUPERSCRIPT end_ARG )
= ( − k 2 2 n + k 2 n + k 2 n − 2 k 2 n ) + ( − α 1 + 1 2 + α 2 2 − 2 ) k 3 n 2 + ( α 1 − α 2 ) k 4 n 3 absent superscript 𝑘 2 2 𝑛 superscript 𝑘 2 𝑛 superscript 𝑘 2 𝑛 2 superscript 𝑘 2 𝑛 subscript 𝛼 1 1 2 subscript 𝛼 2 2 2 superscript 𝑘 3 superscript 𝑛 2 subscript 𝛼 1 subscript 𝛼 2 superscript 𝑘 4 superscript 𝑛 3 \displaystyle=\left(-\frac{k^{2}}{2n}+\frac{k^{2}}{n}+\frac{k^{2}}{n}-\frac{2k%
^{2}}{n}\right)+\left(-\alpha_{1}+\frac{1}{2}+\frac{\alpha_{2}}{2}-2\right)%
\frac{k^{3}}{n^{2}}+\left(\alpha_{1}-\alpha_{2}\right)\frac{k^{4}}{n^{3}} = ( - divide start_ARG italic_k start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 italic_n end_ARG + divide start_ARG italic_k start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG italic_n end_ARG + divide start_ARG italic_k start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG italic_n end_ARG - divide start_ARG 2 italic_k start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG italic_n end_ARG ) + ( - italic_α start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT + divide start_ARG 1 end_ARG start_ARG 2 end_ARG + divide start_ARG italic_α start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_ARG start_ARG 2 end_ARG - 2 ) divide start_ARG italic_k start_POSTSUPERSCRIPT 3 end_POSTSUPERSCRIPT end_ARG start_ARG italic_n start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + ( italic_α start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT - italic_α start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ) divide start_ARG italic_k start_POSTSUPERSCRIPT 4 end_POSTSUPERSCRIPT end_ARG start_ARG italic_n start_POSTSUPERSCRIPT 3 end_POSTSUPERSCRIPT end_ARG
= − k 2 2 n 2 + c 1 k 3 n 2 + c 2 k 4 n 3 absent superscript 𝑘 2 2 superscript 𝑛 2 subscript 𝑐 1 superscript 𝑘 3 superscript 𝑛 2 subscript 𝑐 2 superscript 𝑘 4 superscript 𝑛 3 \displaystyle=-\frac{k^{2}}{2n^{2}}+c_{1}\frac{k^{3}}{n^{2}}+c_{2}\frac{k^{4}}%
{n^{3}} = - divide start_ARG italic_k start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 italic_n start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + italic_c start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT divide start_ARG italic_k start_POSTSUPERSCRIPT 3 end_POSTSUPERSCRIPT end_ARG start_ARG italic_n start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + italic_c start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT divide start_ARG italic_k start_POSTSUPERSCRIPT 4 end_POSTSUPERSCRIPT end_ARG start_ARG italic_n start_POSTSUPERSCRIPT 3 end_POSTSUPERSCRIPT end_ARG
(46)
where ( a ) 𝑎 (a) ( italic_a ) follows by substituting 44 and 45 . Now, since k 4 n 3 ≤ k 3 n 2 superscript 𝑘 4 superscript 𝑛 3 superscript 𝑘 3 superscript 𝑛 2 \frac{k^{4}}{n^{3}}\leq\frac{k^{3}}{n^{2}} divide start_ARG italic_k start_POSTSUPERSCRIPT 4 end_POSTSUPERSCRIPT end_ARG start_ARG italic_n start_POSTSUPERSCRIPT 3 end_POSTSUPERSCRIPT end_ARG ≤ divide start_ARG italic_k start_POSTSUPERSCRIPT 3 end_POSTSUPERSCRIPT end_ARG start_ARG italic_n start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG we have
ln T ≤ − k 2 2 n 2 + ( | c 1 | + | c 2 | ) k 3 n 2 𝑇 superscript 𝑘 2 2 superscript 𝑛 2 subscript 𝑐 1 subscript 𝑐 2 superscript 𝑘 3 superscript 𝑛 2 \displaystyle\ln T\leq-\frac{k^{2}}{2n^{2}}+(|c_{1}|+|c_{2}|)\frac{k^{3}}{n^{2}} roman_ln italic_T ≤ - divide start_ARG italic_k start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 italic_n start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + ( | italic_c start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT | + | italic_c start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT | ) divide start_ARG italic_k start_POSTSUPERSCRIPT 3 end_POSTSUPERSCRIPT end_ARG start_ARG italic_n start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG
(47)
and on the other hand for the same reason
ln T ≥ − k 2 2 n − ( | c 1 | + | c 2 | ) k 3 n 2 𝑇 superscript 𝑘 2 2 𝑛 subscript 𝑐 1 subscript 𝑐 2 superscript 𝑘 3 superscript 𝑛 2 \displaystyle\ln T\geq-\frac{k^{2}}{2n}-(|c_{1}|+|c_{2}|)\frac{k^{3}}{n^{2}} roman_ln italic_T ≥ - divide start_ARG italic_k start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 italic_n end_ARG - ( | italic_c start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT | + | italic_c start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT | ) divide start_ARG italic_k start_POSTSUPERSCRIPT 3 end_POSTSUPERSCRIPT end_ARG start_ARG italic_n start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG
(48)
The proposition follows by noticing | c 1 | + | c 2 | ≤ 15 subscript 𝑐 1 subscript 𝑐 2 15 |c_{1}|+|c_{2}|\leq 15 | italic_c start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT | + | italic_c start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT | ≤ 15 by using that α 1 , α 2 ∈ ( 1 3 , 8 3 ] subscript 𝛼 1 subscript 𝛼 2
1 3 8 3 \alpha_{1},\alpha_{2}\in\left(\frac{1}{3},\frac{8}{3}\right] italic_α start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , italic_α start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ∈ ( divide start_ARG 1 end_ARG start_ARG 3 end_ARG , divide start_ARG 8 end_ARG start_ARG 3 end_ARG ] .
∎
Using Proposition 2 in 41 we have the upper bound
p n , k subscript 𝑝 𝑛 𝑘
\displaystyle p_{n,k} italic_p start_POSTSUBSCRIPT italic_n , italic_k end_POSTSUBSCRIPT
≤ 2 n π ⋅ exp ( − k 2 2 n + 15 k 3 n 2 ) ⋅ exp ( 1 12 n − k ) ⋅ 1 − C / n 1 − 2 C / n absent ⋅ 2 𝑛 𝜋 superscript 𝑘 2 2 𝑛 15 superscript 𝑘 3 superscript 𝑛 2 1 12 𝑛 𝑘 1 𝐶 𝑛 1 2 𝐶 𝑛 \displaystyle\leq\sqrt{\frac{2}{n\pi}}\cdot\exp\left(-\frac{k^{2}}{2n}+\frac{1%
5k^{3}}{n^{2}}\right)\cdot\exp\left(\frac{1}{12n-k}\right)\cdot\sqrt{\frac{1-C%
/\sqrt{n}}{1-2C/\sqrt{n}}} ≤ square-root start_ARG divide start_ARG 2 end_ARG start_ARG italic_n italic_π end_ARG end_ARG ⋅ roman_exp ( - divide start_ARG italic_k start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 italic_n end_ARG + divide start_ARG 15 italic_k start_POSTSUPERSCRIPT 3 end_POSTSUPERSCRIPT end_ARG start_ARG italic_n start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG ) ⋅ roman_exp ( divide start_ARG 1 end_ARG start_ARG 12 italic_n - italic_k end_ARG ) ⋅ square-root start_ARG divide start_ARG 1 - italic_C / square-root start_ARG italic_n end_ARG end_ARG start_ARG 1 - 2 italic_C / square-root start_ARG italic_n end_ARG end_ARG end_ARG
≤ ( a ) 2 n π ⋅ exp ( − k 2 2 n ) ⋅ exp ( 16 k 3 n 2 ) ⋅ 1 − C / n 1 − 2 C / n superscript 𝑎 absent ⋅ 2 𝑛 𝜋 superscript 𝑘 2 2 𝑛 16 superscript 𝑘 3 superscript 𝑛 2 1 𝐶 𝑛 1 2 𝐶 𝑛 \displaystyle\stackrel{{\scriptstyle(a)}}{{\leq}}\sqrt{\frac{2}{n\pi}}\cdot%
\exp\left(-\frac{k^{2}}{2n}\right)\cdot\exp\left(\frac{16k^{3}}{n^{2}}\right)%
\cdot\sqrt{\frac{1-C/\sqrt{n}}{1-2C/\sqrt{n}}} start_RELOP SUPERSCRIPTOP start_ARG ≤ end_ARG start_ARG ( italic_a ) end_ARG end_RELOP square-root start_ARG divide start_ARG 2 end_ARG start_ARG italic_n italic_π end_ARG end_ARG ⋅ roman_exp ( - divide start_ARG italic_k start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 italic_n end_ARG ) ⋅ roman_exp ( divide start_ARG 16 italic_k start_POSTSUPERSCRIPT 3 end_POSTSUPERSCRIPT end_ARG start_ARG italic_n start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG ) ⋅ square-root start_ARG divide start_ARG 1 - italic_C / square-root start_ARG italic_n end_ARG end_ARG start_ARG 1 - 2 italic_C / square-root start_ARG italic_n end_ARG end_ARG end_ARG
≤ ( b ) 2 n π ⋅ exp ( − k 2 2 n ) ⋅ exp ( 16 C 3 n ) ⋅ 1 − C / n 1 − 2 C / n superscript 𝑏 absent ⋅ 2 𝑛 𝜋 superscript 𝑘 2 2 𝑛 16 superscript 𝐶 3 𝑛 1 𝐶 𝑛 1 2 𝐶 𝑛 \displaystyle\stackrel{{\scriptstyle(b)}}{{\leq}}\sqrt{\frac{2}{n\pi}}\cdot%
\exp\left(-\frac{k^{2}}{2n}\right)\cdot\exp\left(\frac{16C^{3}}{\sqrt{n}}%
\right)\cdot\sqrt{\frac{1-C/\sqrt{n}}{1-2C/\sqrt{n}}} start_RELOP SUPERSCRIPTOP start_ARG ≤ end_ARG start_ARG ( italic_b ) end_ARG end_RELOP square-root start_ARG divide start_ARG 2 end_ARG start_ARG italic_n italic_π end_ARG end_ARG ⋅ roman_exp ( - divide start_ARG italic_k start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 italic_n end_ARG ) ⋅ roman_exp ( divide start_ARG 16 italic_C start_POSTSUPERSCRIPT 3 end_POSTSUPERSCRIPT end_ARG start_ARG square-root start_ARG italic_n end_ARG end_ARG ) ⋅ square-root start_ARG divide start_ARG 1 - italic_C / square-root start_ARG italic_n end_ARG end_ARG start_ARG 1 - 2 italic_C / square-root start_ARG italic_n end_ARG end_ARG end_ARG
(49)
where ( a ) 𝑎 (a) ( italic_a ) uses 1 12 n − k + 15 k 3 n 2 ≤ 16 k 3 n 2 1 12 𝑛 𝑘 15 superscript 𝑘 3 superscript 𝑛 2 16 superscript 𝑘 3 superscript 𝑛 2 \frac{1}{12n-k}+\frac{15k^{3}}{n^{2}}\leq\frac{16k^{3}}{n^{2}} divide start_ARG 1 end_ARG start_ARG 12 italic_n - italic_k end_ARG + divide start_ARG 15 italic_k start_POSTSUPERSCRIPT 3 end_POSTSUPERSCRIPT end_ARG start_ARG italic_n start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG ≤ divide start_ARG 16 italic_k start_POSTSUPERSCRIPT 3 end_POSTSUPERSCRIPT end_ARG start_ARG italic_n start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG , and ( b ) 𝑏 (b) ( italic_b ) uses the fact that k ≤ C n 𝑘 𝐶 𝑛 k\leq C\sqrt{n} italic_k ≤ italic_C square-root start_ARG italic_n end_ARG , which yields the upper bound in 28 . The lower bound follows analogously by the Stirling approximation 40 and Proposition 2 .