Smooth Contextual Bandits: Bridging the Parametric and Non-differentiable Regret Regimes: The SmoothBandit Algorithm Has Regret Õ(T^((β+d−αβ)/(2β+d)) + 1) on Every Hölder-Smooth InstanceResearch Paper
Motivation
In a contextual bandit a decision maker repeatedly observes a context , chooses one of two actions, and sees the reward of that action only. Personalised medicine, dynamic pricing and online advertising have this structure: the reward of the action not taken is never observed, so learning and earning must be balanced. Performance is measured by the regret, the expected reward lost against the oracle that knows which action is better at every context.
Two regimes were well understood before this paper. When the mean rewards are linear (parametric) in the context, the regret is of order (up to dimension factors) under a sharp margin condition (Bastani and Bayati 2020, doi:10.1287/opre.2019.1902). When they are only Lipschitz or Hölder with exponent at most one (non-differentiable), the optimal regret is (Rigollet and Zeevi 2010, arXiv:1003.1630; Perchet and Rigollet 2013, arXiv:1110.6084). Hu, Kallus and Mao (arXiv:1909.02553, Operations Research 2022) treat mean rewards with derivatives' worth of smoothness and obtain a rate that moves continuously from the non-differentiable rate at to the parametric rate as .
Setting
The context takes values in a compact set and has a Lebesgue density between and on . At each round , nature draws i.i.d., with rewards ; the decision maker sees , pulls an arm using the history, and receives . Write , the conditional average treatment effect , the oracle policy if and otherwise, and the regret
For let be the largest integer strictly below . A function is -Hölder if it is -times continuously differentiable and its Taylor polynomial of order at approximates it at within . A Lebesgue-measurable set is weakly -regular at if for the closed ball , and a -regular set if this holds at each of its points for all . The class consists of the instances with:
- both -Hölder and -Lipschitz, with ;
- the optimal regions non-empty and -regular;
- the density condition above;
- the margin condition for all .
The algorithm, SmoothBandit (Algorithm 1), partitions into cubes of side and runs in epochs of geometrically growing length , with error tolerance . Each cube is assigned to an exploitation region (always pull that arm) or to the randomisation region (toss a fair coin). After each epoch it estimates at the cube centres by local polynomial regression of degree with bandwidth , using data from inside and outside each cube. It moves a cube to when the estimated effect exceeds in favour of arm , and hands to the other arm the cubes where the data of arm are too sparse to estimate (the inestimable regions ).
Formalization targets
Goal: Theorem 2 (p. 25)
With , uniformly over the class (constants depending only on the parameters):
for all , where is Algorithm 1 run with horizon .
Milestones
In the order the proof uses them:
- Lemma 1: each arm is optimal with probability at least .
- Lemma 2: the number of epochs is at most .
- Lemma 3: the bandwidths are at least .
- Lemma 14: inestimable cubes are cubes where the data support is not regular.
- Lemma 5: on the good events the regions behave as intended.
- The regret decomposition (13):
- Lemma 16: uniform concentration of the local polynomial estimates.
- Theorem 1: .
Significance
Together with the paper's lower bound (Theorem 3, not part of this mission), Theorem 2 identifies the minimax regret of smooth contextual bandits as up to logarithmic factors when . The two known regimes are endpoints of this rate: recovers the non-differentiable rate , and as the exponent tends to the parametric value , which is without a margin condition () and polylogarithmic under a sharp margin (). The result says how much the smoothness of the reward functions is worth in a sequential decision problem, and its algorithm shows that local polynomial estimators, which use data outside each cube, are needed to get there.
The theorem is proved in the paper; no machine-checked proof of it, or of any smooth () nonparametric bandit bound, is known. A formal proof would also check the paper's analysis, which has gaps recorded in this mission: Theorem 2's -form fails for when and its explicit constant needs , Lemma 5's fourth statement needs the sample-size event of the current epoch, and a display of Lemma 3 is misprinted. The definitions layer (the Hölder class with derivatives, regular sets, the local polynomial estimator, an epoch-based algorithm on a probability space with coins) is reusable for other nonparametric bandit and estimation results.
Difficulty
The goal's own proof is short once its ingredients are available: the decomposition (13), Theorem 1 and Lemma 2, then a geometric sum. The weight is in Theorem 1 and Lemma 16. The samples of each epoch are collected adaptively: which contexts are sampled for arm depends on regions computed from earlier data, so the samples are i.i.d. only conditionally on the past, with a conditional density that vanishes on the other arm's exploitation region. The classical analysis of local polynomial regression assumes a design density bounded below on a neighbourhood of every point and so does not apply. The paper restores a usable condition through the regularity of the regions (Assumption 2) and the screening rule (6); making this rigorous requires conditioning on events of the past epochs, a lower bound on the smallest eigenvalue of the local Gram matrices uniformly over the grid, and a union bound over cubes. Bounding the estimation error at one point is not enough: the accuracy event is a supremum over a region that is itself random.
Formalization scope
Contexts live in EuclideanSpace ℝ (Fin d) with ; arms are Bool (true is ); Lean rounds start at . An instance is a structure on a probability space that carries the contexts, the potential rewards of both arms, i.i.d. fair coins independent of the data, the support and the mean rewards . Its laws are stated through iIndepFun, identical distributions and the integral form of . The Hölder condition uses on an open neighbourhood of and iteratedFDeriv. The density condition is stated on measurable subsets of . All standing parameter conditions are explicit (, , , , , , ).
Algorithm 1 is a concrete, total function of the contexts, the coins and the rewards of the pulled arms, with the paper's constants , , , the epoch lengths (10) and every tie-break as printed. Its conventions:
- regions are finite sets of grid cells;
- the bandwidth uses ;
- uses Lebesgue volume (equivalent under Assumption 3);
- the local polynomial estimate is exactly when the least-squares minimiser is not unique.
The goal must be proved for this algorithm. A statement for an abstract policy assumed to satisfy Lemma 5, or with regions or good events taken as hypotheses, would reduce Theorem 2 to the decomposition (13) and is not the theorem. The constants , and are chosen before the instance. The -form clause excludes only the parameters where it fails as printed ( with ) and the boundary .
Welcome contributions: proofs of the milestones; the conditional i.i.d. structure (Lemma 6) and the Gram-matrix bound (Lemma 7), which are used inside Lemma 16 but not stated here; and general lemmas on local polynomial regression and on regular sets.
Selected references
- Y. Hu, N. Kallus, X. Mao, Smooth Contextual Bandits: Bridging the Parametric and Non-differentiable Regret Regimes, Operations Research 70(6), 2022; arXiv:1909.02553v4 (2020). https://arxiv.org/abs/1909.02553
- P. Rigollet, A. Zeevi, Nonparametric Bandits with Covariates, COLT 2010. https://arxiv.org/abs/1003.1630
- V. Perchet, P. Rigollet, The Multi-Armed Bandit Problem with Covariates, Annals of Statistics 41(2), 2013. https://arxiv.org/abs/1110.6084
- H. Bastani, M. Bayati, Online Decision Making with High-Dimensional Covariates, Operations Research 68(1), 2020. https://doi.org/10.1287/opre.2019.1902
- C. J. Stone, Optimal Global Rates of Convergence for Nonparametric Regression, Annals of Statistics 10(4), 1982. https://doi.org/10.1214/aos/1176345969