Optimistic Posterior Sampling for Reinforcement Learning: Worst-Case Regret Bounds II: Along Any h, the Empirical Transition Vector Is within 2√(log(n/ρ)Σγᵢcᵢ²/n) + 3D log(2/ρ)/n of the TruthResearch Paper
Motivation
Model-based reinforcement learning algorithms for Markov decision processes estimate each transition distribution from the transitions they have observed out of the state–action pair . Their regret analyses then need to know how far the empirical distribution can be from the true one . The standard tool is an bound on , as in UCRL2 (Jaksch, Ortner and Auer, 2010), which costs a factor of order in the number of states . What the analysis actually uses, however, is the deviation along one direction: the projection onto a bias (or value) vector whose entries differ by at most the MDP's diameter .
S. Agrawal and R. Jia (arXiv:1705.07041v3, published in Mathematics of Operations Research, 2023, as Optimistic Posterior Sampling for Reinforcement Learning: Worst-Case Regret Bounds) prove a worst-case regret bound of order for an optimistic variant of posterior sampling. One ingredient of that analysis is their Proposition 5.2: a Bernstein-type bound on whose leading term is governed by a variance of under , written in stick-breaking form. This mission formalizes that proposition and the steps of its proof.
Setting
Fix a number of states and order them . A probability vector has and . A multinoulli trial with parameter is a random state with . Draw independent trials and let
be the empirical probability vector, the average of the one-hot outcomes. For and a vector define the stick-breaking quantities
Thus is the mean of under conditioned on the states after , and compares with it. The sum equals the variance of for . In Lean these objects are gammaCoef p i, Hnext p h i, cCoef p h i, phat ω i, the law of one trial is cat p and the law of trials is trialsLaw n p, all in the namespace OptimisticPSRL.Concentration.
Formalization targets
Goal: Proposition 5.2 (p. 9)
Let satisfy for all , let and . Then, with probability at least ,
Milestones
- Fact 1 (p. 17), a deterministic identity: for probability vectors with stick-breaking ratios and ,
- The projection as a sum of martingale differences (proof, p. 21): with and ,
together with the two intermediate expressions of the page. 3. Uncorrelated coordinates (proof, p. 21): for . 4. The predictable variance (proof, p. 21): . 5. Corollary B.1 (p. 26), Bernstein's inequality for bounded martingale differences with random predictable variance : for , , with probability greater than ,
Significance
The result. Proposition 5.2 is the concentration half of the optimism step of the paper (Lemma 5.3): combined with an anti-concentration bound for Dirichlet samples along the same direction (Proposition 5.1), it shows that some posterior sample is optimistic along with constant probability. Because the bound is stated along a single direction and is weighted by the variance of rather than by alone, it avoids the loss that an bound would incur, which is part of how the paper's regret bound improves on UCRL2's . The stick-breaking form of the variance is what lets the same quantity appear on both sides of the optimism argument, in the Dirichlet bound and in the empirical bound.
Formalizing it. The proposition and its proof steps are proved on paper; none of them has a machine-checked proof. The regret theorem of the paper (Theorem 1) is not posed: its constants are hidden in and its algorithm's parameters are given as . Proposition 5.2, by contrast, is fully explicit. Corollary B.1 is a reusable martingale inequality with explicit constants. The predictable-variance identity and Fact 1 are exact identities about stick-breaking coordinates that recur in analyses of Dirichlet and categorical models.
Difficulty
Two steps resist the obvious approach. First, the statement carries the variance in stick-breaking form, not as ; linking the two needs the decomposition of Fact 1 and the identity for the predictable variance, which are exact algebraic identities whose bookkeeping of tails and empty tails is error-prone. Second, the paper's own route goes through Corollary B.1, a martingale Bernstein inequality with a random variance, which rests on a peeling argument over a grid of variance levels (Seldin et al., 2012); Mathlib has neither this inequality nor a Bernstein inequality with explicit constants for bounded variables.
A known subtlety: the paper's proof, through Corollary B.1, yields the second term , while the statement prints the smaller . The printed statement is still true for iid trials (the classical Bernstein inequality gives a bound below it), so the goal is posed as printed. The paper's own proof therefore does not establish the printed constant.
Formalization scope
- States are
Fin Swith its natural order, 0-based; the paper's range is every state but the last (belowLast S). Fact 1 holds for every ordering; the natural order covers this since states are labels. - Probability vectors are
p : Fin S → ℝwithIsProbVec p(nonnegative, summing to one). Zero entries are allowed. A ratio whose denominator is zero evaluates to (Lean's convention), which gives the values the identities need; no positivity of is assumed. - One trial has law
cat p; trials have the product lawtrialsLaw n pMeasure.pi. The empirical vectorphat ωis computed from the sampleω : Fin n → Fin S. "With probability " isENNReal.ofReal (1 - ρ) ≤ trialsLaw n p {good event}; Corollary B.1's "greater than " is strict. isReal.log. - The proof displays (milestones 2–4) are stated in the iid reading of the proposition: the paper writes them for transitions of an MDP out of a fixed , with an indicator and a random count ; here the indicator is one and , and the conditional expectations are expectations under one draw of law . Milestone 2 assumes every drawn state has positive probability (true almost surely), which the page's intermediate sum over needs.
- Corollary B.1 is posed for a general filtered probability space, with -measurable, almost surely and .
The goal cannot be satisfied by a degenerate reading: is built from the sample, the measure is the -fold product of the categorical law , the event is the good event itself (not a complement bound), and the variance term is not identically zero (for , it equals ).
Needed infrastructure: finite-sum manipulations over tails of Fin S; a Bernstein or Freedman inequality with explicit constants (for iid bounded variables for the goal, for martingale differences for Corollary B.1). Contributions of either inequality are reusable well beyond this mission and are welcome, as are alternative proofs of the goal that bypass Corollary B.1.
Selected references
- S. Agrawal and R. Jia, Posterior sampling for reinforcement learning: worst-case regret bounds, arXiv:1705.07041v3, 2020; published as Optimistic Posterior Sampling for Reinforcement Learning: Worst-Case Regret Bounds, Mathematics of Operations Research, 2023. https://arxiv.org/abs/1705.07041
- T. Jaksch, R. Ortner and P. Auer, Near-optimal regret bounds for reinforcement learning, Journal of Machine Learning Research 11, 2010. https://www.jmlr.org/papers/v11/jaksch10a.html
- Y. Seldin, F. Laviolette, N. Cesa-Bianchi, J. Shawe-Taylor and P. Auer, PAC-Bayesian inequalities for martingales, IEEE Transactions on Information Theory 58(12), 2012. https://doi.org/10.1109/TIT.2012.2211334