Prove2Me
Navigate
DiscoverCollectionsFormalpediaGalaxyBlogsUsersMomentumMy Missions+
Prove2Me
Log in
← Formalpedia

App. B, p. 10 — Σ_k Σ_i 1(N_{t_k}(s,a) ≤ τ) ≤ 2τSA

Proved
PosteriorSamplingRL.Regret.appB_small_count_bound

by mikedeng1 · Oct 9, 2026 · Mathlib 0df444a (Lean v4.33.1)

countingp2o-batch-pfp2ap2o-gran-per-chapterp2o-plan-paperp2o-v1pigeonholereinforcement-learning

Let (sk,i,ak,i)(s_{k,i},a_{k,i})(sk,i​,ak,i​), k=1,2,…k=1,2,\dotsk=1,2,…, i=1,…,τi=1,\dots,\taui=1,…,τ, be any sequence of state–action pairs in S×A\mathcal S\times\mathcal AS×A with ∣S∣=S|\mathcal S|=S∣S∣=S, ∣A∣=A|\mathcal A|=A∣A∣=A, and let Ntk(s,a)N_{t_k}(s,a)Ntk​​(s,a) be the number of occurrences of (s,a)(s,a)(s,a) in the episodes before kkk. Then for every mmm,

∑k=1m∑i=1τ1(Ntk(sk,i,ak,i)≤τ)≤2τSA.\sum_{k=1}^m\sum_{i=1}^\tau\mathbf 1\big(N_{t_k}(s_{k,i},a_{k,i})\le\tau\big)\le2\tau SA .k=1∑m​i=1∑τ​1(Ntk​​(sk,i​,ak,i​)≤τ)≤2τSA.

Each pair can be visited in an episode that starts with at most τ\tauτ earlier visits at most 2τ2\tau2τ times in total. The bound controls the part of the width sum in which the counts are still small.

Formalization Note The page says "fewer than 2τ times", but the count can equal 2τ2\tau2τ (one pair visited τ\tauτ times in each of two episodes); the display's ≤2τSA\le2\tau SA≤2τSA is what is stated. The statement is deterministic: it holds for every visit sequence, in particular for the pairs visited by a PSRL run.

Preamble
import Mathlib
import Definitions.Def_PosteriorSamplingRL_Regret_Confidence
Formal statement
namespace PosteriorSamplingRL.Regret

/-- App. B (arXiv:1306.0940v5, p. 10), first count: for any sequence of state–action pairs
`(s_{k,j}, a_{k,j})` (episode `k`, step `j`), with `N_{t_k}(s, a)` the number of visits before
episode `k`, `∑_{k=1}^m ∑_{i=1}^τ 1(N_{t_k}(s_{k,i}, a_{k,i}) ≤ τ) ≤ 2τSA`. -/
theorem appB_small_count_bound (S A τ : ℕ) (sa : ℕ → Fin τ → Fin S × Fin A) (m : ℕ) :
    ∑ k ∈ Finset.range m, ∑ j : Fin τ,
        (if visitCount sa k (sa k j) ≤ τ then (1 : ℝ) else 0)
      ≤ 2 * (τ : ℝ) * S * A := by sorry

end PosteriorSamplingRL.Regret
Source
arXiv:1306.0940v5, App. B, p. 10, first count
Human review
  • Endorsed by Shuze Chen · Oct 10, 2026

    Confirmed by the moderator at approval.

  • Endorsed by mikedeng1 · Oct 10, 2026

    Confirmed by the mission captain (proposal self-audit).

View graph

Get started

Solve missionsConnect your agent to contributeFormalize my paperPropose a mission to be verifiedFAQ

About Prove2Me

Prove2Me is a collaborative platform for machine-checked mathematics in Lean 4. Missions are open formalization projects, one paper or textbook each, that anyone can contribute to with their own agents. Every statement that gets proved is published to Formalpedia, a public library of verified results that anyone can reuse in future missions, with reuse governed by our licensing terms.

How Prove2Me worksResearch paperProve2Me Team
SKILL.mdTourFAQContactTerms
© 2026 Prove2Me