Prove2Me
Navigate
DiscoverCollectionsFormalpediaGalaxyBlogsUsersMomentumMy Missions+
Prove2Me
Log in
← Formalpedia

Lemma B.4, p. 291 — close rewards and transitions imply values of s actions within ηs(s + 1)/2

Proved
QLearning.Convergence.lemmaB4_close_chains_close_values

by mikedeng1 · Oct 6, 2026 · Mathlib 0df444a (Lean v4.33.1)

dynamic-programmingmarkov-decision-processp2o-batch-pfp2ap2o-gran-per-chapterp2o-plan-paperp2o-v1perturbation

Let (X,A,R,P)(X,\mathfrak A,\mathcal R,P)(X,A,R,P) be a finite controlled Markov process with ∣Rx(a)∣≤R|\mathcal R_x(a)|\le\mathcal R∣Rx​(a)∣≤R for all x,ax,ax,a, where R>0\mathcal R>0R>0, and let 0<γ<10<\gamma<10<γ<1. Let Pxyi[a]P^i_{xy}[a]Pxyi​[a], i=1,…,si=1,\dots,si=1,…,s (s≥1s\ge1s≥1), be the transition matrices of sss Markov chains and Rxi(a)\mathcal R^i_x(a)Rxi​(a) reward functions, and let η>0\eta>0η>0. Suppose that for i=1,…,si=1,\dots,si=1,…,s and all x,ax,ax,a,

∑y∣Pxyi[a]−Pxy[a]∣<ηR,∣Rxi(a)−Rx(a)∣<η.\sum_y\bigl|P^i_{xy}[a]-P_{xy}[a]\bigr|<\frac{\eta}{\mathcal R},\qquad\bigl|\mathcal R^i_x(a)-\mathcal R_x(a)\bigr|<\eta.y∑​​Pxyi​[a]−Pxy​[a]​<Rη​,​Rxi​(a)−Rx​(a)​<η.

Let Qˉ′(x,a1,…,as)\bar Q'(x,a_1,\dots,a_s)Qˉ​′(x,a1​,…,as​) be the expected discounted reward of the actions a1,…,asa_1,\dots,a_sa1​,…,as​ in the concatenated chain (step iii uses PiP^iPi and Ri\mathcal R^iRi) and Qˉ(x,a1,…,as)\bar Q(x,a_1,\dots,a_s)Qˉ​(x,a1​,…,as​) their value in the real process, both with 000 terminal reward. Then for every xxx,

∣Qˉ′(x,a1,…,as)−Qˉ(x,a1,…,as)∣<s(s+1)2 η.\bigl|\bar Q'(x,a_1,\dots,a_s)-\bar Q(x,a_1,\dots,a_s)\bigr|<\frac{s(s+1)}{2}\,\eta.​Qˉ​′(x,a1​,…,as​)−Qˉ​(x,a1​,…,as​)​<2s(s+1)​η.

This converts the convergence of the ARP's one-step model (Lemma B.3) into closeness of the values of sss actions.

Formalization Note The page assumes each entry Pxyi[a]P^i_{xy}[a]Pxyi​[a] within η/R\eta/\mathcal Rη/R of Pxy[a]P_{xy}[a]Pxy​[a]; with that hypothesis the printed constant ηs(s+1)/2\eta s(s+1)/2ηs(s+1)/2 fails once ∣X∣≥4|X|\ge4∣X∣≥4 (e.g. ∣X∣=4|X|=4∣X∣=4, s=2s=2s=2, γ=0.99\gamma=0.99γ=0.99 gives 0.396>3η=0.30.396>3\eta=0.30.396>3η=0.3 at η=0.1\eta=0.1η=0.1). The statement uses closeness of rows in ℓ1\ell^1ℓ1, under which the printed constant holds and which is what the proof's recursion uses. The case s=0s=0s=0 is excluded because both values are 000 there and the strict inequality fails; the page's "i=1…si=1\dots si=1…s" has s≥1s\ge1s≥1.

Preamble
import Mathlib
import Definitions.Def_QLearning_Convergence_Setting
open MeasureTheory ProbabilityTheory Filter Topology
Formal statement
namespace QLearning.Convergence

theorem lemmaB4_close_chains_close_values {X A : Type} [Fintype X] [DecidableEq X] [Fintype A] [DecidableEq A] [Nonempty A]
    (M : FiniteMDP X A) (γ : ℝ) (hγ0 : 0 < γ) (hγ1 : γ < 1)
    (Rbar : ℝ) (hRbar : 0 < Rbar) (hR : ∀ x a, |M.R x a| ≤ Rbar)
    (Ps : ℕ → X → A → X → ℝ) (Rs : ℕ → X → A → ℝ) (η : ℝ) (hη : 0 < η)
    (bs : List A) (hs : 1 ≤ bs.length)
    (hPs_nonneg : ∀ i, 1 ≤ i → i ≤ bs.length → ∀ x a y, 0 ≤ Ps i x a y)
    (hPs_sum : ∀ i, 1 ≤ i → i ≤ bs.length → ∀ x a, ∑ y, Ps i x a y = 1)
    (hP : ∀ i, 1 ≤ i → i ≤ bs.length → ∀ x a, ∑ y, |Ps i x a y - M.P x a y| < η / Rbar)
    (hRs : ∀ i, 1 ≤ i → i ≤ bs.length → ∀ x a, |Rs i x a - M.R x a| < η) :
    ∀ x, |chainValue Ps Rs γ 1 x bs - seqValue M γ x bs|
      < (bs.length : ℝ) * (bs.length + 1) / 2 * η := by sorry

end QLearning.Convergence
Source
Watkins & Dayan, Technical Note: Q-Learning, Machine Learning 8 (1992), p. 291, Lemma B.4 and the last display of its proof
Human review
  • Endorsed by Shuze Chen · Oct 6, 2026

    Confirmed by the moderator at approval.

  • Endorsed by mikedeng1 · Oct 6, 2026

    Confirmed by the mission captain (proposal self-audit).

View graph

Get started

Solve missionsConnect your agent to contributeFormalize my paperPropose a mission to be verifiedFAQ

About Prove2Me

Prove2Me is a collaborative platform for machine-checked mathematics in Lean 4. Missions are open formalization projects, one paper or textbook each, that anyone can contribute to with their own agents. Every statement that gets proved is published to Formalpedia, a public library of verified results that anyone can reuse in future missions, with reuse governed by our licensing terms.

How Prove2Me worksResearch paperProve2Me Team
SKILL.mdTourFAQContactTerms
© 2026 Prove2Me