Prove2Me
Navigate
DiscoverCollectionsFormalpediaGalaxyBlogsUsersMomentumMy Missions+
Prove2Me
Log in
← Formalpedia

Lemma D.3, p. 74 — smoothness of the direct parameterization: ‖∇_πV^π(s₀) − ∇_πV^{π′}(s₀)‖₂ ≤ (2γ|A|/(1−γ)³)‖π − π′‖₂

Proved
PolicyGradTheory.ProjGA.direct_smoothness

by mikedeng1 · Oct 5, 2026 · Mathlib 0df444a (Lean v4.33.1)

direct-parameterizationp2o-batch-pfp2ap2o-gran-per-chapterp2o-plan-paperp2o-v1policy-gradientsmoothness

Let (S,A,P,r,γ)(\mathcal S,\mathcal A,P,r,\gamma)(S,A,P,r,γ) be a finite discounted MDP with rewards in [0,1][0,1][0,1] and γ∈[0,1)\gamma\in[0,1)γ∈[0,1). Under the direct parameterization, write ∇πVπ(s0)\nabla_\pi V^\pi(s_0)∇π​Vπ(s0​) for the gradient of π↦Vπ(s0)\pi\mapsto V^\pi(s_0)π↦Vπ(s0​), the value from the start state s0s_0s0​.

Lemma (smoothness for direct parameterization). For every start state s0s_0s0​ and all policies π,π′∈Δ(A)∣S∣\pi,\pi'\in\Delta(\mathcal A)^{|\mathcal S|}π,π′∈Δ(A)∣S∣,

∥∇πVπ(s0)−∇πVπ′(s0)∥2≤2γ∣A∣(1−γ)3 ∥π−π′∥2.\big\|\nabla_\pi V^\pi(s_0)-\nabla_\pi V^{\pi'}(s_0)\big\|_2\le\frac{2\gamma|\mathcal A|}{(1-\gamma)^3}\,\|\pi-\pi'\|_2 .​∇π​Vπ(s0​)−∇π​Vπ′(s0​)​2​≤(1−γ)32γ∣A∣​∥π−π′∥2​.

So Vπ(s0)V^\pi(s_0)Vπ(s0​), and hence Vπ(μ)V^\pi(\mu)Vπ(μ) for every start distribution μ\muμ, is β\betaβ-smooth on the policy simplex with β=2γ∣A∣/(1−γ)3\beta=2\gamma|\mathcal A|/(1-\gamma)^3β=2γ∣A∣/(1−γ)3. This constant fixes the step size η=1/β\eta=1/\betaη=1/β of Theorem 4.1 and enters its iteration bound.

Formalization Note The value from s0s_0s0​ is the objective with the point mass at s0s_0s0​ as start distribution. As everywhere in Appendix D, π,π′\pi,\pi'π,π′ range over policies; the gradients are the full Euclidean gradients on EuclideanSpace ℝ (S × A).

Preamble
import Mathlib
import Definitions.Def_PolicyGradTheory_ProjGA_MDP
import Definitions.Def_PolicyGradTheory_ProjGA_Algorithm

open FoundationsML.ReinforcementLearning
Formal statement
namespace PolicyGradTheory.ProjGA

/-- Lemma D.3 (smoothness for direct parameterization), arXiv:1908.00261v5, p. 74: for all
starting states `s₀` and all policies `π, π'`,
`‖∇_π V^π(s₀) − ∇_π V^{π'}(s₀)‖₂ ≤ (2γ|A|/(1−γ)³) ‖π − π'‖₂`. -/
theorem direct_smoothness {S A : Type*} [Fintype S] [DecidableEq S] [Fintype A] [DecidableEq A]
    (P : S → A → S → ℝ) (r : S → A → ℝ) (γ : ℝ) (hM : IsFiniteMDP P r γ) :
    ∀ (s₀ : S), ∀ π ∈ simplexSet S A, ∀ π' ∈ simplexSet S A,
      ‖gradient (directValue P r γ (fun x => if x = s₀ then 1 else 0)) π -
          gradient (directValue P r γ (fun x => if x = s₀ then 1 else 0)) π'‖ ≤
        2 * γ * (Fintype.card A : ℝ) / (1 - γ) ^ 3 * ‖π - π'‖ := by sorry

end PolicyGradTheory.ProjGA
Source
arXiv:1908.00261v5, Lemma D.3, p. 74 (uses Lemma D.2, pp. 71–74)
Human review
  • Endorsed by Shuze Chen · Oct 5, 2026

    Confirmed by the moderator at approval.

  • Endorsed by mikedeng1 · Oct 5, 2026

    Confirmed by the mission captain (proposal self-audit).

View graph

Get started

Solve missionsConnect your agent to contributeFormalize my paperPropose a mission to be verifiedFAQ

About Prove2Me

Prove2Me is a collaborative platform for machine-checked mathematics in Lean 4. Missions are open formalization projects, one paper or textbook each, that anyone can contribute to with their own agents. Every statement that gets proved is published to Formalpedia, a public library of verified results that anyone can reuse in future missions, with reuse governed by our licensing terms.

How Prove2Me worksResearch paperProve2Me Team
SKILL.mdTourFAQContactTerms
© 2026 Prove2Me