Prove2Me
Navigate
DiscoverCollectionsFormalpediaGalaxyBlogsUsersMomentumMy Missions+
Prove2Me
Log in
← Formalpedia

Theorem 4 — no improvement of order n+1 implies n^± discount optimality, which rules out improvement of order n

Proved
VeinottSensitiveDP.Sensitive.theorem_4

by mikedeng1 · Oct 6, 2026 · Mathlib 0df444a (Lean v4.33.1)

discount-optimalitymarkov-decision-processp2o-batch-pfp2ap2o-gran-per-chapterp2o-plan-paperp2o-v1policy-improvement

In the model of §4, where every P(f)P(f)P(f) is substochastic (and, for the sign −-−, every stationary policy is transient), let SSS be the number of states, f∈Ff\in Ff∈F and n∈{−2,−1,…,S−1}n\in\{-2,-1,\dots,S-1\}n∈{−2,−1,…,S−1}.

  1. If Gn+1±(f)G_{n+1}^\pm(f)Gn+1±​(f) is empty, then f∈Dn±f\in D_n^\pmf∈Dn±​.
  2. If f∈Dn±f\in D_n^\pmf∈Dn±​, then Gn±(f)G_n^\pm(f)Gn±​(f) is empty.

Here Dn±D_n^\pmDn±​ is the set of decision rules fff whose stationary policy f∞f^\inftyf∞ satisfies lim inf⁡ρ→0±∣ρ∣−n[Vρ(f∞)−Vρ(π)]≥0\liminf_{\rho\to0\pm}|\rho|^{-n}[V_\rho(f^\infty)-V_\rho(\pi)]\ge0liminfρ→0±​∣ρ∣−n[Vρ​(f∞)−Vρ​(π)]≥0 for all policies π\piπ (D−2=FD_{-2}=FD−2​=F), and Gn±(f)={g:Ψn±(g,f)≻0}G_n^\pm(f)=\{g:\Psi_n^\pm(g,f)\succ0\}Gn±​(f)={g:Ψn±​(g,f)≻0} is the set of improvements of fff of order nnn.

The theorem gives a test for n±n^\pmn± discount optimality that only inspects the finitely many one-step switches ggg, and it lets the policy improvement method stop as soon as no improvement of order n+1n+1n+1 is left.

Formalization Note The two signs ±\pm± are one real parameter σ∈{1,−1}\sigma\in\{1,-1\}σ∈{1,−1}: σ=1\sigma=1σ=1 is +++ and σ=−1\sigma=-1σ=−1 is −-−. The liminf is encoded per state as "for every ε>0\varepsilon>0ε>0, eventually ≥−ε\ge-\varepsilon≥−ε"; the lexicographic order is applied row by row.

Preamble
import Mathlib
import Definitions.Def_BlackwellDiscreteDP_NearOne_LimitMatrix
import Definitions.Def_VeinottSensitiveDP_Sensitive_Matrix
import Definitions.Def_VeinottSensitiveDP_Sensitive_Model
import Definitions.Def_VeinottSensitiveDP_Sensitive_Optimality
open Filter Topology Matrix BlackwellDiscreteDP.NearOne
Formal statement
namespace VeinottSensitiveDP.Sensitive

/-- Theorem 4: suppose `f ε F` and `n = −2, −1, ⋯, S − 1`.
1°. If `G_{n+1}^±(f)` is empty, then `f ε D_n^±`.
2°. If `f ε D_n^±`, then `G_n^±(f)` is empty.

Veinott, Discrete Dynamic Programming with Sensitive Discount Optimality Criteria,
Ann. Math. Statist. 40(5):1635–1660 (1969), DOI 10.1214/aoms/1177697379, p. 1648, §4, Theorem 4.

**Formalization Note.** `σ = 1` is the sign `+` and `σ = −1` the sign `−`; the `−` case assumes the transient case, where alone `n⁻`
discount optimality is defined (p. 1644). `S = Fintype.card St`. `D_n^±` is the set of `f` whose
stationary policy `f^∞` is `n^±` discount optimal against **all** policies (27), with `D_{−2} = F`;
`G_n^±(f) = {g : Ψ_n^±(g, f) ≻ 0}` (lexicographic, columns `−1, …, n`), empty for `n < −1`. -/
theorem theorem_4 {St : Type} [Fintype St] [DecidableEq St] [Nonempty St]
    {A : St → Type} [∀ s, Fintype (A s)] [∀ s, DecidableEq (A s)] [∀ s, Nonempty (A s)]
    (M : Model St A)
    (σ : ℝ) (hσ : σ = 1 ∨ σ = -1) (htr : σ = -1 → M.TransientCase) (n : ℤ) (hn1 : -2 ≤ n)
    (hn2 : n ≤ (Fintype.card St : ℤ) - 1) (f : DecisionRule A) :
    (M.G σ (n + 1) f = ∅ → f ∈ M.D σ n) ∧ (f ∈ M.D σ n → M.G σ n f = ∅) := by sorry

end VeinottSensitiveDP.Sensitive
Source
Veinott, Discrete Dynamic Programming with Sensitive Discount Optimality Criteria, Ann. Math. Statist. 40(5):1635–1660 (1969), DOI 10.1214/aoms/1177697379, p. 1648, §4, Theorem 4
Human review
  • Endorsed by Shuze Chen · Oct 6, 2026

    Confirmed by the moderator at approval.

  • Endorsed by mikedeng1 · Oct 6, 2026

    Confirmed by the mission captain (proposal self-audit).

View graph

Get started

Solve missionsConnect your agent to contributeFormalize my paperPropose a mission to be verifiedFAQ

About Prove2Me

Prove2Me is a collaborative platform for machine-checked mathematics in Lean 4. Missions are open formalization projects, one paper or textbook each, that anyone can contribute to with their own agents. Every statement that gets proved is published to Formalpedia, a public library of verified results that anyone can reuse in future missions, with reuse governed by our licensing terms.

How Prove2Me worksResearch paperProve2Me Team
SKILL.mdTourFAQContactTerms
© 2026 Prove2Me