Display (5.15) — the differential discounted value equation along a discount optimal policy
ProvedArapostathisAC.SennottACOI.eq_5_15discounted-costoptimality-equationp2o-batch-pfp2bp2o-gran-per-chapterp2o-plan-paperp2o-v1vanishing-discount
In the countable-state controlled Markov process of §5, let , suppose that for every state , and let be -discount optimal. Write . Then for every state the series converges and
This is display (5.15) in the proof of Theorem 5.9 (the identity (5.6) of p. 301 evaluated at the discount optimal action). It is the relation whose limit as produces the average cost optimality inequality.
Formalization Note The paper writes (5.15) for and ; it is stated here for an arbitrary and an arbitrary -discount optimal , under the finiteness of (Assumption 5.14 at this ), which makes a genuine difference of real numbers. Convergence of the series is part of the conclusion.
Preamble
import Mathlib import Definitions.Def_ArapostathisAC_SennottACOI_CMP import Definitions.Def_ArapostathisAC_SennottACOI_Assumptions open MeasureTheory ProbabilityTheory Filter Topology open scoped ENNReal
Formal statement
namespace ArapostathisAC.SennottACOI
variable {A : Type*} [MetricSpace A] [MeasurableSpace A] [BorelSpace A]
/-- Display (5.15) (proof of Theorem 5.9, p. 308), the identity (5.6) along a discount optimal
stationary policy: if `β ∈ (0, 1)`, `J*_β` is finite everywhere and `f ∈ Π_SD` is
`β`-discount optimal, then for every state `i` the series `Σ_j P(j | i, f(i)) h_β(j)` converges and
`(1 − β) J*_β(0) + h_β(i) = c(i, f(i)) + β Σ_j P(j | i, f(i)) h_β(j)`. -/
theorem eq_5_15 (M : ArapostathisAC.VanishingDiscount.CMP A) (β : ℝ) (hβ : β ∈ Set.Ioo (0 : ℝ) 1)
(hfin : ∀ i, ArapostathisAC.VanishingDiscount.discValue M β i ≠ ⊤) (f : StationaryPolicy M) (hf : ArapostathisAC.VanishingDiscount.IsDiscOptimal M β f)
(i : ℕ) :
Summable (fun j => ArapostathisAC.VanishingDiscount.prob M i (f.1 i) j * ArapostathisAC.VanishingDiscount.hRel M β j) ∧
(1 - β) * (ArapostathisAC.VanishingDiscount.discValue M β 0).toReal + ArapostathisAC.VanishingDiscount.hRel M β i =
M.c i (f.1 i) + β * ∑' j, ArapostathisAC.VanishingDiscount.prob M i (f.1 i) j * ArapostathisAC.VanishingDiscount.hRel M β j := by sorry
end ArapostathisAC.SennottACOI
Source
Arapostathis, Borkar, Fernández-Gaucherand, Ghosh, Marcus, Discrete-time controlled Markov processes with average cost criterion: a survey, SIAM J. Control Optim. 31(2) (1993), p. 308, proof of Theorem 5.9, display (5.15); cf. (5.6), p. 301
Human review
Confirmed by the mission captain (proposal self-audit).
Confirmed by the moderator at approval.