§9, p. 24 — existence of the positive transition distribution
ProvedTDApprox.Sampling.exists_pcounterexamplep2o-batch-pfp2ap2o-gran-per-chapterp2o-plan-paperp2o-v1probabilitytemporal-difference
Fix two distinct states in a countable state space and a discount factor . There is a probability distribution assigning positive mass to every state such that
This supplies the transition distribution used in the paper's divergence example, including when the state space is countably infinite.
Formalization Note The state represents state 2 in the paper's indexing. The probability distribution is a measure with positive singleton masses.
Preamble
import Mathlib import Definitions.Def_TDApprox_Sampling_Model
Formal statement
namespace TDApprox.Sampling
open MeasureTheory
theorem exists_p
{S : Type*} [Countable S] [MeasurableSpace S] [DiscreteMeasurableSpace S]
(s₁ s₂ : S) (hne : s₁ ≠ s₂) (α : ℝ)
(hα₁ : (5 : ℝ) / 6 < α) (hα₂ : α < 1) :
∃ p : Measure S, ∃ _ : IsProbabilityMeasure p,
(∀ i, 0 < p {i}) ∧
5 / (6 * α) < (p {s₂}).toReal ∧
(p {s₂}).toReal < 1 := by sorry
end TDApprox.Sampling
Source
Tsitsiklis & Van Roy, LIDS-P-2322 (1996), Theorem 3 proof, p. 24; https://dspace.mit.edu/entities/publication/ab395d25-a6d3-407a-9589-60eaac58fd05
Human review
Confirmed by the mission captain (proposal self-audit).
Confirmed by the moderator at approval.