Lemma 5 — the LS TD limit under visit frequencies
ProvedLeastSquaresTD.Absorbing.lemma5_lstd_limitRun the LS TD estimate from Equation (11) on an arbitrary finite Markov chain. Let be the limiting proportion of transitions departing state . Assume almost surely that every state is visited infinitely often and those departure proportions converge to . If is invertible, then
This identifies the large-sample least-squares estimate with the matrix expression used in Theorem 1.
Formalization Note The paper writes for the limit; convergence is asserted directly here. The ordinary Markov path is specified by its initial distribution and finite-dimensional transition law. All observed transitions enter this estimator, as Equation (11) requires. Small-sample singular matrices use Lean's total inverse; the asserted limit includes eventual invertibility.
import Definitions.Def_LeastSquaresTD_Absorbing_Estimator
namespace LeastSquaresTD.Absorbing
open MeasureTheory Filter Topology
variable {m : ℕ}
/-- Lemma 5, p. 43, for any finite Markov chain. The paper's `θ_LSTD` is
the limit of the estimates; `Tendsto` asserts that this limit exists.
Equation (11) uses every transition of this ordinary chain. -/
theorem lemma5_lstd_limit
{X Ω : Type*} [Fintype X] [DecidableEq X] [Nonempty X]
[MeasurableSpace X] [MeasurableSingletonClass X] [MeasurableSpace Ω]
(μ : Measure Ω) [IsProbabilityMeasure μ]
(C : Chain X) (S : X → ℝ) (R : X → X → ℝ)
(φ : X → Fin m → ℝ) (γ : ℝ) (π : X → ℝ) (Z : ℕ → Ω → X)
(hS_nonneg : ∀ x, 0 ≤ S x) (hS_sum : ∑ x, S x = 1)
(hlaw : HasChainLaw C S μ Z)
(hvisits : ∀ᵐ ω ∂μ, ∀ (x : X) (N : ℕ), ∃ n ≥ N, Z n ω = x)
(hfreq : ∀ᵐ ω ∂μ, ∀ x,
Tendsto (fun n => chainOutFrequency (fun k => Z k ω) x n) atTop (𝓝 (π x)))
(hunit : IsUnit (limitMatrix C φ γ π)) :
∀ᵐ ω ∂μ, Tendsto
(fun n => chainLstdTheta R φ γ (fun k => Z k ω) n)
atTop (𝓝 (Matrix.mulVec (limitMatrix C φ γ π)⁻¹
(limitVector C R φ π))) := by sorry
end LeastSquaresTD.Absorbing
Confirmed by the mission captain (proposal self-audit).
Confirmed by the moderator at approval.