Markov property of the trajectory law across a gap of steps
ProvedTPG.integral_head_mul_gap_tailSetting. is a finite state space, actions are , time is (the source's time ). The environment has transition matrices , initial distribution and reward laws on . For a sequence of treatment probabilities, and denote the law of the trajectory when , the actions are independent with , , and the rewards are conditionally independent with (Env.trajLaw T q); the Bernoulli policy is .
Consider the horizon . The head is the first time steps ; after a gap of steps, the tail is the last steps , re-indexed from . Let with (the identity when ), and for a state let denote expectation for the process restarted at time from .
Lemma. Let for all , and let (a function of a trajectory of length ) and (a function of a trajectory of length ) be bounded and measurable. Then
The head influences the tail only through the distribution of the state at time , propagated by the intermediate kernels. Combined with the contraction of Assumption 1 this yields geometric decay of covariances between functions of time windows separated by steps, which is how the source obtains its variance bound and the strong mixing property.
import Mathlib import Definitions.Def_TPG_Model import Definitions.Def_TPG_Estimators import Definitions.Def_TPG_Markov import Definitions.Def_TPG_Bridge open MeasureTheory ProbabilityTheory open scoped BigOperators ENNReal
namespace TPG
theorem integral_head_mul_gap_tail {S : Type} [Fintype S] [DecidableEq S] [MeasurableSpace S] [MeasurableSingletonClass S]
(E : Env S) (m h n : ℕ) (q : ℕ → ℝ) (hq : ∀ t : ℕ, 0 ≤ q t ∧ q t ≤ 1)
(F : Traj S (m + 1) → ℝ) (G : Traj S n → ℝ) (hF : Measurable F) (hG : Measurable G)
(CF CG : ℝ) (hFb : ∀ ω, |F ω| ≤ CF) (hGb : ∀ ω, |G ω| ≤ CG) :
∫ ω, F (headTraj (m + 1) (h + n) ω) * G (tailTraj h n (tailTraj (m + 1) (h + n) ω))
∂(E.trajLaw (m + 1 + (h + n)) q)
= ∑ a : S, ∑ b : S,
(∫ ω, F ω * E.P m (Z (Fin.last m) ω) (X (Fin.last m) ω) a ∂(E.trajLaw (m + 1) q))
* (((List.Ico (m + 1) (m + 1 + h)).map (fun s => E.PMix (q s) s)).prod a b)
* ∫ ω, G ω ∂((E.shift (m + 1 + h) b).trajLaw n (fun t => q (m + 1 + h + t))) := by sorry
end TPG