Integration against the trajectory law:
ProvedTPG.integral_trajLawcausal-inferencemarkov-chainnonstationaritypolicy-gradienttreatment-effect
Setting. is a finite state space, actions are , time is (the source's time ). The environment has transition matrices , initial distribution and reward laws on . For a sequence of treatment probabilities, and denote the law of the trajectory when , the actions are independent with , , and the rewards are conditionally independent with (Env.trajLaw T q); the Bernoulli policy is .
Lemma. Let for all , let be a measurable real function of the trajectory, and suppose that for every state path and action path the function is integrable with respect to . Then is integrable under and
with the path weight
Every expectation in the mission is computed through this formula.
Preamble
import Mathlib import Definitions.Def_TPG_Model import Definitions.Def_TPG_Markov open MeasureTheory ProbabilityTheory open scoped BigOperators ENNReal
Formal statement
namespace TPG
theorem integral_trajLaw {S : Type} [Fintype S] [MeasurableSpace S] [MeasurableSingletonClass S]
(E : Env S) (T : ℕ) (q : ℕ → ℝ) (hq : ∀ t : ℕ, 0 ≤ q t ∧ q t ≤ 1)
(F : Traj S T → ℝ) (hF : Measurable F)
(hint : ∀ p : (Fin T → S) × (Fin T → Bool),
Integrable (fun y : Fin T → ℝ => F (embedPath p.1 p.2 y)) (E.rewardLaw p.1 p.2)) :
Integrable F (E.trajLaw T q)
∧ ∫ ω, F ω ∂(E.trajLaw T q)
= ∑ p : (Fin T → S) × (Fin T → Bool),
E.pathWeight T q p.1 p.2 * ∫ y, F (embedPath p.1 p.2 y) ∂(E.rewardLaw p.1 p.2) := by sorry
end TPGSource
Johari, Peng, Xing, Estimation of Treatment Effects Under Nonstationarity via the Truncated Policy Gradient Estimator, revised manuscript of October 2026 (revision of arXiv:2506.05308v3, https://arxiv.org/abs/2506.05308); numbering follows the revised manuscript. Auxiliary statement, not a numbered result of the paper: it is the expectation formula implied by the model of §2.2, pp. 8–9.