Proposition 2, variance part — without mixing
ProvedTPG.untruncated_variance_boundSetting. is a finite state space and the action is control or treatment. Time is , which is the source's time . Under action at time the state moves according to a transition matrix , i.e. ; the matrices may depend on arbitrarily. The initial state has distribution , and the reward has distribution on , with for all and conditional independence from everything else given . Under the policy every action is treatment with probability , independently of everything else; , and refer to the law of the trajectory under .
Let be the untruncated policy gradient estimator, computed from a trajectory generated under .
Statement. For every , with no mixing assumption,
This is the variance half of Proposition 2: without truncation the variance bound grows linearly in the horizon.
import Mathlib import Definitions.Def_TPG_Model import Definitions.Def_TPG_Estimators open MeasureTheory ProbabilityTheory open scoped BigOperators ENNReal
namespace TPG
theorem untruncated_variance_bound {S : Type} [Fintype S] [MeasurableSpace S] [MeasurableSingletonClass S]
(E : Env S) (T : ℕ) (M : ℝ) (hT : 0 < T) (hM : E.RewardBound M) :
variance (tpgEst (1 / 2) T T) (E.expLaw T (1 / 2)) ≤ 2 * M ^ 2 * ((T : ℝ) + 1) := by sorry
end TPG