Conditioning the trajectory law on one action:
ProvedTPG.integral_cond_ZSetting. is a finite state space, actions are , time is (the source's time ). The environment has transition matrices , initial distribution and reward laws on . For a sequence of treatment probabilities, and denote the law of the trajectory when , the actions are independent with , , and the rewards are conditionally independent with (Env.trajLaw T q); the Bernoulli policy is .
Lemma. Let for all , let with , let , and let be a measurable function of the trajectory that is integrable along every state–action path. Then the expectation of under the conditional law given is the expectation under the law in which the action at time is with probability one:
where is with replaced by .
With this turns the -values of the source into expected rewards under modified randomization sequences.
import Mathlib import Definitions.Def_TPG_Model import Definitions.Def_TPG_Estimators import Definitions.Def_TPG_Markov import Definitions.Def_TPG_Bridge open MeasureTheory ProbabilityTheory open scoped BigOperators ENNReal
namespace TPG
theorem integral_cond_Z {S : Type} [Fintype S] [MeasurableSpace S] [MeasurableSingletonClass S]
(E : Env S) (T : ℕ) (q : ℕ → ℝ) (hq : ∀ t : ℕ, 0 ≤ q t ∧ q t ≤ 1)
(u : Fin T) (z : Bool) (hu : 0 < q (u : ℕ) ∧ q (u : ℕ) < 1)
(F : Traj S T → ℝ) (hF : Measurable F)
(hint : ∀ p : (Fin T → S) × (Fin T → Bool),
Integrable (fun y : Fin T → ℝ => F (embedPath p.1 p.2 y)) (E.rewardLaw p.1 p.2)) :
∫ ω, F ω ∂(ProbabilityTheory.cond (E.trajLaw T q) {ω | Z u ω = z})
= ∫ ω, F ω ∂(E.trajLaw T (Function.update q (u : ℕ) (if z then 1 else 0))) := by sorry
end TPG