Derivative of the expected reward along a partially varying randomization sequence
ProvedTPG.hasDerivAt_meanRewardcausal-inferencemarkov-chainnonstationaritypolicy-gradienttreatment-effect
Let be the expected reward at time under a sequence of treatment probabilities, with and (time is -indexed).
Fix a set of times and a sequence , and for a real number let be the sequence equal to at the times in and to at the other times.
Lemma. The function is differentiable at every , with
where is with the entry at time replaced by .
Since is affine in each coordinate, its derivative along a diagonal direction is the sum of the one-coordinate differences. With this is the policy gradient of the value function; with it is the truncated policy gradient.
Preamble
import Mathlib import Definitions.Def_TPG_Model import Definitions.Def_TPG_Estimators import Definitions.Def_TPG_Markov import Definitions.Def_TPG_Bridge open MeasureTheory ProbabilityTheory open scoped BigOperators ENNReal
Formal statement
namespace TPG
theorem hasDerivAt_meanReward {S : Type} [Fintype S] [DecidableEq S]
(E : Env S) (t : ℕ) (A : ℕ → Prop) [DecidablePred A] (c : ℕ → ℝ) (θ₀ : ℝ) :
HasDerivAt (fun θ : ℝ => E.meanReward (fun s => if A s then θ else c s) t)
(∑ u ∈ Finset.range (t + 1),
if A u then
(E.meanReward (Function.update (fun s => if A s then θ₀ else c s) u 1) t
- E.meanReward (Function.update (fun s => if A s then θ₀ else c s) u 0) t)
else 0)
θ₀ := by sorry
end TPGSource
Johari, Peng, Xing, Estimation of Treatment Effects Under Nonstationarity via the Truncated Policy Gradient Estimator, revised manuscript of October 2026 (revision of arXiv:2506.05308v3, https://arxiv.org/abs/2506.05308); numbering follows the revised manuscript. Auxiliary statement, not a numbered result of the paper: it is the product rule for d/dθ ρ^⊤ P_θ^{a→b} r_θ of Appendix A.3, p. 27, as used in Appendices G.1, G.3 and G.4, pp. 42–46.