Proposition 1 — the policy gradient is the average -value difference
ProvedTPG.proposition_1_policy_gradientSetting. is a finite state space and the action is control or treatment. Time is , which is the source's time . Under action at time the state moves according to a transition matrix , i.e. ; the matrices may depend on arbitrarily. The initial state has distribution , and the reward has distribution on , with for all and conditional independence from everything else given . Under the policy every action is treatment with probability , independently of everything else; , and refer to the law of the trajectory under .
The value function is , and the -value of action at time under the experimental policy is the expected cumulative reward from time to the end of the horizon given that action,
Proposition. For every , the function is differentiable at and
This identifies the derivative of the policy value at the balanced design with a quantity that the untruncated estimator estimates without bias, and is the basis of the policy-gradient interpretation of the TPG estimator.
Formalization Note The -values do not depend on the state; they are expectations under the conditional law given the event , which has probability .
import Mathlib import Definitions.Def_TPG_Model import Definitions.Def_TPG_Estimators open MeasureTheory ProbabilityTheory open scoped BigOperators ENNReal
namespace TPG
theorem proposition_1_policy_gradient {S : Type} [Fintype S] [MeasurableSpace S] [MeasurableSingletonClass S]
(E : Env S) (T : ℕ) (M : ℝ) (hT : 0 < T) (hM : E.RewardBound M) :
HasDerivAt (fun θ : ℝ => E.J T θ)
((1 / (T : ℝ)) * ∑ t : Fin T, (E.Q (1 / 2) T t true - E.Q (1 / 2) T t false))
(1 / 2) := by sorry
end TPGRead-back
What the Lean code literally says, in plain math · claude-opus-5-5
What is quantified. The statement holds for every choice of the following data.
-
A finite set , carrying an arbitrary -algebra in which every singleton is measurable (as is finite, every subset of is then measurable).
-
An "environment" on , meaning the following package:
- for every natural number (all of them, not only those below the horizon) and every Boolean value , a real matrix whose entries are all and whose rows each sum to ;
- a function with for all and ;
- for every and every Boolean , a Borel probability measure on .
Because sums to , such an can only exist when is nonempty.
-
A natural number , with the hypothesis .
-
A real number (no sign condition), with the hypothesis that for every and every Boolean . The number occurs nowhere else in the statement. If the interval is empty and this hypothesis cannot be satisfied.
The sample space. Let be the set of all sequences with each , with the product -algebra (the given one on , the discrete one on the Booleans, the Borel one on ). Write , and for the three components of . Times run over .
The measure . Fix a real number (any real number, not only one in ). For a sequence and a Boolean sequence define the real number
(The middle product is empty, equal to , when . The matrix governs the passage from index to index ; matrices with are never used.) Then
the sum running over all pairs , where denotes a point mass and the big tensor product is the product measure on . Under the -th summand, and for all almost surely, and are independent with distributed according to .
Negative weights are replaced by . When no weight is negative, is a probability measure, and it is the joint law of the following process: has distribution ; each is with probability , independently of ; given the past, has distribution ; and, given all the and , the are independent with distributed according to . When lies outside some weights are negative and are set to , so is then a finite measure that is in general not of total mass .
The function being differentiated. For every real ,
This is a function defined for all real . Integrals are Lebesgue (Bochner) integrals, which by library convention are if the integrand is not integrable; under the hypothesis on each satisfies almost everywhere for each measure involved.
The claimed value of the derivative. For a measure on and an event , let denote the measure (by library convention this is the zero measure if ; here it is used only with and , an event of -measure ). For each time and each Boolean define the real number
that is, the sum over all times with (the term included) of the conditional expectation of under given the single event . The conditioning is on the value of only, not on or on anything else, so is a number and not a function on .
Conclusion. The function is differentiable at the point (in the ordinary two-sided sense), and its derivative there is
Precisely: as in , where is the right-hand side above. The point of differentiation is the fixed number and the quantities are computed under ; no other value of the parameter is addressed.
Confirmed by the mission captain (proposal self-audit).