Lemma 3.1 — Averaged reward-to-go recursion
ProvedXuDRMDP.Finite.lemma_3_1Let be a finite nonempty staged state set with finite nonempty action sets , initial stage consisting of alone, finite horizon , zero terminal rewards and discount . Assume the nested local parameter sets () are nonempty, convex and compact sets of valid transition/reward vectors. Local sets may couple all actions. Let . The local class consists of all Borel probability laws supported on with , and consists of their statewise product laws. Policies are all history-dependent randomized policies on state/action histories, including null histories. Returns are the independent finite trajectory expectations, subsequently integrated against the parameter law.
Fix any policy , admissible marginal family , its global product law , and any admissible history with . Write and
Then both equalities hold:
The history need not have positive probability under the chosen policy or any parameter realization.
Differences from the source:
- encoding: finite states and state-dependent actions, one-based staged histories, and expected discounted trajectory returns → nonempty Fin types, zero-based stages, total policy extensions on unused histories, and explicit finite trajectory sums followed by Bochner integration; reason: finite relabeling and unused extensions preserve all source instances; compact outer support ensures integrability and finite bounded returns
- reading: Lemma 3.1 prints “we-” followed by “0 have” → the two displayed equalities with no extra condition; reason: this broken wording is visibly typography, not a mathematical hypothesis
- encoding: sum over all next states S in equation (2) → sum restricted by an indicator to the next stage; reason: staged transition probabilities vanish at all other states, including for mean parameters; their impossible continuations contribute zero
import Definitions.Def_XuDRMDP_Finite_Setup import Definitions.Def_XuDRMDP_Finite_Policies import Definitions.Def_XuDRMDP_Finite_Ambiguity import Definitions.Def_XuDRMDP_Finite_Returns import Definitions.Def_XuDRMDP_Finite_Robustness open MeasureTheory
namespace XuDRMDP.Finite
theorem lemma_3_1 {n : ℕ} [NeZero n] {a : Fin n → ℕ} [∀ s, NeZero (a s)]
(G : Setup n a) (U : Uncertainty G) (π : Policy a)
(μ : Marginals n a) (hμ : admissibleMarginals U μ)
(h : History a) (hh : Admissible G h) (ht : G.stage h.state + 1 < G.horizon) :
expectedFrom G π (productLaw μ) h =
∫ x, averagedStep G π (productLaw μ) h x ∂(μ h.state) ∧
(∫ x, averagedStep G π (productLaw μ) h x ∂(μ h.state)) =
averagedStep G π (productLaw μ) h (globalMean (productLaw μ) h.state) := by sorry
end XuDRMDP.Finite
Read-back
What the Lean code literally says, in plain math · claude-opus-5-5
Data.
- states and actions with every .
- A setup : horizon , discount , stage map with , and initial state , the unique state of stage .
- An uncertainty structure : confidence levels , and for each state nested nonempty convex compact sets in . Their points are valid local MDP coordinates: they equal some imported-type MDP's data at , have zero rewards if , and otherwise zero transition entries to states not at stage .
- An arbitrary policy : a probability vector on for every history , possibly depending on the whole past.
- A marginal family that is admissible: each is a probability measure on with and for all .
- A history that is admissible, i.e. of the form with , whose current state satisfies .
Let be the product law on . Write points as with .
Return. For a history with current state and , is the finite sum, over sequences and final states , of
where:
- ;
- is the history whose past is followed by , and whose current state is .
Let .
Averaged step. For a local point :
where appends to and sets the current state to . The continuation values use the full product law , including its -coordinate.
Claim. Both of the following equalities hold:
and
where is the -coordinate mean under the product law, a vector-valued Bochner integral.
Degenerate cases.
- The hypothesis forces . In particular the statement says nothing about histories at the last stage, nor about any history when .
- The structure enters only through the admissibility of .
- All integrals are Bochner integrals, which by convention equal (or the zero vector) when the integrand is not integrable. The equalities are asserted with that convention in force.
- If no state has stage , the inner sum in is empty and .
- If , is the point mass on the single action.
Confirmed by the mission captain (proposal self-audit).
Confirmed by the moderator at approval.