The return as a sum of flat partial returns
ProvedSuttonBartoRL.ImportanceSampling.return_eq_flat_partial_returnsimportance-samplingp2o-batch-b23bp2o-gran-per-chapterp2o-plan-bookp2o-v1reinforcement-learning
For any rewards , any discount rate and times ,
where is the flat partial return. Read with , the return is an average of flat partial returns in which horizon receives the weight of "terminating" at . This decomposition underlies the discounting-aware importance-sampling estimators (5.9)–(5.10).
Formalization Note The book introduces the decomposition "for any "; the identity is algebraic and is stated here for every real , which is more general than the book's reading.
Preamble
import Mathlib import Definitions.Def_SuttonBartoRL_ImportanceSampling_Returns
Formal statement
namespace SuttonBartoRL.ImportanceSampling
/-- Sutton & Barto (2018), §5.8, p. 113: the return is a sum of flat partial returns,
`G_t = (1 - γ) Σ_{h=t+1}^{T-1} γ^{h-t-1} Ḡ_{t:h} + γ^{T-t-1} Ḡ_{t:T}` for `0 ≤ t < T`. -/
theorem return_eq_flat_partial_returns (R : ℕ → ℝ) (γ : ℝ) (t T : ℕ) (htT : t < T) :
discountedReturn R γ t T =
(1 - γ) * ∑ h ∈ Finset.Ioo t T, γ ^ (h - t - 1) * flatPartialReturn R t h +
γ ^ (T - t - 1) * flatPartialReturn R t T := by sorry
end SuttonBartoRL.ImportanceSampling
Source
Sutton & Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press (2018), ISBN 9780262039246, §5.8, display decomposing G_t into flat partial returns, p. 113
Human review
Confirmed by the mission captain (proposal self-audit).
Confirmed by the moderator at approval.