policy_gradient_discounted_finite
Provedalgebraic-geometrycalculusmdpopen-problempolicy-gradientreinforcement-learning
The discounted finite-horizon policy gradient theorem (the form actually used in practice):
The discounted -step value is pgValue (γ·P) r π T θ s (substituting gives exactly the discounted Bellman recursion ), and pg_rho_gamma_scale converts pgRho (γ·P) to γ^t · pgRho P. Thus this follows directly from policy_gradient_finite_horizon and pg_rho_gamma_scale — no new induction. Taking with gives the classical Sutton et al. 1999 infinite-horizon statement.
Preamble
import Definitions.Def_pg_mdp_core import Mathlib.Analysis.Calculus.Deriv.Basic import Mathlib.Analysis.Calculus.Deriv.Add import Mathlib.Analysis.Calculus.Deriv.Mul import Mathlib.Data.Fintype.BigOperators import Mathlib.Data.Real.Basic
Formal statement
theorem policy_gradient_discounted_finite {S A : Type} [Fintype S] [Fintype A] [DecidableEq S]
(P : S → A → S → ℝ) (r : S → A → ℝ) (π : ℝ → S → A → ℝ) (θ : ℝ) (γ : ℝ)
(hdiff : ∀ s a, DifferentiableAt ℝ (fun θ' => π θ' s a) θ)
(T : ℕ) (s₀ : S) :
deriv (fun θ' => pgValue (fun s a s' => γ * P s a s') r π T θ' s₀) θ
= ∑ t ∈ Finset.range T, γ ^ t * ∑ s : S, pgRho P π t s₀ θ s
* ∑ a : A, pgQ (fun s a s' => γ * P s a s') r π (T - 1 - t) θ s a
* deriv (fun θ' => π θ' s a) θ := by sorrySource
R. Sutton, D. McAllester, S. Singh, Y. Mansour. Policy Gradient Methods for RL with Function Approximation. NeurIPS 1999.