policy_gradient_log_form
Provedanalysiscalculusmdpopen-problempolicy-gradientreinforcement-learning
The log-derivative (score-function / REINFORCE) form of the finite-horizon policy gradient theorem:
This is the form actually implemented by REINFORCE / vanilla policy gradient. Follows from policy_gradient_finite_horizon by rewriting at each (requires ).
Preamble
import Definitions.Def_pg_mdp_core import Mathlib.Analysis.Calculus.Deriv.Basic import Mathlib.Analysis.Calculus.Deriv.Add import Mathlib.Analysis.Calculus.Deriv.Mul import Mathlib.Analysis.SpecialFunctions.Log.Deriv import Mathlib.Data.Fintype.BigOperators import Mathlib.Data.Real.Basic
Formal statement
theorem policy_gradient_log_form {S A : Type} [Fintype S] [Fintype A] [DecidableEq S]
(P : S → A → S → ℝ) (r : S → A → ℝ) (π : ℝ → S → A → ℝ) (θ : ℝ)
(hdiff : ∀ s a, DifferentiableAt ℝ (fun θ' => π θ' s a) θ)
(hpos : ∀ s a, 0 < π θ s a)
(T : ℕ) (s₀ : S) :
deriv (fun θ' => pgValue P r π T θ' s₀) θ
= ∑ t ∈ Finset.range T, ∑ s : S, pgRho P π t s₀ θ s
* ∑ a : A, π θ s a * pgQ P r π (T - 1 - t) θ s a
* deriv (fun θ' => Real.log (π θ' s a)) θ := by sorrySource
R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. ML 8, 1992 (REINFORCE); Sutton et al. NeurIPS 1999.