Proof of Theorem 1, p. 36, last display — ∇φ Σ_{R,P} ∇φᵀ = (I − γP̃^{π*})⁻¹W⁻¹[D_R + γ²D_Q]((I − γP̃^{π*})⁻¹)ᵀ
ProvedRLInference.QValueCLT.covariance_identitycovariancedelta-methodmatrix-identityp2o-batch-pfp2ap2o-gran-per-chapterp2o-plan-paperp2o-v1reinforcement-learning
Let , , let be any action map and any positive weight vector. Let
be the gradient of App. A.1, and the covariance (8). Then
which is the covariance of (5)–(6).
This identity turns the delta-method covariance into the closed form stated in Theorem 1.
Formalization Note. The identity is pure matrix algebra, with matrices indexed by and . It does not use stationarity of , the strict argmax, or ; only is assumed, as in the paper.
Preamble
import Mathlib import Definitions.Def_RLInference_QValueCLT_Setting open MeasureTheory ProbabilityTheory Filter Matrix Topology
Formal statement
namespace RLInference.QValueCLT
theorem covariance_identity {S A : Type} [Fintype S] [DecidableEq S] [Nonempty S]
[Fintype A] [DecidableEq A] [Nonempty A]
(γ : ℝ) (P : S × A → PMF S) (R : S × A → Measure ℝ) (aStar : S → A)
(w : S × A → ℝ) (hw_pos : ∀ x, 0 < w x) :
gradPhi γ (Pmat P) aStar (Vstar (bellmanFix γ (muR R) (Pmat P))) *
SigmaRP (Pmat P) (sigma2R R) w *
(gradPhi γ (Pmat P) aStar (Vstar (bellmanFix γ (muR R) (Pmat P))))ᵀ =
SigmaQ γ (Pmat P) aStar (Vstar (bellmanFix γ (muR R) (Pmat P))) (sigma2R R) w := by sorry
end RLInference.QValueCLT
Source
Zhu, Dong & Lam, Uncertainty Quantification and Exploration for Reinforcement Learning, arXiv:1910.05471v3, p. 36, App. A.1, last display; (5)–(6) p. 10, (8) p. 12
Human review
Confirmed by the mission captain (proposal self-audit).
Confirmed by the moderator at approval.