§6.3.4, before (49) — policy value is stable under reward perturbation
ProvedBreakingSample.Perturbed.reward_stabilityp2o-batch-pfp2ap2o-gran-per-chapterp2o-plan-paperp2o-v1reward-perturbationstabilityvalue-function
Keep a transition kernel and deterministic policy fixed, with . If the immediate rewards and differ by at most at every state-action pair, then their policy values satisfy
This bounds how much a bounded reward perturbation changes the performance of any fixed policy.
Formalization Note The statement allows real rewards. The paper's displayed identity just before this inequality omits a discount factor inside a resolvent; the displayed bound uses the discounted resolvent and is the claim formalized here.
Preamble
import Mathlib import Definitions.Def_BreakingSample_Perturbed_Model import Definitions.Def_FoundationsML_ReinforcementLearning_IsTransitionKernel
Formal statement
namespace BreakingSample.Perturbed
open FoundationsML.ReinforcementLearning
/-- The reward perturbation bound in §6.3.4 before (49), arXiv:2005.12900v8, p. 18. -/
theorem reward_stability {S A : Type} [Fintype S] [DecidableEq S] [Fintype A] [DecidableEq A]
(P : S → A → S → ℝ) (r r' : S → A → ℝ) (γ ξ : ℝ)
(hP : IsTransitionKernel P) (hγ₀ : 0 ≤ γ) (hγ₁ : γ < 1)
(hξ : 0 ≤ ξ) (hr : ∀ s a, |r s a - r' s a| ≤ ξ) :
∀ (π : S → A) (s : S),
|value P r γ π s - value P r' γ π s| ≤ ξ / (1 - γ) := by sorry
end BreakingSample.Perturbed
Source
arXiv:2005.12900v8, §6.3.4, display before (49), p. 18
Human review
Confirmed by the mission captain (proposal self-audit).
Confirmed by the moderator at approval.