Motivation
Approximate dynamic programming replaces the exact value functions of a Markov decision process by approximations, typically produced by a regression algorithm. The classical guarantees for the resulting policies are stated in the max norm: if every state-action value is estimated to within ε uniformly over the state space, the greedy policy loses at most a multiple of ε (Williams and Baird 1993; Bertsekas and Tsitsiklis 1996). Regression algorithms, however, control an average error under the distribution on which they are trained, not a worst-case error, so these guarantees rarely apply to the approximations actually computed.
Chapter 5 of S. M. Kakade's PhD thesis (On the Sample Complexity of Reinforcement Learning, UCL 2003, pp. 57–66) identifies the distribution under which the error of an approximate policy iteration algorithm should be measured. For the finite-horizon, non-stationary approximate policy iteration algorithm (NAPI), the loss relative to any competitor policy is bounded by twice the horizon times the regression error averaged under the future state-time distribution of that competitor. Taking the competitor optimal, the relevant "test" distribution is the one visited by an optimal policy, generally not the distribution on which the regressor was trained; the thesis calls this the "mismeasure" of reinforcement learning. The key tool, the performance difference lemma, was proved for the discounted case by Kakade and Langford (ICML 2002) and is used throughout the conservative policy iteration and policy search literature.
Setting
A T-epoch MDP has a finite state set S, a finite nonempty action set A, decision epochs t∈{0,…,T−1}, a transition kernel P(s′∣s,a) and a deterministic reward r(s,a)∈[0,1]. A policy π is a sequence of decision rules π(⋅∣s,t), distributions on A; it may be stochastic and non-stationary. A deterministic policy h chooses the action h(s,t).
Values are normalized. The t-value is Vπ,t(s)=T1E[∑τ=tT−1r(sτ,aτ)∣st=s], so Vπ,t=0 for t≥T and
Vπ,t(s)=a∑π(a∣s,t)(T1r(s,a)+s′∑P(s′∣s,a)Vπ,t+1(s′)),Vπ=Vπ,0.
The state-action value is Qπ,t(s,a)=T1r(s,a)+Es′∼P(⋅∣s,a)Vπ,t+1(s′), the advantage is Aπ,t=Qπ,t−Vπ,t, and the optimal value is V∗(s)=supπVπ(s). The future state-time distribution of π from s0 is
dπ,s0(s,t)=T1Pr(st=s∣π,s0)on S×{0,…,T−1}.
NAPI (Algorithm 6) starts from an arbitrary deterministic policy and updates its decision rules backwards, t=T−1,…,0: at update t a PolicyChooser receives the current policy π and returns a decision rule ht, which replaces the rule at time t. The output is (h0,…,hT−1). With RegressionPolicyChooser (Algorithm 7), a regressor returns an approximation Q~π,t of Qπ,t and ht is greedy for it. The per state error and per state regression error at update t are
εt(s)=amaxQπ,t(s,a)−Qπ,t(s,ht(s)),ε~t(s)=amax∣Qπ,t(s,a)−Q~π,t(s,a)∣,
both measured against the input policy π of update t.
The discounted companions use the normalized discounted value Vπ,γ=(1−γ)E[∑tγtr(st,at)], its advantage Aπ,γ, and the Bellman error BJ=sups,a∣QJ(s,a)−J(s)∣ of a state vector J, where QJ(s,a)=(1−γ)r(s,a)+γEs′J(s′).
Formalization targets
Goal: Theorem 5.3.2 (p. 64)
For the policy π returned by NAPI with RegressionPolicyChooser, every policy π′ and every state s0,
Vπ(s0) ≥ Vπ′(s0)−2TE(s,t)∼dπ′,s0[ε~t(s)].
The regression outputs are arbitrary; the statement holds for every regressor and every tie-breaking rule.
Milestones
- Lemma 5.2.1, undiscounted (p. 59): Vπ′(s0)−Vπ(s0)=TE(s,t)∼dπ′,s0Ea∼π′(⋅∣s,t)[Aπ,t(s,a)].
- Lemma 5.3.1 (p. 63): for NAPI's output π~ and the input policy π of update t, Qπ~,t=Qπ,t and Aπ~,t(s,a)≤εt(s).
- Display in the proof of Theorem 5.3.2 (p. 64): εt(s)≤2ε~t(s) when ht is greedy for Q~π,t.
- Corollary 5.2.2 (p. 61): Vπ(s0)≥V∗(s0)−T∥Aπ,t∥∞ and Vπ,γ(s0)≥Vγ∗(s0)−1−γ1∥Aπ,γ∥∞.
- Theorem 5.1.3 (Williams–Baird, p. 59): a policy greedy for QJ satisfies Vπ,γ(s)≥Vγ∗(s)−1−γ2BJ.
The discounted half of Lemma 5.2.1 is already on the platform as ApproxOptRL.CPI.performance_difference and is referenced, not restated.
Significance
The goal replaces a max-norm hypothesis by an average one, under an explicit distribution. Its consequences: the approximation error only matters on the state-time pairs an optimal (or any good) policy visits; a regressor trained under a distribution μ over state-times performs well when μ resembles the state-time distribution of a good policy; and only one factor of the horizon T appears, against two factors of 1/(1−γ) in the max-norm bounds for discounted approximate value and policy iteration (Chapter 3 of the thesis). The same chain, Lemma 5.2.1 and Lemma 5.3.1, underlies the μ-policy search analysis of Chapter 6 and, in the discounted case, conservative policy iteration.
All statements are proved in the thesis; none of the finite-horizon ones is formalized on the platform. A related performance difference identity, FoundationsRL.RLBasics.performance_difference, is proved on the platform for episodic MDPs with layer-dependent kernels and unnormalized values, with roll-in by one policy; it is the same mathematics up to the 1/T normalization but a different object, and is not reused. The mission produces a reusable normalized T-epoch layer (values, advantages, state-time distributions) and machine-checked proofs of the chapter's results.
Difficulty
The obvious approach to Theorem 5.3.2 compares Vπ with Vπ′ through max-norm contraction arguments, as in the discounted analysis of Chapter 3; that route only yields a bound in terms of maxs,tε~t(s) and loses the averaging under dπ′,s0. The averaged bound needs an exact identity between value differences and advantages along the competitor's trajectory distribution, which for non-stationary stochastic competitors requires a careful telescoping over epochs of the state distribution recursion. The second difficulty is bookkeeping in the algorithm: the regression error at update t is measured against the input policy of that update, whose decision rules at later times have already been replaced, while the bound concerns the returned policy; relating the two requires showing that later updates do not change Q⋅,t.
Formalization scope
- Model.
S, A are Fintypes with decidable equality, A nonempty; P s a s' is P(s′∣s,a) with IsTransitionKernel P; rewards satisfy 0 ≤ r s a ≤ 1 in every statement (the thesis's standing assumption, p. 22). The thesis also allows infinite state spaces; here S is finite.
- Epochs and normalization. Epochs are 0-based, t∈{0,…,T−1}, with
0 < T. All values carry the factor 1/T (discounted: 1−γ), and dπ,s0 carries 1/T; the constants T and 2T are right only in this normalization.
- Policies. A policy is
ℕ → S → A → ℝ, valid when every rule at a time t<T is a distribution. Deterministic policies ℕ → S → A enter through their indicator policies. V∗ is the supremum over the subtype of valid policies. Discounted statements use the published ApproxOptRL.Shared.Model; Vγ∗ is the value of an optimal stationary policy.
- The NAPI run is an explicit recursion (
napiPolicy): the policy after updates T−1,…,t, so the input of update t is the policy at stage t+1 and the output is the policy at stage 0. The printed loop "For t=T−1,…,1" omits the update at t=0, which would leave the time-0 rule random and make the theorem false; the loop here runs to t=0, as in the proof and in Algorithms 8–9.
- Ruled out. A "run" whose returned policy is unconstrained, or regression errors measured against the returned policy instead of the input policy of each update, would trivialize or alter the goal; both are excluded by the explicit recursion. That the two measurements agree is Lemma 5.3.1, a milestone, not a definition.
- Maxima. Maxima over actions are
Finset.sup'; ∥⋅∥∞ on state-action functions is Mathlib's sup norm on S → A → ℝ. BJ is a single number, as printed in Definition 5.1.2.
Welcome contributions: proofs of the milestones, general lemmas about the T-epoch layer (values in [0,1], normalization of dπ,s0, the path-expectation form of Vπ,t), and links to the platform's discounted performance difference results.
Selected references
- S. M. Kakade, On the Sample Complexity of Reinforcement Learning, PhD thesis, Gatsby Computational Neuroscience Unit, University College London, 2003. https://discovery.ucl.ac.uk/id/eprint/10100726/
- S. Kakade and J. Langford, Approximately Optimal Approximate Reinforcement Learning, ICML 2002. https://dl.acm.org/doi/10.5555/645531.656005
- R. J. Williams and L. C. Baird, Tight Performance Bounds on Greedy Policies Based on Imperfect Value Functions, Technical Report NU-CCS-93-14, Northeastern University, 1993.
- D. P. Bertsekas and J. N. Tsitsiklis, Neuro-Dynamic Programming, Athena Scientific, 1996.
- M. L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming, Wiley, 1994. https://doi.org/10.1002/9780470316887