Motivation
A Markov decision process (MDP) is evaluated by solving a linear fixed-point equation built from its transition probabilities. In practice those probabilities are rarely known exactly: they are estimated from data, elicited from experts, or obtained by aggregating the states of a larger model, and each estimate comes with an error band. Givan, Leach and Dean (Bounded-parameter Markov decision processes, Artificial Intelligence, 2000) replace each transition probability by a closed interval and ask what can still be said about the value of a policy. Their answer for a fixed policy is an interval value: at each state, the smallest and the largest expected discounted reward the policy can obtain over all exact MDPs consistent with the intervals.
The same model reappears under other names. Robust MDPs with rectangular uncertainty sets (Nilim and El Ghaoui, Operations Research, 2005; Iyengar, Mathematics of Operations Research, 2005) take the pessimistic end of the interval, and optimistic model-based reinforcement learning (for example UCRL-type algorithms) takes the optimistic end over a confidence set. Interval transition models are also used to bound the error of state aggregation. In all of these, the basic computational question is how to evaluate a fixed policy against a whole family of MDPs at once.
Setting
Fix a finite set Q of states and a finite nonempty set A of actions. An exact MDP M=⟨Q,A,F,R⟩ has transition probabilities Fpq(α), each row q↦Fpq(α) a probability distribution, a reward R(q)∈R per state, and a discount rate 0≤γ<1. A policy is a map π:Q→A. Its value function VM,π is the expected discounted cumulative reward of the Markov chain with transition matrix Fpq(π(p)):
VM,π(p)=t≥0∑γt(PπtR)(p),Pπ(p,q)=Fpq(π(p)).
On the space V of value functions v:Q→R, with the sup norm ∥v∥=maxq∣v(q)∣, the value-iteration operator is VIM,π(v)(p)=R(p)+γ∑qFpq(π(p))v(q), and VIM,α is the same with the action α in place of π(p). An operator T on V is a contraction mapping if ∥Tv−Tu∥≤λ∥v−u∥ for all u,v and some 0≤λ<1.
A bounded-parameter MDP (BMDP) M↕ gives intervals [F↓pq(α),F↑pq(α)]⊆[0,1] with ∑qF↓pq(α)≤1≤∑qF↑pq(α), a reward R and a discount rate γ. An exact MDP belongs to M↕ if it has the same R and γ and every transition probability lies in its interval. The interval value of a policy π (Definition 3) is
V↕π(q)=[V↓π(q),V↑π(q)]=[M∈M↕minVM,π(q), M∈M↕maxVM,π(q)].
Interval policy evaluation acts on an interval value function V↕=[V↓,V↑] by
IVI↕π(V↕)(p)=[M∈M↕minVIM,π(V↓)(p), M∈M↕maxVIM,π(V↑)(p)],
with lower and upper bound maps IVI↓π,IVI↑π:V→V.
For the optional part, interval value iteration adds a maximization over actions: IVI↕opt and IVI↕pes take, at each state, the largest of the candidate intervals [minMVIM,α(V↓)(p),maxMVIM,α(V↑)(p)] for the lexicographic orders ≤opt (upper end first) and ≤pes (lower end first). For a value function V, ρV(p) and σV(p) are the actions maximizing maxMVIM,α(V)(p) and minMVIM,α(V)(p), and IVI↓opt,V(V′)=IVI↓opt([V′,V]), IVI↑pes,V(V′)=IVI↑pes([V,V′]).
Formalization targets
Goal: convergence of interval policy evaluation (Section 5.1, p. 22)
For every BMDP, every policy π, and every interval value function V↕0=[V↓0,V↑0] with V↓0≤V↑0,
n→∞limIVI↕πn(V↕0)=V↕π.
The limit is the interval value of Definition 3, an extremum over infinitely many MDPs; nothing about the starting point is assumed beyond its being an interval value function.
Milestones on the goal's path
- Theorem 10 (p. 21): IVI↓π and IVI↑π are contraction mappings on V.
- Theorem 11 (p. 22): IVI↓π(V↓π)=V↓π, IVI↑π(V↑π)=V↑π, and hence IVI↕π(V↕π)=V↕π.
Further results on interval value iteration (Section 5.2)
- Lemma 4 (p. 25): IVI↓opt,V(V′)(p)=maxα∈ρV(p)minMVIM,α(V′)(p) and IVI↑pes,V(V′)(p)=maxα∈σV(p)maxMVIM,α(V′)(p).
- Theorem 13(a) (p. 25): IVI↑opt and IVI↓pes are contraction mappings.
- Theorem 13(b) (p. 25): for every value function V, IVI↓opt,V and IVI↑pes,V are contraction mappings.
Significance
The result. The interval value is defined as a minimum and a maximum over a continuum of MDPs, and there is no a priori reason it should be computable by a dynamic program. The goal says it is: one operator, whose per-state step is a small linear program over a box intersected with a simplex, iterated from any start, converges to both ends of the interval simultaneously. The pessimistic end is the robust value of the policy under rectangular uncertainty, so the goal is also a policy-evaluation theorem for interval-rectangular robust MDPs; the optimistic end is the value used by optimism-based exploration. Theorem 11 is the bridge between the "game" definition (nature picks the MDP) and the fixed-point characterization; Theorem 13 is the corresponding first step for the optimal interval values of the companion mission.
Formalizing it. The results are proved in the paper (2000), with Theorem 11 relying on the existence of a single member MDP that is π-minimizing at every state simultaneously (the paper's Theorem 7). No machine-checked version of the BMDP model or of these results is known. Nearby statements formalized elsewhere concern other models: robust MDPs with costs and general row sets, and optimistic Bellman equations over compact confidence sets. This mission produces the BMDP model and its interval operators, the contraction and fixed-point theorems, and the convergence theorem, all stated against the member-MDP definition of V↕π.
Difficulty
The contraction estimates (Theorems 10 and 13) follow the familiar pattern for Bellman operators, with the extra point that the minimum or maximum over the infinite family M↕ must be attained. The real difficulty is Theorem 11. The obvious argument compares V↓π(q) with VIM,π(V↓π)(q) state by state; but V↓π(q) is a minimum taken separately at each state, possibly by a different MDP at each state, and V↓π need not a priori be the value function of any single member MDP. Without a member MDP that is minimizing at all states at once, the inequality IVI↓π(V↓π)≥V↓π does not follow. Establishing that simultaneous minimizer is the central step.
Formalization scope
All statements live in the namespace BoundedParamMDP.IntervalEval. The exact MDP, the BMDP, its member MDPs, VM,π, VIM,π, VIM,α and Definition 3's V↓π, V↑π are the shared definitions BoundedParamMDP.Optimal.MDP and BoundedParamMDP.Optimal.BMDP used by the companion mission, so its theorems apply to the same objects. Conventions:
- Q and A are finite types, A is nonempty; Q may be empty (every statement then holds for the paper's own reasons). Policies are deterministic and stationary,
Q → A.
- An exact MDP is a structure with
F p α q =Fpq(α), rows nonnegative and summing to 1, a state reward R, and the discount rate 0≤γ<1 as a field. Rewards are tight (the paper's footnote 3): members share the BMDP's R and γ.
- VM,π is the discounted series ∑tγtPπtR, not a solution chosen from the Bellman equation.
- Minima and maxima over M↕ are the real infimum and supremum over the nonempty subtype
Member B; all quantities are bounded, so these are the genuine values (and attained, by the paper's Lemma 1).
- An interval value function is a pair
(V↓, V↑) of functions Q → ℝ. The operators are defined on all pairs; the goal assumes V↓0≤V↑0 as the paper's interval notion, though the conclusion does not need it.
- The norm is Mathlib's norm on
Q → ℝ (the sup norm (4)); convergence of pairs is in the product topology. A contraction mapping is "there exists λ∈[0,1)", as on p. 6; the modulus is not fixed to γ.
- maxα,≤opt is the supremum over the finite action set in the lexicographic order on (upper end, lower end); ≤pes uses (lower end, upper end). ρV(p), σV(p) are sets of actions.
- Theorem 13(a) is stated for IVI↑opt as the upper end of IVI↕opt with arbitrary lower inputs on both sides: the bound by λ∥U′−U∥ includes the independence from the lower input that the paper uses (p. 23) to regard IVI↑opt as a map on V.
- Lemma 4's second line is printed with minM; the evident reading maxM (eq. (29), proof of Theorem 13(b), p. 47) is stated.
A trivializing formalization is excluded: the limit in the goal is Definition 3's infimum and supremum over member MDPs, not the fixed point of IVI↕π, so the goal is not Banach's theorem alone; and the member type is nonempty for every BMDP, so the infima and suprema are not the junk value 0.
The proof of Theorem 11 uses the paper's Lemma 1 (attainment in the order-maximizing family), Theorem 3 (VM,π is the unique fixed point of VIM,π), Theorem 6 (comparison principle) and Theorem 7 (existence of π-minimizing and π-maximizing MDPs). These are milestones of the companion mission Bounded-parameter Markov Decision Processes 1 and are not restated here. The Banach fixed-point theorem is Mathlib's ContractingWith. Contributions welcome: proofs of the milestones, a reusable lemma on the attained minimum of a linear function over a box intersected with the simplex, and the order-maximizing construction itself.
Selected references
- R. Givan, S. Leach, T. Dean, Bounded-parameter Markov decision processes, Artificial Intelligence 122 (2000). https://doi.org/10.1016/S0004-3702(00)00047-3
- A. Nilim, L. El Ghaoui, Robust control of Markov decision processes with uncertain transition matrices, Operations Research 53(5), 2005. https://doi.org/10.1287/opre.1050.0216
- G. N. Iyengar, Robust dynamic programming, Mathematics of Operations Research 30(2), 2005. https://doi.org/10.1287/moor.1040.0129
- M. L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming, Wiley, 1994. https://doi.org/10.1002/9780470316887