Robust Optimization XII: Finite-Horizon Robust Bellman RecursionTextbook
Decision making with uncertain transitions
A routing decision can have a known immediate cost while its consequences depend on uncertain weather. A controller observes the current state, chooses an action, and then faces a random next state. A policy optimized for one estimated transition model may perform poorly when those estimates change. Chapter 13 of Ben-Tal, El Ghaoui and Nemirovski, Robust Optimization studies this issue through finite-state control problems and illustrates it with aircraft routing. The mathematical target here is the finite-horizon theorem that evaluates and optimizes policies against admissible changes in transition probabilities.
The uncertainty model matters. Requiring nature to use one fixed transition matrix throughout the horizon couples its choices across time. Allowing each time, state and action to have its own admissible probability row gives a different problem. The latter is the setting of this mission and of the chapter's robust Bellman recursion. No assertion of exact solution of the stationary-uncertainty problem is included.
Finite states, actions, and independent uncertainty rows
Let the state space be a finite nonempty set , and let the action space be a finite nonempty set , available in every state. The decision times are , with terminal time . A deterministic Markov policy assigns an action to every time and state . The stage cost and terminal cost are real numbers; they need not be nonnegative. The initial state has a given probability distribution .
A transition row gives the probabilities of the next state, conditional on time , current state , and action . For fixed policies, distributions evolve according to
The expected total cost is defined from these forward distributions:
For robust control, every row belongs to a nonempty closed set in the probability simplex. Rectangular uncertainty means that nature's admissible policies are the entire product of these sets: choices at different times, states and actions are independent constraints. The sets themselves may depend on time and need not be convex. These are the standing assumptions of Sections 13.2–13.3 of the book.
Formalization targets
The first supporting target is Theorem 13.1.2. For a fixed controller and fixed transition rows, the expected cost equals the attained maximum of the linear program with terminal vector and inequalities
whose objective is . The second target, Theorem 13.1.4, is the nominal Bellman theorem: the recursion that minimizes the corresponding one-step expression over actions gives the optimal remaining cost at every time and state, and greedy policies for its values are optimal.
The goal is the full Theorem 13.3.1. With , define
Then is the minimum worst-case remaining expected cost starting at state at time , and
The target includes existence of controller policies choosing the displayed minimizing actions and nature policies choosing maximizing rows. Every such controller is optimal, and corresponding maximizing rows realize its worst cost. For any fixed controller , the separate evaluation recursion
must give its attained worst expected remaining cost, including the total value .
What the results establish
These statements connect three different objects: a forward stochastic evolution, optimization over complete policies, and backward value calculations. The robust theorem supplies both an optimal decision rule and an evaluation of uncertainty for a prescribed rule. Keeping the initial distribution arbitrary retains the distinction between statewise values and the value of the original control problem.
The results are established mathematics in the source; the work requested is their formal verification. A related proved robust-DP theorem uses time-independent uncertainty sets, nonnegative stage costs and a fixed initial state. This mission preserves the book's different data and attainment conclusions. Its local statements are open proof targets, not completed Lean proofs.
Difficulty
The central issue is relating the independent forward cost to the policy optimization and preserving all attainments. Choosing an individually favorable action does not by itself establish optimality over complete policies. Likewise, replacing a maximum by a real supremum leaves the existence of a maximizing nature policy unresolved. Closedness of the row sets and their probability constraints are substantive assumptions here. The fixed-policy and optimizing recursions must also be distinguished: nature's worst choices for an arbitrary controller need not be the rows selected from the optimal value function.
Formalization scope
States, actions and decision times use finite index types, with nonempty state and action spaces. All costs and probabilities are real. Probability vectors have nonnegative coordinates summing to one. The model retains arbitrary initial distributions, time-dependent uncertainty sets and terminal costs. A zero horizon has no decisions and yields terminal expected cost.
The definition bundle includes forward distributions, expected costs, both Bellman recursions, the independent worst-cost function, and the policy-selection predicates. The theorems compare these independently defined quantities, rather than defining expected cost to be the desired Bellman value. Extrema use attained order predicates, with internal suprema restricted to the admissible nature policies. Full-horizon policies encode suffix policies by ignoring coordinates before the starting time.
Formalization Note Each theorem carries its needed concrete definitions in a separate namespace so it compiles without importing unpublished drafts. The definition item records the common reusable interface. Equation (13.1.3) supplies the forward-distribution definition rather than a tautological milestone. The nominal LP Theorem 13.1.3 is not a dependency in this selected route; its printed index and arbitrary-optimizer issues are recorded locally. No corrected or strengthened variant is posed.
Selected references
- A. Ben-Tal, L. El Ghaoui and A. Nemirovski, Robust Optimization, Princeton University Press, 2009, Chapter 13, especially Theorems 13.1.2, 13.1.4 and 13.3.1, printed pp. 345–346 and 350. DOI.