More Risk-Sensitive Markov Decision Processes 2: The Finite-Horizon Discounted Problem Is Solved by Value Iteration on the State Extended by Accumulated Cost and DiscountResearch Paper
Motivation
Sequential decisions often have costs whose timing matters. A controller may choose an action now, observe a random next state, and choose again. Ordinary expected-cost minimization averages the sum of the costs. A risk-sensitive criterion first applies an increasing function to the total cost and then takes its expectation. The curvature of changes how uncertain costs affect the ranking of policies: convex utility penalizes variability in a cost-minimization problem, while concave utility has the opposite tendency. Bäuerle and Rieder study this criterion for controlled Markov processes with general continuous increasing utility, rather than fixing an exponential function. Their finite-horizon discounted result is Theorem 3.6, pp. 10–12 of the authors' manuscript.
Discounting gives early and late costs different weights. For linear , the remaining expected cost can be described using only the current physical state. For general , the effect of a future cost also depends on what has already been paid and on the discount weight currently attached to the next cost. The mission targets the paper's finite-horizon treatment of these two additional quantities. The infinite-horizon problem is a separate part of the paper.
Setting
The state space and action space are Borel spaces. At state , the nonempty set contains the actions that may be chosen. If action is selected, the next state has distribution and the stage cost is . The cost is measurable and bounded between positive constants and . The discount factor satisfies , and is continuous and strictly increasing. The paper imposes its compactness and continuity conditions (CC) on , , , and ; these are stated in the model item and correspond to §2, pp. 3–4.
A history policy chooses measurably from the states and actions observed through time . Starting from , it induces a law for the state-action trajectory. The first stages incur the discounted cost , with . The paper minimizes over every admissible history policy. The inverse utility appearing in the paper's original certainty-equivalent criterion can be omitted when selecting a minimizer because is strictly increasing §2, p. 3; §3.3, p. 10.
The extended state is . Its coordinates record the current physical state, accumulated cost, and current discount weight. Define and . A measurable extended-state decision rule chooses an action in . The paper's operator integrates a continuation value at ; takes the infimum of the same integral over equations (3.8)–(3.9), pp. 10–11.
Formalization targets
The first target identifies the cost iteration of any sequence of extended-state rules :
The second target is the paper's value iteration for the infimum over all history policies, together with membership of every in its regularity class :
The goal is Theorem 3.6(c): minimizers of exist, and their stage-dependent history rules attain the original objective for every . At stage , the rule uses at the current state, the observed sum , and . “Optimal” compares against all admissible history policies, including those that use more of the history than these three quantities.
Significance
Theorem 3.6 gives a finite sequence of minimum-operator evaluations for a problem whose utility of total cost is not additively separable in the physical state alone. Its policy statement also identifies which observable quantities an optimal controller needs to retain. Together, these results relate the original history-dependent problem to an extended-state Markov decision problem §3.3, pp. 10–12.
The paper proves these statements. This mission asks for machine-checked definitions and proofs of the finite-horizon discounted theorem, including the comparison with all admissible history policies. The model layer and the finite-dimensional expectation construction can support later formalizations of other risk-sensitive objectives. The mission itself has no completed proof at the drafting stage.
Difficulty
The extra discount coordinate is essential: after one action, the next cost enters utility with weight , not . Keeping only accumulated cost would describe a different process. The other demanding point is the policy comparison. An iteration over extended-state decision rules has to establish the value of an infimum over arbitrary measurable history policies. Regularity and measurable selection must also be maintained at each step under (CC). The proof of Theorem 3.6 invokes the corresponding total-cost argument from Theorem 3.1, pp. 5–6; the formal development must supply the precise discounted version.
Formalization scope
Lean represents and as Borel subsets of Polish spaces, with standard Borel measurable structures. The controlled process is a separate reusable definition containing , , and history policies; the risk-sensitive model adds , , and . The admissible graph is Borel and every section is nonempty. A history is a chronological finite list of earlier state-action pairs and a current state. Each rule is defined on all lists, but only admissible lists of the matching length are visited. The transition kernel is total as a Lean object, and its values outside are irrelevant.
Policy values use nested kernel integrals representing the finite-dimensional marginals of the path law. They are defined from the cost and utility, so the Bellman recursion remains a theorem. The optimized value is an infimum over the subtype of measurable, admissible infinite history policies. The real infimum is meaningful here because this policy class is nonempty under (CC) and finite-horizon costs keep utility bounded on the relevant interval. The extended state uses real coordinates restricted to and ; its transition stays in that domain. “Increasing” in means componentwise nondecreasing in . A minimizer is an admissible measurable rule satisfying on the entire extended domain.
The source's proof sentence that preserves is recorded with explicit integrability of each one-step continuation. For an arbitrary real-valued member of on an unbounded state space, the source's conditions alone do not ensure a finite real expectation. This domain condition prevents Lean's zero value for a nonintegrable real integral from turning the assertion into a different one. It does not narrow the finite-horizon values appearing in the goal. Contributions toward finite-dimensional expectation identities, kernel integrability, the regularity of , and measurable minimizer selection are within scope.
Selected references
- N. Bäuerle and U. Rieder, More Risk-Sensitive Markov Decision Processes, authors' manuscript, KIT repository 1000039663; published in Mathematics of Operations Research 39(1):105–120, 2014. Manuscript; DOI.