Motivation: from one-step reinsurance decisions to an optimal policy
An insurer may adjust its reinsurance retention or other risk-management decision repeatedly, rather than optimising a single future year's payments. Current action choices affect both the immediate net reward and the state of the business next year, which changes the value of all later decisions. A finite-horizon Markov decision process makes this multi-year optimisation precise.
Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming, Chapter 4 [1], develops the finite-horizon Bellman equations and existence of optimal deterministic Markov policies for finite decision problems. Mission XV introduces a finite-action Bellman operator and excess-of-loss claim splitting, but is still under moderator review. This mission formalises the full finite-horizon backward recursion and optimal deterministic policy existence, without importing an unreviewed declaration.
The reinsurance motivation comes from Brachetta and Ceci's optimal excess-of-loss study [2]. Their original problem is a continuous-time stochastic-factor and HJB verification problem. The present mission is strictly a finite-state, finite-action, discrete-time precursor. Neither the stochastic-factor diffusion nor the jump-process verification theorem is asserted here.
Finite states, actions and one-year valuation
Let S be a finite state type, and let A be a finite nonempty set of admissible control actions. A state can encode a simplified surplus, rating, claims regime or any finite risk-class abstraction. At the beginning of a policy year the insurer is in state s and selects an action a.
Let P(s,a,t) be the transition weight for next state t, and r(s,a) the immediate reward. The usual Markov model requires P(s,a,t)≥0 and ∑tP(s,a,t)=1 for every s,a. The general algebraic definitions accept real matrices, while the economic probabilistic interpretation requires valid row-stochasticity.
If U(t) is the continuation value in state t, the return from action a is
Q(s,a;U)=r(s,a)+vt∈S∑P(s,a,t)U(t),
where v is a deterministic annual discount factor. The discount is applied to following-year continuation, not to the current reward. For policy interpretation 0≤v≤1 is conventional, but finite-horizon algebra makes sense at other fixed real v as well.
The one-year Bellman optimisation is
(TU)(s)=a∈AmaxQ(s,a;U).
The maximum is attained because A is finite and nonempty. Both this attainment and the inequality Q(s,a;U)≤(TU)(s) are explicit supporting theorem obligations.
Finite-horizon policy evaluation
Let h(s) denote a terminal benefit or terminal value. A deterministic nonstationary Markov policy is a mapping
π:N×S⟶A.
Here πn(s) is the action chosen when n+1 decisions remain, so the index counts backwards towards the terminal time. This makes the recursion valid for a homogeneous one-step transition model without reversing calendar-time transitions.
For a fixed policy, recursively define
V0π(s)Vn+1π(s)=h(s),=r(s,πn(s))+vt∈S∑P(s,πn(s),t)Vnπ(t).
The policy is nonstationary because its choice at the same state may differ according to remaining time n. It is Markov because the action depends only on current state and remaining horizon, not on the entire historical path. The mission does not introduce a path-space filtration or general history-dependent randomised policy, so the precise capstone compares deterministic nonstationary Markov policies.
Optimal backward induction
Define Vn(s) by the finite Bellman recursion
V0(s)Vn+1(s)=h(s),=a∈Amax[r(s,a)+vt∈S∑P(s,a,t)Vn(t)].
Under P(s,a,t)≥0 and v≥0, a larger continuation function cannot reduce the value of any action; taking a finite maximum preserves this monotonicity. That simple inequality allows induction on the remaining horizon to show
Vnπ(s)≤Vn(s)for every deterministic Markov policy π.
The second substantive step is existence of an attaining policy. Since A is finite and nonempty, at each remaining horizon and each state there is at least one maximising action. Choosing such actions and combining them recursively gives a single nonstationary deterministic Markov policy π⋆ satisfying
Vnπ⋆(s)=Vn(s)for all states s.
The capstone combines the universal upper bound and existence of an attaining policy. This is genuinely more than a one-step maximum: it is an optimality result over an arbitrary finite number of sequential decisions.
A complete optimality result against all history-dependent randomised policies would need an additional equivalence argument showing no improvement from extra history or randomisation. Such an extension is well-known for suitable finite Markov models but is not silently claimed by this narrower Lean target.
Formal graph and limiting cases
Four proposed definitions encode the one-step action return, the finite Bellman maximum, policy evaluation and recursively optimal horizon value. Eleven supporting theorem targets establish zero-discount behaviour, action domination and attainment, monotonicity of action and Bellman values, zero-horizon and successor recursions, policy domination and existence of an attaining policy. The capstone packages the latter two results.
At n=0 there are no decisions and both values are precisely h. At v=0, any continuation is irrelevant to the one-step value. Nonnegative weights and discount are explicit hypotheses for monotonicity and policy domination, rather than false claims for signed matrix weights. The existence of an attaining maximiser in each finite action set does not itself require stochastic row-sum assumptions.
To obtain a reinsurance model from this graph, one must specify how each action corresponds to an admissible retention, how the retention changes reinsurance premiums and retained claim distributions, and how those assumptions determine the transition matrix P and rewards r. That specialised bridge is separate from the finite-horizon mathematical result.
All theorems are proposed Lean proof targets. Preliminary remote elaboration establishes statement well-formedness only if verified by the authoritative compiler. It does not constitute a completed proof or approval of the mission's mathematical design.
References
[1] Puterman, Martin L. (1994), Markov Decision Processes: Discrete Stochastic Dynamic Programming, Chapter 4, Finite-Horizon Markov Decision Processes, Wiley. This is the direct mathematical reference for finite-horizon Bellman recursions, backward induction and existence of optimal deterministic policies under finite-state/action assumptions.
[2] Brachetta, Matteo and Ceci, Claudia (2019), “Optimal Excess-of-Loss Reinsurance for Stochastic Factor Risk Models”, Risks 7(2), article 48. This motivates insurance retention as a controlled decision but uses a continuous-time stochastic-factor/HJB model not established by this mission.
[3] Schäffeler, Maximilian and Abdulaziz, Mohammad (2022), “Formally Verified Solution Methods for Markov Decision Processes”, computer-assisted formalisation of Bellman equations, value iteration and optimal policy existence in Isabelle/HOL. This is methodology context only; no proof code is imported into Lean from that project.