Under a stationary policy (f,f,…) the actions are f(X_n) and the state process is the Markov chain with kernel Q(·|x,f(x))
ProvedRiskSensMDP.AverageCost.stationary_state_processionescu-tulceamarkov-chainrisk-sensitivestationary-policy
Let be the Borel MDP of Bäuerle & Rieder (§2) and let be a stationary policy given by a measurable decision rule with . Let be the law of the controlled trajectory started at . Then
- the actions are for all , -almost surely;
- the law of the state process under is the law of the time-homogeneous Markov chain with transition kernel
started at .
This is the identification, used implicitly on p. 17 of the paper ("the corresponding controlled Markov chain "), that lets Markov-chain results such as the Harris law of large numbers be applied to the cost process of a stationary policy.
Formalization Note is pathMeasure M f.toPolicy x and the chain law is chainLaw (stateKernel M f) x; item 2 says that the pushforward of under equals the chain law.
Preamble
import Mathlib import Definitions.Def_RiskSensMDP_AverageCost_Model import Definitions.Def_RiskSensMDP_AverageCost_HarrisRecurrence open MeasureTheory ProbabilityTheory Filter Finset
Formal statement
namespace RiskSensMDP.AverageCost
/-- Under a stationary policy the actions are `A_n = f(X_n)` a.s., and the state process is the
Markov chain with kernel `P_f(x, ·) = Q(· | x, f(x))` started at `x`. -/
theorem stationary_state_process {E A : Type*} [MeasurableSpace E] [StandardBorelSpace E]
[MeasurableSpace A] [StandardBorelSpace A]
(M : Model E A) (f : StationaryRule M) (x : E) :
(∀ᵐ ω ∂(pathMeasure M f.toPolicy x), ∀ n : ℕ, (ω n).2 = f.f (ω n).1) ∧
(pathMeasure M f.toPolicy x).map (fun ω n => (ω n).1) = chainLaw (stateKernel M f) x := by sorry
end RiskSensMDP.AverageCost
Source
Bäuerle & Rieder, More Risk-Sensitive Markov Decision Processes, authors' manuscript (KIT repository 1000039663; Math. Oper. Res. 39(1):105–120, 2014), §2 p. 3 (construction of P^σ_x) and §5.1 p. 17 (Theorem 5.1: 'the corresponding controlled Markov chain (X_n)')