Motivation
Many performance measures of stochastic models in operations research are expectations computed along a Markov chain until a random time. Examples are the expected discounted cost until a queue empties, the expected reward until a system enters a failure set, and the price of a claim that pays at a hitting time. When the transition law depends on a parameter θ (an arrival rate, a service-time parameter, a short rate in the discount factor), sensitivity analysis asks whether the measure is differentiable in θ and what the derivative is. Derivative formulas underlie gradient-based simulation optimization, likelihood-ratio and infinitesimal-perturbation estimators, and the "greeks" of financial models.
The classical answer for finite state spaces is a matrix calculation: invert I−K(θ0) and differentiate the linear system the measure satisfies. On general state spaces, and when the horizon T has no moment bounds, that calculation needs a substitute for the inverse. C.-H. Rhee and P. W. Glynn (arXiv:1707.03870; Mathematics of Operations Research, 2022) give conditions in terms of stochastic Lyapunov functions: inequalities that involve only the one-step transition kernel and can be checked directly from the model.
Setting
Let S be a measurable state space and Λ=(a,b) an open interval containing θ0. For each θ, P(θ)=(P(θ,x,dy)) is a one-step Markov transition kernel on S. Let f,g:S→R be measurable, and let ∅=C⊆S be measurable. The chain X=(Xn)n≥0 is driven by P(θ), and
T=inf{n≥0:Xn∈Cc}
is its exit time from C. T=∞ is allowed, including the case Cc=∅. The quantity of interest is
u∗(θ,x)=Exθ[j=0∑T−1e∑k<jg(Xk)f(Xj)+e∑k<Tg(Xk)f(XT)],x∈C.
The discounted kernel is K(θ,x,dy)=eg(x)P(θ,x,dy). Its restriction to y∈C is a non-negative kernel K(θ) on C, with powers Kn(θ). The potential kernel is G(θ)=∑n≥0Kn(θ), and G(θ,x,⋅) may be an infinite measure. The one-step reward with exit is f~(θ,x)=f(x)+∫CcK(θ,x,dy)f(y). Then
u∗(θ)=n≥0∑Kn(θ)f~(θ)=G(θ)f~(θ),
and u∗(θ)=f~(θ)+K(θ)u∗(θ) on C. Write K=K(θ0) and G=G(θ0).
The parameter enters through a density: K(θ,x,dy)=k(θ,x,y)K(x,dy) for x∈C (assumptions A1 and A3). Here k(⋅,x,y) is continuously differentiable on [θ0−ε,θ0+ε] with derivative k′ (A2, A3). Put ωε(x,y)=sup∣θ−θ0∣<ε∣k′(θ,x,y)∣, k′(x,y)=k′(θ0,x,y), and r~ε(x)=∫Ccωε(x,y)∣f(y)∣K(x,dy), which A3 assumes finite.
Formalization targets
Goal: Theorem 3.1 (p. 11)
Assume A1–A3, and let v0,v1≥0 be finite functions on C such that
K(θ)v0≤v0−∣f~(θ)∣ (∣θ−θ0∣<ε),Kv1≤v1−∫Cωε(⋅,y)v0(y)K(⋅,dy)−r~ε.
Then for every x∈C, u∗(⋅,x) is differentiable at θ0 with
u∗′(θ0,x)=∫CG(x,dy)∫Ck′(y,z)u∗(θ0,z)K(y,dz)+∫CG(x,dy)∫Cck′(y,z)f(z)K(y,dz).
Assume further that ∫Cωε(x,y)v1(y)K(x,dy)<∞ and that the second Lyapunov inequality holds with K(θ) for every θ∈[θ0−ε,θ0+ε] (3.12′). Then u∗(⋅,x) is continuously differentiable on [θ0−ε,θ0+ε].
The derivative formula is the proof's (3.25). The printed (3.13) omits a term and fails on a two-state example. The second clause is stated with one-sided derivatives at θ0±ε, because with two-sided derivatives it fails at the endpoints.
Milestones
- (3.8), p. 9: for f≥0, the path expectation equals ∑jKj(θ)f~(θ).
- Proposition 3.1, p. 11: Qv≤v−f with f≥0 and v finite implies ∑nQnf≤v.
- Proof of Theorem 3.1, p. 12: ∑nKn(θ)∣f~(θ)∣≤v0, ∣u∗(θ)∣≤v0, and u∗(θ)=f~(θ)+K(θ)u∗(θ).
- Proof of Theorem 3.1, p. 12: the G-integrated Lipschitz bound ≤∣h∣v1(x) and continuity at θ0.
- (3.24), p. 13: the dominating function for the derivative has finite G-integral.
- Proof of Theorem 3.1, p. 13: u∗′(θ)=f~′(θ)+K′(θ)u∗(θ)+K(θ)u∗′(θ) on the closed interval.
Significance
The theorem gives a derivative formula for random-horizon discounted expectations on general state spaces. It needs no moment or tail condition on T, no bound on f, and no sign condition on g. Positive g (growth instead of discounting) is allowed. The hypotheses are one-step drift inequalities, the same kind of condition used for stability of Markov chains, so they are often checkable in queueing and storage models. Because g may depend on θ through the kernel K(θ), the result also covers derivatives with respect to discount-rate parameters ("rho" in finance). The formula GK′Gf~+GKCc′f is also the form an unbiased derivative estimator takes.
The result is proved in the paper; to our knowledge it has no machine-checked proof. This mission asks for a Lean statement of Theorem 3.1 with the two corrections above and machine-checked proofs of it, of the comparison bound (Proposition 3.1), of the series representation of the path expectation, and of each analytic step of the proof. Proposition 3.1 and the potential-kernel machinery are reusable for any drift-condition argument about Markov kernels.
Difficulty
The obvious argument differentiates u∗=f~+K(θ)u∗ and inverts I−K(θ0). On a general state space I−K need not be invertible on any fixed function space, and G need not be a finite kernel. Any rigorous version must work with G, which may be an infinite measure, so every subtraction and every exchange of limit and integral has to be justified for non-negative quantities first. The difference quotients of k are dominated only by the supremum ωε, whose measurability and integrability (once against K and once against G) are themselves part of the work. Continuity of u∗′ on the interval requires the same control at every θ, with K(θ) in place of K.
Formalization scope
- State space and kernels. S is an arbitrary measurable space, with no topology or countability. P(θ) is a Mathlib
Kernel S S with IsMarkovKernel, given for every real θ but constrained only on Λ. K(θ) is Kernel.withDensity of P(θ) by eg; integrals over C and Cc are set integrals.
- Powers and potential. K0=id, Kn+1=K∘kKn, and G(θ,x,⋅) is
Measure.sum of the powers.
- The reading of u∗. u∗ is defined as ∫f~(θ)dG(θ,x,⋅), the series representation the proof uses. For signed f the path expectation can be undefined while every hypothesis holds; milestone (3.8) identifies the two for f≥0.
- Values. ωε, r~ε and the Lyapunov inequalities live in [0,∞], the inequalities written additively. k′ is the derivative within [θ0−ε,θ0+ε].
- Added hypotheses. One ε serves A2, A3 and the theorem, as in the paper's proof. [θ0−ε,θ0+ε]⊆Λ. f is integrable on Cc, the presupposition that f~ is a number. The density is a.e. non-negative, and every function that is integrated is measurable.
- No trivializing reading. The statement is not made vacuous or easy by junk values. The derivative formula carries an integrability clause, so it cannot hold through a Bochner integral defaulting to 0. The supremum ωε is [0,∞]-valued, so an unbounded derivative cannot weaken (3.12). u∗ is built from P and g, never "some solution of u=f~+Ku".
Not formalized: Theorem 4.1 (stationary expectations, which rests on external Harris-chain results), the operator-theoretic Theorems 2.1–2.3, the higher-order Theorem 3.2 (proof omitted in the paper), and the G/G/1 example. Contributions that build general potential-kernel and drift-inequality lemmas for Mathlib kernels are especially welcome.
Selected references