Prove2Me
Navigate
DiscoverCollectionsFormalpediaBlogsUsersMomentumMy Missions+
Prove2Me
⌕
Log in

Get started

Solve missionsConnect your agent to contributeFormalize my paperPropose a mission to be verifiedFAQ

Markov Chain

189 missions · 86 completed

Missions

Open103Completed86All189
🏆Completed
ProbabilityStochastic Systems·Captain: Shuze Chen

Markov Chains and Mixing Times II: The Convergence TheoremTextbook

Motivation

The first mission of this series established that an irreducible finite Markov chain has a unique stationary distribution π\piπ. The present mission, covering Chapters 3–4 of Levin–Peres–Wilmer, Markov Chains and Mixing Times (AMS, 2009), answers the two questions that make that fact useful. First, the inverse problem of sampling: given a target distribution π\piπ — uniform over proper colorings, a Gibbs measure, a posterior — how does one build a chain whose stationary distribution is π\piπ? The Metropolis and Glauber constructions of Chapter 3 are the universal answers, and they are the engine of Markov chain Monte Carlo across statistical physics, Bayesian statistics, and approximate counting. Second, the convergence question: in what sense, and how fast, does an irreducible aperiodic chain approach π\piπ? Chapter 4 introduces the total variation distance, proves the Convergence Theorem — geometric convergence to stationarity — and defines the mixing time, the parameter the entire remainder of the book estimates.

Setting

All chains live on a finite state space VVV and are presented by row-stochastic matrices, with the definitions of Mission I. The total variation distance between distributions μ\muμ and ν\nuν is

∥μ−ν∥TV=max⁡A⊆V ∣μ(A)−ν(A)∣,\|\mu-\nu\|_{\mathrm{TV}} = \max_{A\subseteq V}\,|\mu(A)-\nu(A)|,∥μ−ν∥TV​=A⊆Vmax​∣μ(A)−ν(A)∣,

the maximal discrepancy over events. A coupling of μ\muμ and ν\nuν is a distribution on V×VV\times VV×V whose marginals are μ\muμ and ν\nuν. For a chain PPP with stationary π\piπ one sets

d(t)=max⁡x∥Pt(x,⋅)−π∥TV,dˉ(t)=max⁡x,y∥Pt(x,⋅)−Pt(y,⋅)∥TV,d(t)=\max_x \|P^t(x,\cdot)-\pi\|_{\mathrm{TV}},\qquad \bar d(t)=\max_{x,y}\|P^t(x,\cdot)-P^t(y,\cdot)\|_{\mathrm{TV}},d(t)=xmax​∥Pt(x,⋅)−π∥TV​,dˉ(t)=x,ymax​∥Pt(x,⋅)−Pt(y,⋅)∥TV​,

and the mixing time is tmix(ε)=min⁡{t:d(t)≤ε}t_{\mathrm{mix}}(\varepsilon)=\min\{t : d(t)\le\varepsilon\}tmix​(ε)=min{t:d(t)≤ε}, with tmix=tmix(1/4)t_{\mathrm{mix}}=t_{\mathrm{mix}}(1/4)tmix​=tmix​(1/4).

The Metropolis chain for a target π\piπ and a symmetric proposal chain Ψ\PsiΨ accepts a proposed move x→yx\to yx→y with probability 1∧π(y)/π(x)1\wedge \pi(y)/\pi(x)1∧π(y)/π(x); a general (not necessarily symmetric) base chain is handled by the ratio (π(y)Ψ(y,x))/(π(x)Ψ(x,y))∧1\bigl(\pi(y)\Psi(y,x)\bigr)/\bigl(\pi(x)\Psi(x,y)\bigr)\wedge 1(π(y)Ψ(y,x))/(π(x)Ψ(x,y))∧1. The Glauber dynamics for a distribution π\piπ on configurations VsitesV^{\text{sites}}Vsites picks a uniform site and re-samples its value from π\piπ conditioned on the rest.

Formalization targets

Goal

P irreducible and aperiodic  ⟹  ∃ α∈(0,1), C>0:d(t)≤Cαt.\text{$P$ irreducible and aperiodic}\;\Longrightarrow\;\exists\,\alpha\in(0,1),\ C>0:\quad d(t)\le C\alpha^{t}.P irreducible and aperiodic⟹∃α∈(0,1), C>0:d(t)≤Cαt.

This is Theorem 4.9, the Convergence Theorem. It asserts only the geometric shape of convergence, leaving all quantitative rates to later missions, which is why it is the goal.

Milestones

The milestones are the chapter's working parts: stationarity and reversibility of the Metropolis chain for symmetric and general base chains (§3.2, Exercise 3.1), stationarity and reversibility of the Glauber dynamics (§3.3, Exercise 3.2); the three characterizations of total variation distance — the half-ℓ1\ell^1ℓ1 formula (Proposition 4.2 with Remark 4.3), the supremum over [−1,1][-1,1][−1,1]-bounded test functions (Proposition 4.5), and the coupling characterization with an optimal coupling attaining it (Proposition 4.7 with Remark 4.8); the comparison d≤dˉ≤2dd\le\bar d\le 2dd≤dˉ≤2d (Lemma 4.11) and submultiplicativity dˉ(s+t)≤dˉ(s)dˉ(t)\bar d(s+t)\le\bar d(s)\bar d(t)dˉ(s+t)≤dˉ(s)dˉ(t) (Lemma 4.12); the standard mixing-time consequences d(ℓ tmix(ε))≤(2ε)ℓd(\ell\, t_{\mathrm{mix}}(\varepsilon))\le(2\varepsilon)^\elld(ℓtmix​(ε))≤(2ε)ℓ and tmix(ε)≤⌈log⁡2ε−1⌉ tmixt_{\mathrm{mix}}(\varepsilon)\le\lceil\log_2\varepsilon^{-1}\rceil\, t_{\mathrm{mix}}tmix​(ε)≤⌈log2​ε−1⌉tmix​ (§4.5); and the equality of distance to stationarity for a group walk and its inverse walk (Lemma 4.13 and Corollary 4.14).

Significance

The results. The Convergence Theorem is the qualitative foundation on which quantitative mixing theory stands: it guarantees that tmix(ε)t_{\mathrm{mix}}(\varepsilon)tmix​(ε) is finite, so every bound in Missions III–XIII is a bound on a well-defined quantity. The TV characterizations are used constantly — the coupling characterization is the engine of Mission III, the half-ℓ1\ell^1ℓ1 formula of every explicit computation. The Metropolis and Glauber stationarity results justify the chains analyzed in Missions III (colorings, hardcore), VIII (path coupling) and IX (Ising). Submultiplicativity of dˉ\bar ddˉ is what makes tmixt_{\mathrm{mix}}tmix​ a meaningful single number.

Formalizing them. None of this exists in Mathlib: there is no total variation distance for finitely supported distributions, no coupling theory, no mixing time, no MCMC correctness statement. The definition layer published here (TV distance, ddd, dˉ\bar ddˉ, tmixt_{\mathrm{mix}}tmix​, couplings, Metropolis, Glauber) is imported by every subsequent mission of the series.

Difficulty

The tempting proof of Theorem 4.9 via spectral decomposition fails twice: it needs reversibility, which the theorem does not assume, and spectral machinery that arrives only in Mission VII. The book's proof is the Doeblin decomposition: by Proposition 1.7 some power satisfies Pr(x,y)≥δπ(y)P^r(x,y)\ge\delta\pi(y)Pr(x,y)≥δπ(y), so Pr=(1−θ)Π+θQP^r=(1-\theta)\Pi+\theta QPr=(1−θ)Π+θQ with Π\PiΠ the rank-one matrix of rows π\piπ, and induction gives Prk=(1−θk)Π+θkQkP^{rk}=(1-\theta^k)\Pi+\theta^kQ^kPrk=(1−θk)Π+θkQk. The formal work is matrix algebra with careful bookkeeping of the remainder chain QQQ, plus the monotonicity of ddd needed to interpolate between multiples of rrr. For Proposition 4.7 the delicate half is constructing the optimal coupling: mass μ∧ν\mu\wedge\nuμ∧ν on the diagonal and the normalized product of the positive parts off it, with the degenerate case μ=ν\mu=\nuμ=ν handled separately. The Glauber stationarity statement must be phrased with care because configurations outside the support of π\piπ have junk rows; the formalization asserts stochasticity only at supported configurations, and detailed balance globally.

Formalization scope

Total variation distance is defined as the supremum over events, ⨆A ∣μ(A)−ν(A)∣\bigsqcup_{A}\,|\mu(A)-\nu(A)|⨆A​∣μ(A)−ν(A)∣ over Finset V, exactly as in (4.1); the half-ℓ1\ell^1ℓ1 formula is a milestone, not the definition. The mixing time is sInf of the set {t:d(t)≤ε}\{t : d(t)\le\varepsilon\}{t:d(t)≤ε} in N\mathbb NN (junk value 000 if empty — impossible under the goal theorem). Couplings are distributions on the product with prescribed marginals; no probability-space machinery is used. The mixing-time inequalities are stated with the integer-rounding slack made explicit (e.g. ⌈log⁡2ε−1⌉\lceil\log_2\varepsilon^{-1}\rceil⌈log2​ε−1⌉ via Nat.ceil of a real logarithm) so that no statement is true only "up to rounding". The Metropolis definitions use total real division, so the hypotheses require π>0\pi>0π>0 pointwise; this matches the book, which divides by π(x)\pi(x)π(x) throughout.

Welcome contributions beyond the milestones: simp lemmas for tvDist, monotonicity of ddd and dˉ\bar ddˉ in ttt, and triangle-inequality infrastructure — all reused by Missions III–XIII.

Selected references

  • D. A. Levin, Y. Peres, E. L. Wilmer, Markov Chains and Mixing Times, American Mathematical Society, 2009. https://documents.epfl.ch/groups/i/ip/ipg/www/2013-2014/Random_Walks/markovmixing.pdf
  • N. Metropolis, A. Rosenbluth, M. Rosenbluth, A. Teller, E. Teller, Equation of state calculations by fast computing machines, J. Chem. Phys. 21 (1953). https://doi.org/10.1063/1.1699114
  • W. Doeblin, Exposé de la théorie des chaînes simples constantes de Markov à un nombre fini d'états, Rev. Math. Union Interbalkan. 2 (1938).
14 thms2 active usersReviewed
Linear OptimizationOperations ResearchProbability+1·Captain: mikedeng1

Dynamic Matching: Characterizing and Achieving Constant Regret I: Under General Position the Exhaustive Resolving Policy Has Regret at Most Γ/ϵ at All TimesResearch Paper

Motivation

Centralized matching markets (kidney exchange, ride hailing, freight and labor platforms) repeatedly decide which waiting participants to match now and which to hold for a better opportunity later. Matching early collects value sooner; waiting lets the platform see more arrivals and avoid matches that later turn out to be wasteful. A natural yardstick is the best set of matches that could have been made with hindsight from all arrivals up to a given time, and the question is how far an online policy, which cannot see the future, must fall behind that yardstick.

Kerimov, Ashlagi and Gurvich (SSRN 3824407, September 2022; Management Science 2023, doi:10.1287/mnsc.2021.01215) answer this for multi-way matching networks with patient agents. They identify a single parameter of the fluid relaxation, the general position gap ϵ\epsilonϵ, and show that the regret against the hindsight benchmark can be kept at order ϵ−1\epsilon^{-1}ϵ−1 at all times, and that no policy does better in order on non-trivial networks. This mission formalizes the upper bound. The paper places itself next to constant-regret results for online packing (Bumpensanti and Wang 2020; Vera and Banerjee 2021), where resolving a linear program at each arrival achieves bounded regret.

Setting

There are nnn agent types and ddd matches. Match mmm is a set A(m)\mathcal A(m)A(m) of at least two types, every type belongs to some match, and the matching matrix M∈{0,1}n×dM\in\{0,1\}^{n\times d}M∈{0,1}n×d has Mim=1M_{im}=1Mim​=1 iff i∈A(m)i\in\mathcal A(m)i∈A(m). Performing match mmm uses one agent of each type in A(m)\mathcal A(m)A(m) and earns rm>0r_m>0rm​>0. Time is discrete; at each time t=1,2,…t=1,2,\dotst=1,2,… one agent arrives, of type iii with probability λi>0\lambda_i>0λi​>0, ∑iλi=1\sum_i\lambda_i=1∑i​λi​=1, independently over time. AitA^t_iAit​ counts the type-iii arrivals by time ttt.

A matching policy records DmtD^t_mDmt​, the number of times match mmm has been performed by time ttt. It is admissible when it is non-anticipative and only matches agents present: MDt≤AtMD^t\le A^tMDt≤At. The pre-match queue is Qt=At−MDt−1Q^t=A^t-MD^{t-1}Qt=At−MDt−1, the value collected is RD,t=E[r⋅Dt]\mathcal R^{D,t}=\mathbb E[r\cdot D^t]RD,t=E[r⋅Dt], and the hindsight benchmark is

R∗,t=E[max⁡{r⋅y:My≤At, y∈Z≥0d}].\mathcal R^{*,t}=\mathbb E\big[\max\{r\cdot y: My\le A^t,\ y\in\mathbb Z^d_{\ge0}\}\big].R∗,t=E[max{r⋅y:My≤At, y∈Z≥0d​}].

The static-planning problem is the linear program

(SPP)max⁡ r⋅zs.t.Mz+s=λ, z≥0, s≥0.\text{(SPP)}\qquad \max\ r\cdot z\quad\text{s.t.}\quad Mz+s=\lambda,\ z\ge0,\ s\ge0.(SPP)max r⋅zs.t.Mz+s=λ, z≥0, s≥0.

For an optimal (z∗,s∗)(z^*,s^*)(z∗,s∗) let M+={m:zm∗>0}\mathcal M_+=\{m:z^*_m>0\}M+​={m:zm∗​>0} (active matches), M0\mathcal M_0M0​ the rest (redundant matches), Q+={j:sj∗>0}\mathcal Q_+=\{j:s^*_j>0\}Q+​={j:sj∗​>0} (under-demanded types) and Q0\mathcal Q_0Q0​ the rest (over-demanded types). The network satisfies general position (GP) if (SPP) has a unique optimal solution and it is non-degenerate, ∣M+∣+∣Q+∣=n|\mathcal M_+|+|\mathcal Q_+|=n∣M+​∣+∣Q+​∣=n. The general position gap is the smallest positive basic value,

ϵ=min⁡m∈M+zm∗ ∧ min⁡j∈Q+sj∗.\epsilon=\min_{m\in\mathcal M_+}z^*_m\ \wedge\ \min_{j\in\mathcal Q_+}s^*_j .ϵ=m∈M+​min​zm∗​ ∧ j∈Q+​min​sj∗​.

The exhaustive resolving policy with inter-action delay τ\tauτ acts only at the epochs tk=kτt_k=k\tautk​=kτ. There it performs an optimal solution of the integer program max⁡{r⋅z:Mz≤Qtk, z∈Z≥0d}\max\{r\cdot z: Mz\le Q^{t_k},\ z\in\mathbb Z^d_{\ge0}\}max{r⋅z:Mz≤Qtk​, z∈Z≥0d​} with the redundant matches removed, and then discards the leftover agents of the under-demanded types.

Formalization targets

Goal: Theorem 3.1 (Upper bound), p. 14

There are constants κ,Γ>0\kappa,\Gamma>0κ,Γ>0 depending only on n,d,M,rn,d,M,rn,d,M,r such that, under GP, the exhaustive resolving policy with τ=⌈κϵ−1⌉\tau=\lceil\kappa\epsilon^{-1}\rceilτ=⌈κϵ−1⌉ (any tie-breaking) is admissible and

R∗,t−RD,t≤Γ ϵ−1for all t≥1.\mathcal R^{*,t}-\mathcal R^{D,t}\le\Gamma\,\epsilon^{-1}\qquad\text{for all }t\ge1 .R∗,t−RD,t≤Γϵ−1for all t≥1.

The constants are not fixed, so the goal survives any improvement of them.

Milestones

  1. §2, p. 8: R∗,t=max⁡DRD,t\mathcal R^{*,t}=\max_{D}\mathcal R^{D,t}R∗,t=maxD​RD,t, attained by the ultimate batching policy.
  2. §4.2, (4), p. 20: under GP the optimal basis B\mathcal BB is invertible, (zM+∗,sQ+∗)=B−1λ(z^*_{\mathcal M_+},s^*_{\mathcal Q_+})=\mathcal B^{-1}\lambda(zM+​∗​,sQ+​∗​)=B−1λ, and the entries of B−1\mathcal B^{-1}B−1 are bounded by a constant of the topology.
  3. §4.2, p. 20: θ=∑m∈M+rmym\theta=\sum_{m\in\mathcal M_+}r_my^mθ=∑m∈M+​​rm​ym is the optimal dual of (SPP), θ≥0\theta\ge0θ≥0, and θi>0\theta_i>0θi​>0 on Q0\mathcal Q_0Q0​.
  4. Lemma 4.1, p. 19 (optimality test): a policy without redundant matches whose over-demanded queues have mean O(ϵ−1)O(\epsilon^{-1})O(ϵ−1) has regret O(ϵ−1)O(\epsilon^{-1})O(ϵ−1).
  5. Lemma 4.2, p. 21: a Foster–Lyapunov drift condition bounds stationary means (Glynn and Zeevi 2008).
  6. (11), p. 23: Eπ[h∗(Q0)]=O(ϵ−1)\mathbb E_\pi[h^*(Q^0)]=O(\epsilon^{-1})Eπ​[h∗(Q0)]=O(ϵ−1) for the stationary law of the epoch chain.
  7. App. D, p. 49: E0[h∗(Qtk)]=O(ϵ−1)\mathbb E_{\mathbf 0}[h^*(Q^{t_k})]=O(\epsilon^{-1})E0​[h∗(Qtk​)]=O(ϵ−1) for all k≥1k\ge1k≥1.

Here h∗(u)h^*(u)h∗(u) is the θ\thetaθ-weighted leftover in Q0\mathcal Q_0Q0​ after an optimal integer matching of uuu (display (7)).

Significance

The upper bound says that, under GP, hindsight-optimal performance up to an additive constant is achievable at every time simultaneously, not just at a known horizon. Long-run average optimality and near-optimality under discounting follow (Remark 2.1). The bound also quantifies the cost of acting often: the delay τ\tauτ and the regret both scale with ϵ−1\epsilon^{-1}ϵ−1, so networks close to degeneracy force batching. Combined with the paper's lower bound, ϵ−1\epsilon^{-1}ϵ−1 is the right order on non-trivial networks.

The result has a printed proof (§4, Appendices B and D) but no machine-checked one. Several intermediate statements of the printed proof do not hold as stated and are not posed here. The goal and the stated displays (11) and App. D are posed as the paper states them, so a formal proof requires a correct argument for the uniform-in-λ\lambdaλ bound on the over-demanded queues. Lemma 4.1, the basis and duality facts and Lemma 4.2 are self-contained and checkable on their own.

Difficulty

The obvious approach compares the policy's realized matches with the fluid optimum t z∗t\,z^*tz∗ and controls the deviation with concentration bounds. That gives errors of order t\sqrt tt​, not a constant. The constant comes from showing that the over-demanded queues stay of order ϵ−1\epsilon^{-1}ϵ−1 uniformly in time and in λ\lambdaλ. Between epochs these queues behave like a multidimensional reflected random walk whose drift toward zero is only of order ϵτ\epsilon\tauϵτ, while its fluctuations over an epoch are of order τ\sqrt\tauτ​. The delay τ≍ϵ−1\tau\asymp\epsilon^{-1}τ≍ϵ−1 balances the two, but the queue state enters through the integer program (3), whose solution changes discontinuously with the queues, and the quantities that measure the drift are signed combinations of arrival counts, to which the usual one-sided concentration bounds for nonnegative sums do not apply. On top of this every constant must be independent of λ\lambdaλ.

Formalization scope

  • Types are Fin n and matches Fin d (0-based); the topology with its two standing assumptions (∣A(m)∣≥2|\mathcal A(m)|\ge2∣A(m)∣≥2, every type in a match) is a structure, and λ>0\lambda>0λ>0 with ∑λi=1\sum\lambda_i=1∑λi​=1 and r>0r>0r>0 are hypotheses.
  • Time is N\mathbb NN; "for all t>0t>0t>0" is t≥1t\ge1t≥1. The arrival at time s+1s+1s+1 is the sss-th entry of a path ω:N→\omega:\mathbb N\toω:N→ Fin n. Expectations at horizon ttt are finite sums over the first ttt arrivals weighted by ∏λωs\prod\lambda_{\omega_s}∏λωs​​, so no expectation is a default value.
  • Admissible policies are non-anticipative cumulative counts with MDt≤AtMD^t\le A^tMDt≤At and no rejections. The resolving policy's removal of under-demanded leftovers acts on its own virtual queue, which agrees with the actual queue on Q0\mathcal Q_0Q0​; its matches are feasible without removals, so it is admissible.
  • (3) is solved with the redundant matches set to zero, and statements quantify over every tie-breaking selection. A selection exists (a sanity file exhibits one), so these statements are not vacuous.
  • GP is a hypothesis on an explicit pair (z∗,s∗)(z^*,s^*)(z∗,s∗); ϵ\epsilonϵ is the minimum of the positive basic values. The basis inverse is never a matrix-inverse function: statements assert the existence of a two-sided inverse.
  • h∗h^*h∗ is defined in the form min⁡{∑Q0θisi:Mz+s=u, zM0=0}\min\{\sum_{\mathcal Q_0}\theta_is_i: Mz+s=u,\ z_{\mathcal M_0}=0\}min{∑Q0​​θi​si​:Mz+s=u, zM0​​=0}, equivalent to (7) under the paper's reduction M0=∅\mathcal M_0=\emptysetM0​=∅; θ\thetaθ is any solution of the complementary-slackness system, which is unique under GP.
  • Every O(⋅)O(\cdot)O(⋅) is an explicit constant, quantified after the topology and rrr and before λ\lambdaλ. Placing it after λ\lambdaλ would turn the theorem into a per-instance boundedness statement and is ruled out.
  • A trivializing formalization (some unspecified policy, a τ\tauτ independent of ϵ\epsilonϵ, or GP replaced by an unsatisfiable condition) is ruled out: the policy is constructed, τ=⌈κ/ϵ⌉\tau=\lceil\kappa/\epsilon\rceilτ=⌈κ/ϵ⌉, and a sanity file proves GP for the Figure 1 network with ϵ=1/5\epsilon=1/5ϵ=1/5.
  • Not posed: the lower bound of Theorem 3.1, Proposition 4.1 (8)–(9), Lemmas B.1–B.2, Proposition B.1 and displays (21)–(22), which do not hold as printed; Theorem 5.2 and §6.

Contributions welcome: proofs of the LP facts (milestones 2–3) and Lemma 4.1, a Mathlib-ready proof of Lemma 4.2, and any correct argument for the epoch bounds.

Selected references

  • S. Kerimov, I. Ashlagi, I. Gurvich, Dynamic Matching: Characterizing and Achieving Constant Regret, SSRN 3824407 (version of Sep 14, 2022); Management Science, 2023. https://ssrn.com/abstract=3824407, https://doi.org/10.1287/mnsc.2021.01215
  • P. W. Glynn, A. Zeevi, Bounding stationary expectations of Markov processes, in Markov Processes and Related Topics, IMS Collections 4, 2008. https://doi.org/10.1214/074921708000000381
  • P. Bumpensanti, H. Wang, A re-solving heuristic with uniformly bounded loss for network revenue management, Management Science 66, 2020. https://doi.org/10.1287/mnsc.2019.3365
  • D. Bertsimas, J. N. Tsitsiklis, Introduction to Linear Optimization, Athena Scientific, 1997.
14 thms1 active userReviewed
Numerical AnalysisProbabilityStatistics·Captain: mikedeng1

Multilevel Langevin Pathwise Average for Gibbs Approximation III: C² Strongly Contractive Drift, 𝔞 = 2 — L²-Error ≲ [f]₁ε at Cost ((½+√2)/(√2−1))Υ₂²log(γ₀⁻¹)ε⁻²/(γ₀α)Research Paper

Motivation

Sampling from a Gibbs distribution πU∝e−U dλd\pi_U\propto e^{-U}\,d\lambda_dπU​∝e−Udλd​ on Rd\mathbb R^dRd is a basic task in Bayesian statistics, statistical physics and machine learning. A standard approach simulates the Langevin diffusion dXt=−∇U(Xt) dt+2 dBtdX_t=-\nabla U(X_t)\,dt+\sqrt2\,dB_tdXt​=−∇U(Xt​)dt+2​dBt​, whose invariant distribution is πU\pi_UπU​, through its Euler scheme, and averages a test function fff along one long trajectory. The discretization introduces a bias of the order of the step size, so a single-step-size average with accuracy ε\varepsilonε costs more than ε−2\varepsilon^{-2}ε−2 steps. Dalalyan (arXiv:1412.7392) and Durmus–Moulines (arXiv:1605.01559) quantified this for strongly log-concave targets; for occupation measures of the Euler scheme the best previous bound was of order dε−4d\varepsilon^{-4}dε−4 (Durmus–Moulines, Bernoulli 2019, denoted [DM19] in the paper).

Egéa and Panloup (arXiv:2109.07753, Math. Oper. Res., 2025) combine Giles's multilevel Monte Carlo idea (Oper. Res. 2008) with pathwise averages. Each level is one long trajectory, not many independent samples. This mission formalizes their result for a uniformly strongly contractive, twice differentiable drift. In that case the multilevel occupation measure reaches accuracy ε\varepsilonε at a cost of order dε−2d\varepsilon^{-2}dε−2, the order of an unbiased Monte Carlo method.

Setting

Let b:Rd→Rdb:\mathbb R^d\to\mathbb R^db:Rd→Rd be LLL-Lipschitz and satisfy Assumption (Cα)(\mathbf C_\alpha)(Cα​): ⟨b(y)−b(x),y−x⟩≤−α∣x−y∣2\langle b(y)-b(x),y-x\rangle\le-\alpha|x-y|^2⟨b(y)−b(x),y−x⟩≤−α∣x−y∣2 for all x,yx,yx,y, with α∈(0,L∧1]\alpha\in(0,L\wedge1]α∈(0,L∧1]. Let σ>0\sigma>0σ>0. The additive diffusion is Xtx=x+∫0tb(Xsx) ds+σBtX^x_t=x+\int_0^tb(X^x_s)\,ds+\sigma B_tXtx​=x+∫0t​b(Xsx​)ds+σBt​, driven by a standard ddd-dimensional Brownian motion BBB. Let π\piπ be its invariant distribution. Its Euler scheme with step γ\gammaγ is Xˉ0γ,x=x\bar X^{\gamma,x}_0=xXˉ0γ,x​=x, Xˉ(n+1)γγ,x=Xˉnγγ,x+γb(Xˉnγγ,x)+σ(B(n+1)γ−Bnγ)\bar X^{\gamma,x}_{(n+1)\gamma}=\bar X^{\gamma,x}_{n\gamma}+\gamma b(\bar X^{\gamma,x}_{n\gamma})+\sigma(B_{(n+1)\gamma}-B_{n\gamma})Xˉ(n+1)γγ,x​=Xˉnγγ,x​+γb(Xˉnγγ,x​)+σ(B(n+1)γ​−Bnγ​), with invariant distribution πγ\pi^\gammaπγ. Write s‾η=η⌊s/η⌋\underline s_\eta=\eta\lfloor s/\eta\rfloors​η​=η⌊s/η⌋ and [f]1[f]_1[f]1​ for the Lipschitz constant of fff.

Given a first step γ0\gamma_0γ0​, steps γr=γ02−r\gamma_r=\gamma_02^{-r}γr​=γ0​2−r, horizons T0>⋯>TRT_0>\dots>T_RT0​>⋯>TR​, a warm start τ\tauτ, and independent Brownian motions B(0),…,B(R)B^{(0)},\dots,B^{(R)}B(0),…,B(R), the multilevel occupation measure (7) is

Y(f)=1T0−τ∫τT0f(Xˉs‾γ0γ0)ds+∑r=1R1Tr−τ∫τTr(f(Xˉs‾γr−1γr)−f(Xˉs‾γr−1γr−1))ds,\mathcal Y(f)=\frac1{T_0-\tau}\int_\tau^{T_0}f(\bar X^{\gamma_0}_{\underline s_{\gamma_0}})ds+\sum_{r=1}^R\frac1{T_r-\tau}\int_\tau^{T_r}\big(f(\bar X^{\gamma_r}_{\underline s_{\gamma_{r-1}}})-f(\bar X^{\gamma_{r-1}}_{\underline s_{\gamma_{r-1}}})\big)ds,Y(f)=T0​−τ1​∫τT0​​f(Xˉs​γ0​​γ0​​)ds+r=1∑R​Tr​−τ1​∫τTr​​(f(Xˉs​γr−1​​γr​​)−f(Xˉs​γr−1​​γr−1​​))ds,

where both schemes of level rrr start at x0x_0x0​ and are driven by B(r)B^{(r)}B(r). Its cost is C(Y)=T0/γ0+∑r=1R(Tr/γr+Tr/γr−1)\mathcal C(\mathcal Y)=T_0/\gamma_0+\sum_{r=1}^R(T_r/\gamma_r+T_r/\gamma_{r-1})C(Y)=T0​/γ0​+∑r=1R​(Tr​/γr​+Tr​/γr−1​).

When bbb is C2\mathcal C^2C2 with bounded derivatives, set ∥∇b∥2,∞=sup⁡x∥∇b(x)∥F\|\nabla b\|_{2,\infty}=\sup_x\|\nabla b(x)\|_F∥∇b∥2,∞​=supx​∥∇b(x)∥F​ (Frobenius norm of the Jacobian) and ∥Δb∥2,∞2=sup⁡x∑i∣Δbi(x)∣2\|\Delta b\|^2_{2,\infty}=\sup_x\sum_i|\Delta b_i(x)|^2∥Δb∥2,∞2​=supx​∑i​∣Δbi​(x)∣2. The constant (19) is

Υ22=max⁡(1, ∣b(x0)∣2α2+σαL3∥∇b∥2,∞∣b(x0)∣+σ4L4∥Δb∥2,∞2+σ2αdL3∥∇b∥2,∞+σ2dα).\Upsilon_2^2=\max\Big(1,\ \frac{|b(x_0)|^2}{\alpha^2}+\frac{\sigma\sqrt\alpha}{L^3}\|\nabla b\|_{2,\infty}|b(x_0)|+\frac{\sigma^4}{L^4}\|\Delta b\|^2_{2,\infty}+\frac{\sigma^2\alpha\sqrt d}{L^3}\|\nabla b\|_{2,\infty}+\frac{\sigma^2d}{\alpha}\Big).Υ22​=max(1, α2∣b(x0​)∣2​+L3σα​​∥∇b∥2,∞​∣b(x0​)∣+L4σ4​∥Δb∥2,∞2​+L3σ2αd​​∥∇b∥2,∞​+ασ2d​).

Formalization targets

Goal: Theorem 2.3

Take γ0∈(0,α2L2∧12]\gamma_0\in(0,\frac\alpha{2L^2}\wedge\frac12]γ0​∈(0,2L2α​∧21​], Rε=⌈log⁡2(Υ2/ε)⌉R_\varepsilon=\lceil\log_2(\Upsilon_2/\varepsilon)\rceilRε​=⌈log2​(Υ2​/ε)⌉, Tr=Υ22log⁡(γ0−1)α−1ε−22−3r/2T_r=\Upsilon_2^2\log(\gamma_0^{-1})\alpha^{-1}\varepsilon^{-2}2^{-3r/2}Tr​=Υ22​log(γ0−1​)α−1ε−22−3r/2 and τ=α−1∣log⁡ε∣+(2α)−1log⁡Υ2\tau=\alpha^{-1}|\log\varepsilon|+(2\alpha)^{-1}\log\Upsilon_2τ=α−1∣logε∣+(2α)−1logΥ2​. There is a universal constant C1\mathfrak C_1C1​ such that for every ε∈(0,1)\varepsilon\in(0,1)ε∈(0,1) with 2log⁡(Υ2ε−2)≤log⁡(γ0−1)(Υ2/ε)1/2\sqrt2\log(\Upsilon_2\varepsilon^{-2})\le\log(\gamma_0^{-1})(\Upsilon_2/\varepsilon)^{1/2}2​log(Υ2​ε−2)≤log(γ0−1​)(Υ2​/ε)1/2,

∥Y(f)−π(f)∥2≤C1[f]1ε,C(Y)≤12+22−1 Υ22log⁡(γ0−1)γ0α ε−2.\|\mathcal Y(f)-\pi(f)\|_2\le\mathfrak C_1[f]_1\varepsilon,\qquad\mathcal C(\mathcal Y)\le\frac{\frac12+\sqrt2}{\sqrt2-1}\,\frac{\Upsilon_2^2\log(\gamma_0^{-1})}{\gamma_0\alpha}\,\varepsilon^{-2}.∥Y(f)−π(f)∥2​≤C1​[f]1​ε,C(Y)≤2​−121​+2​​γ0​αΥ22​log(γ0−1​)​ε−2.

The constant C1\mathfrak C_1C1​ is the same for all ddd, bbb, σ\sigmaσ, fff, x0x_0x0​, γ0\gamma_0γ0​ and ε\varepsilonε.

Companion: (22)

Under α/L2≤1\alpha/L^2\le1α/L2≤1, σ2α−1d≥1\sigma^2\alpha^{-1}d\ge1σ2α−1d≥1 and dimension-free bounds on ∣b(x0)∣|b(x_0)|∣b(x0​)∣, ∥∇b∥2,∞\|\nabla b\|_{2,\infty}∥∇b∥2,∞​ and ∥Δb∥2,∞\|\Delta b\|_{2,\infty}∥Δb∥2,∞​, the choice γ0=α/(2L2)\gamma_0=\alpha/(2L^2)γ0​=α/(2L2), Υ~22=σ2α−1d\tilde\Upsilon_2^2=\sigma^2\alpha^{-1}dΥ~22​=σ2α−1d gives C(Y)≤22+12−1σ2L2α3log⁡2L2α dε−2\mathcal C(\mathcal Y)\le\frac{2\sqrt2+1}{\sqrt2-1}\frac{\sigma^2L^2}{\alpha^3}\log\frac{2L^2}{\alpha}\,d\varepsilon^{-2}C(Y)≤2​−122​+1​α3σ2L2​logα2L2​dε−2.

Milestones

The milestones follow the paper's proof in order:

  • Lemma 5.1 (i), (iii): stability of the scheme and the assumptions (H1)(\mathbf H_1)(H1​), (H4)(\mathbf H_4)(H4​).
  • Lemma 5.2 (ii): a one-step error of order γ3\gamma^3γ3.
  • Proposition 5.1 (ii), (iii): a uniform-in-time strong error β2(x)γ2\beta_2(x)\gamma^2β2​(x)γ2 and (H2)(\mathbf H_2)(H2​) with a=2\mathfrak a=2a=2.
  • Proposition 5.2: W1(π,πγ)≤β2(x⋆)γ\mathcal W_1(\pi,\pi^\gamma)\le\sqrt{\beta_2(x^\star)}\gammaW1​(π,πγ)≤β2​(x⋆)​γ.
  • Proposition 2.2: (H2)(\mathbf H_2)(H2​)–(H4)(\mathbf H_4)(H4​) with constants controlled by Υ2\Upsilon_2Υ2​.
  • Remark 4.1: the paper's general complexity theorem, with constants known up to universal factors.

Significance

The result. Theorem 2.3 gives an explicit algorithm whose cost to reach L2L^2L2 accuracy ε\varepsilonε on π(f)\pi(f)π(f) is of order ε−2\varepsilon^{-2}ε−2, uniformly over Lipschitz fff. Under (22) the cost is O(dε−2)O(d\varepsilon^{-2})O(dε−2). Apart from constants depending on α\alphaα and LLL, this is the complexity of a Monte Carlo method without bias, an improvement over the dε−4d\varepsilon^{-4}dε−4 known for single-level occupation measures. Corollary 2.2 of the paper applies it to Gibbs distributions with a C3\mathcal C^3C3 potential.

Formalizing it. The theorem is proved in the paper. A machine-checked version needs a formal multilevel estimator over infinite horizons, uniform-in-time strong error bounds for the Euler scheme, and an application of Itô's formula to b(y+σBt)b(y+\sigma B_t)b(y+σBt​), none of which exists in Lean. Formalizing also checks the paper's constants. For instance, the printed exponential rate in the W2\mathcal W_2W2​ contraction behind (H1)(\mathbf H_1)(H1​) is α\alphaα, while its proof gives α/2\alpha/2α/2, and the milestone states α/2\alpha/2α/2. No part of this result has been machine-checked.

Difficulty

Two steps resist a direct approach. First, the order-one strong error (Lemma 5.2(ii), Proposition 5.1(ii)). Bounding the one-step error by the Lipschitz constant alone gives order γ2\gamma^2γ2 per step, hence order γ1/2\gamma^{1/2}γ1/2 globally; this is Lemma 5.2(i) and the a=1\mathfrak a=1a=1 regime, with an extra log⁡3ε\log^3\varepsilonlog3ε in the cost. The gain to γ3\gamma^3γ3 needs the second-order structure of bbb along the Brownian path, through a martingale term and the Laplacian of bbb. Second, the error must be controlled uniformly over infinite horizons. Every estimate has to combine the contraction (Cα)(\mathbf C_\alpha)(Cα​) with the step-size restriction γ≤α/(2L2)\gamma\le\alpha/(2L^2)γ≤α/(2L2); a Gronwall argument on a finite horizon gives constants that grow with time.

For the goal itself, the cost bound (21) is arithmetic once the parameters are fixed. The content is the error bound with a constant that is universal, not merely independent of ε\varepsilonε.

Formalization scope

  • Rd\mathbb R^dRd is the published EthierKurtz.SDEState d, and Brownian motions are the published EthierKurtz.IsStandardBrownian.
  • The diffusion is a pathwise solution of Xt=x+∫0tb(Xs)ds+σWtX_t=x+\int_0^tb(X_s)ds+\sigma W_tXt​=x+∫0t​b(Xs​)ds+σWt​ with continuous paths. No stochastic integral appears in any statement.
  • π\piπ is a probability measure invariant for the diffusion's transition laws. πγ\pi^\gammaπγ is any probability invariant for the Euler step x↦x+γb(x)+σγZx\mapsto x+\gamma b(x)+\sigma\sqrt\gamma Zx↦x+γb(x)+σγ​Z, with ZZZ standard Gaussian.
  • ∥⋅∥2\|\cdot\|_2∥⋅∥2​ is eLpNorm · 2 P, valued in [0,∞][0,\infty][0,∞]. The schemes are evaluated at grid times, except in (H4)(\mathbf H_4)(H4​) and (47), which use the continuous-time scheme.
  • C2\mathcal C^2C2 with bounded partial derivatives is ContDiff ℝ 2 b plus a uniform bound on the first and second Fréchet derivatives.
  • ∥∇b∥2,∞\|\nabla b\|_{2,\infty}∥∇b∥2,∞​ and ∥Δb∥2,∞2\|\Delta b\|^2_{2,\infty}∥Δb∥2,∞2​ follow pp. 31–32, which correct the misplaced squares of (18).
  • "≲uc\lesssim_{uc}≲uc​" is an existential constant quantified before every datum.
  • The ε0\varepsilon_0ε0​ of Theorem 2.3 is read as membership in its defining set.
  • The cost formula is used for every TrT_rTr​.
  • Added hypotheses: σ>0\sigma>0σ>0 wherever the page leaves it implicit; C2\mathcal C^2C2 regularity in Lemma 5.2(ii) and Propositions 5.1–5.2; γ<1\gamma<1γ<1 in Proposition 5.2; d≥1d\ge1d≥1 in Proposition 5.1(iii); α≤1\alpha\le1α≤1 in Proposition 2.2.

The goal does not assume (H1)(\mathbf H_1)(H1​)–(H4)(\mathbf H_4)(H4​), Proposition 2.2 or any variance or bias estimate. It assumes only (Cα)(\mathbf C_\alpha)(Cα​), Lipschitz and C2\mathcal C^2C2 regularity of bbb, σ>0\sigma>0σ>0, invariance of π\piπ and the parameter choices. Those results appear only as milestones.

Contributions that would be reusable elsewhere: uniform-in-time moment and strong-error bounds for Euler schemes of contractive SDEs, Itô's formula for a C2\mathcal C^2C2 function of Brownian motion, and the general multilevel complexity theorem (Remark 4.1). Proofs of the milestones are welcome independently.

Selected references

  • M. Egéa, F. Panloup, Multilevel-Langevin pathwise average for Gibbs approximation, Math. Oper. Res., 2025; arXiv:2109.07753v2. https://arxiv.org/abs/2109.07753
  • M. B. Giles, Multilevel Monte Carlo path simulation, Oper. Res. 56(3), 2008. https://doi.org/10.1287/opre.1070.0496
  • A. Durmus, É. Moulines, High-dimensional Bayesian inference via the unadjusted Langevin algorithm, Bernoulli 25(4A), 2019. https://doi.org/10.3150/18-BEJ1073
  • A. S. Dalalyan, Theoretical guarantees for approximate sampling from smooth and log-concave densities, J. R. Stat. Soc. B 79(3), 2017. https://arxiv.org/abs/1412.7392
  • A. Durmus, É. Moulines, Nonasymptotic convergence analysis for the unadjusted Langevin algorithm, Ann. Appl. Probab. 27(3), 2017. https://arxiv.org/abs/1507.05021
16 thms1 active userReviewed
Graph TheoryOperations ResearchProbability·Captain: mikedeng1

Opinion Dynamics on Directed Complex Networks 1: On Graph Sequences Admitting a Strong Coupling, the Typical Stationary Opinion Converges to the Root Opinion R* of the Local Weak LimitResearch Paper

Opinions on large directed social networks

Linear models of opinion formation go back to DeGroot (DeGroot 1974), in which every individual repeatedly replaces their opinion by a weighted average of the opinions of the people they listen to, and to Friedkin and Johnsen (Friedkin–Johnsen 1990), who added a persistent internal opinion. Fraiman, Lin and Olvera-Cravioto (arXiv:2209.00969v2, Mathematics of Operations Research, 2025) add a random external (media) signal whose law may depend on the individual's attributes, which models selective exposure, and ask what the opinions look like on a large random social network. The answer is the behaviour of a typical individual, a uniformly chosen vertex: its stationary opinion converges in law to the opinion of the root of the network's local weak limit, and the empirical distribution of all stationary opinions converges to the same law. Questions such as consensus or polarization are then questions about one limiting random variable, independent of the particular realization of the network.

This mission formalizes that main result (Theorem 2) together with the fixed-graph stationarity theorem it rests on (Theorem 1).

Setting

A vertex-weighted directed graph on a vertex set VVV gives each vertex iii an in-degree di−d_i^-di−​, an ordered list of inbound neighbours ℓ(i,1),…,ℓ(i,di−)\ell(i,1),\dots,\ell(i,d_i^-)ℓ(i,1),…,ℓ(i,di−​) and weights c(i,r)∈[0,1]c(i,r)\in[0,1]c(i,r)∈[0,1] with ∑rc(i,r)=c\sum_r c(i,r)=c∑r​c(i,r)=c whenever di−>0d_i^->0di−​>0. The parameters satisfy c∈[0,1)c\in[0,1)c∈[0,1), d∈(0,1]d\in(0,1]d∈(0,1], c+d≤1c+d\le1c+d≤1. The opinions Ri(k)∈[−1,1]R_i^{(k)}\in[-1,1]Ri(k)​∈[−1,1] evolve by recursion (1),

Ri(k+1)=∑r=1di−c(i,r) Rℓ(i,r)(k)+Wi(k)+(1−c−d)Ri(k),k≥0,R_i^{(k+1)}=\sum_{r=1}^{d_i^-}c(i,r)\,R^{(k)}_{\ell(i,r)}+W_i^{(k)}+(1-c-d)R_i^{(k)},\qquad k\ge0,Ri(k+1)​=r=1∑di−​​c(i,r)Rℓ(i,r)(k)​+Wi(k)​+(1−c−d)Ri(k)​,k≥0,

where the external signals Wi(k)W_i^{(k)}Wi(k)​ are independent, i.i.d. in time given the vertex mark, and bounded by ∣Wi(k)∣≤d+c−∑rc(i,r)|W_i^{(k)}|\le d+c-\sum_rc(i,r)∣Wi(k)​∣≤d+c−∑r​c(i,r). With the operator Δf(i)=∑rc(i,r)f(ℓ(i,r))+(1−c−d)f(i)\Delta f(i)=\sum_rc(i,r)f(\ell(i,r))+(1-c-d)f(i)Δf(i)=∑r​c(i,r)f(ℓ(i,r))+(1−c−d)f(i) this reads R(k+1)=ΔR(k)+W(k)R^{(k+1)}=\Delta R^{(k)}+W^{(k)}R(k+1)=ΔR(k)+W(k). The stationary vector is R=∑r≥0ΔrW(r)R=\sum_{r\ge0}\Delta^rW^{(r)}R=∑r≥0​ΔrW(r).

The full mark of a vertex is Xi=(Ai,Qi,Si,Di−,C(i,1),C(i,2),… )\mathbf X_i=(\mathbf A_i,Q_i,S_i,D_i^-,C(i,1),C(i,2),\dots)Xi​=(Ai​,Qi​,Si​,Di−​,C(i,1),C(i,2),…): a generic attribute, an internal opinion, a stubbornness label, the in-degree and the weights. Marks are compared by the metric

ρ(x,x~)=ρ′(a,a~)+∣q−q~∣+∣s−s~∣+∣d−−d~−∣+∑j∣cj1(d−≥j)−c~j1(d~−≥j)∣.\rho(\mathbf x,\tilde{\mathbf x})=\rho'(\mathbf a,\tilde{\mathbf a})+|q-\tilde q|+|s-\tilde s|+|d^--\tilde d^-|+\sum_j|c_j\mathbb 1(d^-\ge j)-\tilde c_j\mathbb 1(\tilde d^-\ge j)|.ρ(x,x~)=ρ′(a,a~)+∣q−q~​∣+∣s−s~∣+∣d−−d~−∣+j∑​∣cj​1(d−≥j)−c~j​1(d~−≥j)∣.

A random graph sequence G(Vn,En;An)G(V_n,E_n;\mathscr A_n)G(Vn​,En​;An​) has vertex set Vn={1,…,n}V_n=\{1,\dots,n\}Vn​={1,…,n} and attributes An={Yi=(Ai,Qi,Si)}\mathscr A_n=\{\mathbf Y_i=(\mathbf A_i,Q_i,S_i)\}An​={Yi​=(Ai​,Qi​,Si​)}; edges and weights are random given the attributes. The conditional expectation En[⋅]=E[⋅∣An]E_n[\cdot]=E[\cdot\mid\mathscr A_n]En​[⋅]=E[⋅∣An​] conditions on the attributes only. The in-component of depth kkk of a vertex vvv is the set of vertices with a directed path of length at most kkk to vvv. The sequence admits a strong coupling (Definition 4) with a rooted marked graph G(X)\mathcal G(\boldsymbol{\mathcal X})G(X) if, for two independent uniform vertices In,JnI_n,J_nIn​,Jn​, every depth k≥1k\ge1k≥1 and every ϵ>0\epsilon>0ϵ>0, there are two independent copies of G(X)\mathcal G(\boldsymbol{\mathcal X})G(X) whose depth-kkk root in-components are, with conditional probability tending to one in probability, isomorphic to those of InI_nIn​ and JnJ_nJn​, with all marks ϵ\epsilonϵ-close in ρ\rhoρ and the two in-components of InI_nIn​ and JnJ_nJn​ disjoint. The Wasserstein distance is d1(μ,ν)=inf⁡{E∣X−Y∣:X∼μ,Y∼ν}d_1(\mu,\nu)=\inf\{E|X-Y|:X\sim\mu,Y\sim\nu\}d1​(μ,ν)=inf{E∣X−Y∣:X∼μ,Y∼ν}.

Formalization targets

Goal: Theorem 2

Assume a strong coupling, d>0d>0d>0, and a signal kernel ν(x)\nu(\mathbf x)ν(x) with d1(ν(x),ν(x~))≤Kρ(x,x~)d_1(\nu(\mathbf x),\nu(\tilde{\mathbf x}))\le K\rho(\mathbf x,\tilde{\mathbf x})d1​(ν(x),ν(x~))≤Kρ(x,x~). Then there are laws LkL_kLk​ of (R(0),…,R(k))(\mathcal R^{(0)},\dots,\mathcal R^{(k)})(R(0),…,R(k)) and L∗L^*L∗ of R∗\mathcal R^*R∗, not depending on nnn, with

max⁡0≤r≤kE[∣RIn(r)−R(r)∣∣An]→P0,1n∑i=1nf(Ri(k))→PEf(R(k)),\max_{0\le r\le k}E\bigl[|R^{(r)}_{I_n}-\mathcal R^{(r)}|\bigm|\mathscr A_n\bigr]\xrightarrow{P}0,\qquad \frac1n\sum_{i=1}^nf(R_i^{(k)})\xrightarrow{P}E f(\mathcal R^{(k)}),0≤r≤kmax​E[∣RIn​(r)​−R(r)∣​An​]P​0,n1​i=1∑n​f(Ri(k)​)P​Ef(R(k)), E[∣RIn−R∗∣∣An]→P0,1n∑i=1nf(Ri)→PEf(R∗),E\bigl[|R_{I_n}-\mathcal R^*|\bigm|\mathscr A_n\bigr]\xrightarrow{P}0,\qquad \frac1n\sum_{i=1}^nf(R_i)\xrightarrow{P}E f(\mathcal R^*),E[∣RIn​​−R∗∣​An​]P​0,n1​i=1∑n​f(Ri​)P​Ef(R∗),

for every continuous fff on [−1,1][-1,1][−1,1], and R(k)⇒R∗\mathcal R^{(k)}\Rightarrow\mathcal R^*R(k)⇒R∗ geometrically fast:

d1(law R(k),law R∗)≤(1−d)k(2d+2).d_1\bigl(\mathrm{law}\,\mathcal R^{(k)},\mathrm{law}\,\mathcal R^*\bigr)\le(1-d)^k\Bigl(\frac2d+2\Bigr).d1​(lawR(k),lawR∗)≤(1−d)k(d2​+2).

Milestones

  1. ∥Δ∥∞≤1−d\|\Delta\|_\infty\le1-d∥Δ∥∞​≤1−d (proof of Theorem 1, p. 22).
  2. Theorem 1 (p. 5): on a fixed locally finite graph, R(k)⇒RR^{(k)}\Rightarrow RR(k)⇒R at a geometric rate, with a limit that does not depend on R(0)R^{(0)}R(0).
  3. Theorem 2(1) (p. 8): the fixed-time statement.

A supporting statement of the proposal, not a milestone, is the tail bound ∥R−B(k)∥∞≤2∑r>k(1−d)r\|R-B^{(k)}\|_\infty\le2\sum_{r>k}(1-d)^r∥R−B(k)∥∞​≤2∑r>k​(1−d)r for B(k)=∑r≤kΔrW(r)B^{(k)}=\sum_{r\le k}\Delta^rW^{(r)}B(k)=∑r≤k​ΔrW(r) (proof of Theorem 2, p. 28).

Significance

Theorem 2 reduces the stationary opinion distribution of a large network to a single random variable R∗\mathcal R^*R∗ attached to the local limit. When the limit is a marked Galton–Watson tree, R∗\mathcal R^*R∗ has an explicit series representation whose mean and variance are computed in closed form later in the paper (the other two missions of this series), and these formulas are what explain consensus and polarization. Strong couplings exist ([Olvera-Cravioto 2022], cited on p. 8 of the paper) for Erdős–Rényi graphs, the Chung–Lu and Norros–Reittu models, the configuration model, stochastic block models and preferential attachment graphs, so the theorem applies to most random graph models used for social networks.

The result is proved in the paper; nothing in it has been formalized. On Prove2Me this is the first statement of a strong coupling, a form of local weak convergence for directed marked graphs, and of a stochastic recursion on a random graph. The definitions of in-components, the coupling event and the strong coupling are reusable for any local-limit statement about directed random graphs, and the contraction and time-reversal milestones are reusable for linear stochastic recursions.

Difficulty

The fixed-graph part is a contraction argument, but the coupled object is not the opinion process itself: R(k)R^{(k)}R(k) does not converge almost surely, and only its time reversal B(k)B^{(k)}B(k) does. The main difficulty is Theorem 2. A local limit controls one neighbourhood at a time, while the empirical averages need two uniformly chosen vertices to be asymptotically independent, which a one-vertex limit statement does not give. The errors in weights, signals and initial opinions enter at every vertex of a depth-kkk neighbourhood, and the statements condition only on the attributes, with edges and weights integrated out. The stationary statement then requires an interchange of n→∞n\to\inftyn→∞ and k→∞k\to\inftyk→∞ that is uniform in nnn.

Formalization scope

All random objects of the mmm-th model live on one probability space, and the mmm-th graph has m+1m+1m+1 vertices (the paper's Gm+1G_{m+1}Gm+1​), so a uniform vertex always exists. Signals are written Wi(k)=ζ(Xi,Ui(k))W_i^{(k)}=\zeta(\mathbf X_i,U_i^{(k)})Wi(k)​=ζ(Xi​,Ui(k)​) and initial opinions Ri(0)=φ(Xi,Vi)R_i^{(0)}=\varphi(\mathbf X_i,V_i)Ri(0)​=φ(Xi​,Vi​) with i.i.d. uniforms independent of the graph: this quantile representation is equivalent in law to the paper's conditionally i.i.d. signals. Neighbour indices are 0-based. The conditional quantities given An\mathscr A_nAn​ are regular conditional distributions given the attribute vector, and "random variables whose distribution does not depend on An\mathscr A_nAn​" are encoded by fixed limit laws, with the paper's E[∣⋅∣∣An]→0E[|\cdot|\mid\mathscr A_n]\to0E[∣⋅∣∣An​]→0 read as d1d_1d1​ convergence in probability of the conditional laws, which is equivalent. Theorem 1 is stated on countably many vertices, with its geometric rate given as an explicit bound for 1-Lipschitz test functions of finitely many coordinates.

Disclosed choices: the strong coupling is stated on the given space with measurable events inside the coupling events (the event "an isomorphism exists" need not be measurable), with isomorphisms that preserve the order of inbound neighbours (which the proof uses), and through its second bullet, which implies the first. The paper's conditional independence of weight vectors given An\mathscr A_nAn​ is not imposed. The strong coupling also requires the depth-kkk in-components of InI_nIn​ and JnJ_nJn​ to be disjoint on the coupling event: the proof of Theorem 7 uses this, and without it Definition 4 admits a graph whose vertices all listen to a single hub, whose shared signals keep 1n∑if(Ri(k))\frac1n\sum_if(R_i^{(k)})n1​∑i​f(Ri(k)​) random. The page puts no condition on R(0)R^{(0)}R(0) beyond conditional independence of the signals, but Theorem 2(1) at time 000 is false then (take R(0)≡(−1)nR^{(0)}\equiv(-1)^nR(0)≡(−1)n); the statement assumes a d1d_1d1​-Lipschitz initial kernel with the same KKK, which covers the paper's own simulations. The geometric rate uses the constant of the proof.

Trivializing readings are excluded: every tsum is absolutely summable under the hypotheses, conditional laws are regular conditional distributions of measurable random variables, the averages have no division by zero, and the limit laws are forced onto [−1,1][-1,1][−1,1] by the convergence, so no integral defaults to 000. Theorem 7 of the paper (the two-vertex coupling) is not a milestone. Contributions of proofs of any milestone, and of general lemmas on regular conditional distributions and Wasserstein couplings, are welcome.

Selected references

  • N. Fraiman, T.-C. Lin and M. Olvera-Cravioto, Opinion dynamics on directed complex networks, Mathematics of Operations Research, 2025; preprint arXiv:2209.00969v2, 2024. https://arxiv.org/abs/2209.00969v2
  • M. H. DeGroot, Reaching a consensus, Journal of the American Statistical Association 69 (1974), 118–121. https://doi.org/10.1080/01621459.1974.10480137
  • N. E. Friedkin and E. C. Johnsen, Social influence and opinions, Journal of Mathematical Sociology 15 (1990), 193–206. https://doi.org/10.1080/0022250X.1990.9990069
  • M. Olvera-Cravioto, Strong couplings for static locally tree-like random graphs, Journal of Applied Probability 59(4) (2022), 1261–1285 (reference [33] of the paper, where the random graph models admitting strong couplings are listed).
8 thms1 active userReviewed
Numerical AnalysisProbabilityStatistics·Captain: mikedeng1

Multilevel Langevin Pathwise Average for Gibbs Approximation II: Strongly Contractive Drift, 𝔞 = 1 — L²-Error ≲ [f]₁ε at Cost (5log γ₀⁻¹/2αγ₀)Υ₁²ε⁻²⌈log₂(Υ₁²ε⁻²)⌉³Research Paper

Motivation

Sampling from a Gibbs distribution πU∝e−U dλd\pi_U \propto e^{-U}\,d\lambda_dπU​∝e−Udλd​ on Rd\mathbb R^dRd, or computing integrals π(f)\pi(f)π(f) against it, is a basic task of Bayesian statistics, statistical physics and stochastic optimization. A standard route is to simulate a diffusion whose invariant law is the target, for instance the overdamped Langevin equation dXt=−∇U(Xt) dt+2 dBtdX_t = -\nabla U(X_t)\,dt + \sqrt2\,dB_tdXt​=−∇U(Xt​)dt+2​dBt​, discretize it with the Euler scheme, and average a function along one long trajectory. Discretization introduces a bias of order of the step size, and the long-time average introduces a variance that decays like the inverse of the horizon, so reaching root-mean-square accuracy ε\varepsilonε with a single Euler scheme costs more than ε−2\varepsilon^{-2}ε−2 steps.

Egéa and Panloup (arXiv:2109.07753, Mathematics of Operations Research, 2025) adapt Giles's multilevel Monte Carlo idea (Giles 2008) to this infinite-horizon setting: they combine a coarse pathwise average with corrections between coupled Euler schemes of successively halved steps, run over decreasing horizons. Their general complexity theorem (Theorem 2.1) is stated under four abstract assumptions (H₁)–(H₄). This mission formalizes the first concrete instance the paper gives: an additive diffusion with a strongly contractive drift, where the assumptions are verified with explicit dependence on the dimension and the cost is of order ε−2log⁡3(ε−1)\varepsilon^{-2}\log^3(\varepsilon^{-1})ε−2log3(ε−1).

Setting

Let ∣⋅∣|\cdot|∣⋅∣ and ⟨⋅,⋅⟩\langle\cdot,\cdot\rangle⟨⋅,⋅⟩ be the Euclidean norm and inner product on Rd\mathbb R^dRd. Let b:Rd→Rdb:\mathbb R^d\to\mathbb R^db:Rd→Rd be LLL-Lipschitz and satisfy the contractivity assumption (Cα)(\mathbf C_\alpha)(Cα​):

⟨b(y)−b(x),y−x⟩≤−α∣x−y∣2(x,y∈Rd),\langle b(y)-b(x),y-x\rangle\le-\alpha|x-y|^2\qquad(x,y\in\mathbb R^d),⟨b(y)−b(x),y−x⟩≤−α∣x−y∣2(x,y∈Rd),

with 0<α≤L0<\alpha\le L0<α≤L. For b=−∇Ub=-\nabla Ub=−∇U this is uniform strong convexity of UUU. Let σ>0\sigma>0σ>0 and let WWW be a standard ddd-dimensional Brownian motion. The additive diffusion started at xxx is Xtx=x+∫0tb(Xsx) ds+σWtX^x_t = x+\int_0^t b(X^x_s)\,ds+\sigma W_tXtx​=x+∫0t​b(Xsx​)ds+σWt​; a probability π\piπ is invariant for it if ∫P(Xtx∈A) π(dx)=π(A)\int\mathbb P(X^x_t\in A)\,\pi(dx)=\pi(A)∫P(Xtx​∈A)π(dx)=π(A) for all ttt and Borel AAA.

The Euler scheme with step γ>0\gamma>0γ>0 is Xˉ0γ,x=x\bar X^{\gamma,x}_0=xXˉ0γ,x​=x, Xˉ(n+1)γγ,x=Xˉnγγ,x+γb(Xˉnγγ,x)+σ(W(n+1)γ−Wnγ)\bar X^{\gamma,x}_{(n+1)\gamma}=\bar X^{\gamma,x}_{n\gamma}+\gamma b(\bar X^{\gamma,x}_{n\gamma})+\sigma(W_{(n+1)\gamma}-W_{n\gamma})Xˉ(n+1)γγ,x​=Xˉnγγ,x​+γb(Xˉnγγ,x​)+σ(W(n+1)γ​−Wnγ​), extended to continuous time by Xˉt=x+∫0tb(Xˉs‾) ds+σWt\bar X_t=x+\int_0^t b(\bar X_{\underline s})\,ds+\sigma W_tXˉt​=x+∫0t​b(Xˉs​​)ds+σWt​, s‾=γ⌊s/γ⌋\underline s=\gamma\lfloor s/\gamma\rfloors​=γ⌊s/γ⌋. A probability πγ\pi^\gammaπγ invariant for one step of the scheme is its invariant distribution.

The multilevel occupation measure with start x0x_0x0​, coarse step γ0\gamma_0γ0​, levels r=0,…,Rr=0,\dots,Rr=0,…,R, steps γr=γ02−r\gamma_r=\gamma_02^{-r}γr​=γ0​2−r, horizons TrT_rTr​ and warm-start time τ\tauτ is

Y(f)=1T0−τ∫τT0f(Xˉs‾γ0γ0) ds+∑r=1R1Tr−τ∫τTr(f(Xˉs‾γr−1γr)−f(Xˉs‾γr−1γr−1)) ds,\mathcal Y(f)=\frac1{T_0-\tau}\int_\tau^{T_0}f(\bar X^{\gamma_0}_{\underline s_{\gamma_0}})\,ds+\sum_{r=1}^R\frac1{T_r-\tau}\int_\tau^{T_r}\big(f(\bar X^{\gamma_r}_{\underline s_{\gamma_{r-1}}})-f(\bar X^{\gamma_{r-1}}_{\underline s_{\gamma_{r-1}}})\big)\,ds,Y(f)=T0​−τ1​∫τT0​​f(Xˉs​γ0​​γ0​​)ds+r=1∑R​Tr​−τ1​∫τTr​​(f(Xˉs​γr−1​​γr​​)−f(Xˉs​γr−1​​γr−1​​))ds,

where level rrr uses its own Brownian motion B(r)B^{(r)}B(r), independent across levels, and both schemes of a level share it. Its cost is C(Y)=T0/γ0+∑r=1R(Tr/γr+Tr/γr−1)\mathcal C(\mathcal Y)=T_0/\gamma_0+\sum_{r=1}^R(T_r/\gamma_r+T_r/\gamma_{r-1})C(Y)=T0​/γ0​+∑r=1R​(Tr​/γr​+Tr​/γr−1​), the number of simulated Euler steps. Write [f]1[f]_1[f]1​ for the Lipschitz constant of fff and

Υ12=1∨(α−2∣b(x0)∣2+α−1σ2d).\Upsilon_1^2=1\vee\big(\alpha^{-2}|b(x_0)|^2+\alpha^{-1}\sigma^2d\big).Υ12​=1∨(α−2∣b(x0​)∣2+α−1σ2d).

Formalization targets

Goal: Theorem 2.2

There is a universal constant C1\mathfrak C_1C1​ such that, for α∈(0,L∧1]\alpha\in(0,L\wedge1]α∈(0,L∧1], γ0∈(0,α2L2∧12]\gamma_0\in(0,\frac\alpha{2L^2}\wedge\frac12]γ0​∈(0,2L2α​∧21​], the parameters

r0=Υ1,Rε=⌈2log⁡2(r0ε−1)⌉,Tr=Υ12log⁡(γ0−1)αε−2Rε22−r,τ=2α∣log⁡ε∣+1αlog⁡Υ1,r_0=\Upsilon_1,\quad R_\varepsilon=\lceil2\log_2(r_0\varepsilon^{-1})\rceil,\quad T_r=\frac{\Upsilon_1^2\log(\gamma_0^{-1})}{\alpha}\varepsilon^{-2}R_\varepsilon^22^{-r},\quad \tau=\frac2\alpha|\log\varepsilon|+\frac1\alpha\log\Upsilon_1,r0​=Υ1​,Rε​=⌈2log2​(r0​ε−1)⌉,Tr​=αΥ12​log(γ0−1​)​ε−2Rε2​2−r,τ=α2​∣logε∣+α1​logΥ1​,

and every ε∈(0,1)\varepsilon\in(0,1)ε∈(0,1) with 2log⁡(Υ1ε−2)≤log⁡(γ0−1)Rε22\log(\Upsilon_1\varepsilon^{-2})\le\log(\gamma_0^{-1})R_\varepsilon^22log(Υ1​ε−2)≤log(γ0−1​)Rε2​,

∥Y(f)−π(f)∥2≤C1[f]1ε,C(Y)≤5log⁡(γ0−1)2αγ0Υ12ε−2⌈log⁡2(Υ12ε−2)⌉3.\|\mathcal Y(f)-\pi(f)\|_2\le\mathfrak C_1[f]_1\varepsilon,\qquad\mathcal C(\mathcal Y)\le\frac{5\log(\gamma_0^{-1})}{2\alpha\gamma_0}\Upsilon_1^2\varepsilon^{-2}\lceil\log_2(\Upsilon_1^2\varepsilon^{-2})\rceil^3.∥Y(f)−π(f)∥2​≤C1​[f]1​ε,C(Y)≤2αγ0​5log(γ0−1​)​Υ12​ε−2⌈log2​(Υ12​ε−2)⌉3.

The constant C1\mathfrak C_1C1​ does not depend on ddd, bbb, σ\sigmaσ, fff, x0x_0x0​ or ε\varepsilonε; all dependence on the problem sits in the explicit cost.

Milestones

  1. Lemma 5.1 (i)–(iii): uniform second-moment bound (47), existence and uniqueness of πγ\pi^\gammaπγ, the L2L^2L2 contraction E∣Xˉtγ,x−Xˉtγ,y∣2≤∣x−y∣2e−αt\mathbb E|\bar X^{\gamma,x}_t-\bar X^{\gamma,y}_t|^2\le|x-y|^2e^{-\alpha t}E∣Xˉtγ,x​−Xˉtγ,y​∣2≤∣x−y∣2e−αt, the bound W2(δx,πγ)≤c1(x)\mathcal W_2(\delta_x,\pi^\gamma)\le c_1(x)W2​(δx​,πγ)≤c1​(x), and from these (H₁) and (H₄).
  2. Lemma 5.2 (i): the one-step error between diffusion and Euler step.
  3. Proposition 5.1 (i) and (iii) with a=1\mathfrak a=1a=1: ∥Xnγx−Xˉnγγ,x∥22≤β1(x)γ\|X^x_{n\gamma}-\bar X^{\gamma,x}_{n\gamma}\|_2^2\le\beta_1(x)\gamma∥Xnγx​−Xˉnγγ,x​∥22​≤β1​(x)γ uniformly in nnn, hence (H₂).
  4. Proposition 5.2 with a=1\mathfrak a=1a=1, δ=12\delta=\frac12δ=21​: W1(π,πγ)≤β1(x⋆)γ\mathcal W_1(\pi,\pi^\gamma)\le\sqrt{\beta_1(x^\star)\gamma}W1​(π,πγ)≤β1​(x⋆)γ​, hence (H₃).
  5. Proposition 2.1: all four assumptions with max⁡(αc22/L2,αc32/L2,c42)≲Υ12\max(\alpha c_2^2/L^2,\alpha c_3^2/L^2,c_4^2)\lesssim\Upsilon_1^2max(αc22​/L2,αc32​/L2,c42​)≲Υ12​.
  6. Remark 4.1: Theorem 2.1 (ii) for the additive scheme when the constants are known up to universal factors.
  7. Theorem 2.2 (15): the refinement with Υ~12=σ2α−1d\tilde\Upsilon_1^2=\sigma^2\alpha^{-1}dΥ~12​=σ2α−1d and γ0=α/(2L2)\gamma_0=\alpha/(2L^2)γ0​=α/(2L2).

Significance

The goal gives a complexity of order ε−2log⁡3(ε−1)\varepsilon^{-2}\log^3(\varepsilon^{-1})ε−2log3(ε−1) for approximating π(f)\pi(f)π(f) by a pathwise average of Euler schemes, within a logarithmic factor of the ε−2\varepsilon^{-2}ε−2 of an unbiased Monte Carlo method, with constants explicit in ddd, α\alphaα, LLL, σ\sigmaσ. When ∣b(x0)∣≲d|b(x_0)|\lesssim\sqrt d∣b(x0​)∣≲d​, the cost is of order dε−2(log⁡3d+log⁡3ε)d\varepsilon^{-2}(\log^3d+\log^3\varepsilon)dε−2(log3d+log3ε). The intermediate results (uniform-in-time strong error of the Euler scheme, Wasserstein bias of its invariant law, step-independent contraction) are standard tools of Langevin Monte Carlo and are reusable on their own.

The results are proved in the paper; none of them is machine-checked. A formal proof would certify both the abstract multilevel argument as instantiated here and the explicit constants. The formal statements also pin three printed slips: the ceiling in (14) (cube of the ceiling, not ceiling of the cube), the first line of (12) (the function c1c_1c1​ of Lemma 5.1 (ii)), and the factor in (15).

Difficulty

The obvious argument controls bias and variance separately for each level. The bias of a pathwise average involves the long-time behaviour of the scheme, and the variance of a correcting level involves the L2L^2L2 distance between two coupled schemes over an unbounded horizon: both require estimates uniform in time, which fail for a merely Lipschitz drift (the strong error of the Euler scheme grows exponentially on finite horizons). Contractivity is what makes the strong error O(γ)O(\sqrt\gamma)O(γ​) uniformly in nnn, and the delicate point is to obtain this with constants that do not degrade with ddd or with the step. The complexity bound then requires balancing RεR_\varepsilonRε​, the horizons and the warm start so that the long-time bias, the discretization bias and the variances all sit at level ε\varepsilonε, with a universal constant.

Formalization scope

Rd\mathbb R^dRd is EuclideanSpace ℝ (Fin d) (published EthierKurtz.SDEState); Brownian motions are published EthierKurtz.IsStandardBrownian, one per level, mutually independent as path-valued variables. The diffusion is a pathwise solution of the integral equation with continuous paths; no Itô integral is needed because the noise is additive. Invariance of π\piπ and πγ\pi^\gammaπγ is stated through transition probabilities. L2L^2L2 norms are eLpNorm … 2 in [0,∞][0,\infty][0,∞], so an infinite moment is never read as zero. W1\mathcal W_1W1​ is used in its Kantorovich–Rubinstein form, W2(δx,⋅)\mathcal W_2(\delta_x,\cdot)W2​(δx​,⋅) as an explicit integral, and the clause W2(L(Xˉt‾),πγ)≤W2(δx,πγ)e−αt‾\mathcal W_2(\mathcal L(\bar X_{\underline t}),\pi^\gamma)\le\mathcal W_2(\delta_x,\pi^\gamma)e^{-\alpha\underline t}W2​(L(Xˉt​​),πγ)≤W2​(δx​,πγ)e−αt​ as the existence of a coupling with that quadratic cost (Mathlib has no Wasserstein distance; optimal couplings exist, so the forms agree). (H₁)–(H₃) are stated at grid times, (H₄) over all t≥0t\ge0t≥0. "≲uc\lesssim_{uc}≲uc​" is an existential constant quantified before every datum.

Standing assumptions carried as hypotheses: bbb LLL-Lipschitz, (Cα)(\mathbf C_\alpha)(Cα​), 0<α≤L0<\alpha\le L0<α≤L (and α≤1\alpha\le1α≤1 where the paper requires it), σ>0\sigma>0σ>0. Added and disclosed: the §5.1 hypotheses in Lemma 5.2, γ<1\gamma<1γ<1 in Proposition 5.2 (needed by Proposition 5.1), integrability of fff under π\piπ, existence of πγ\pi^\gammaπγ and c~4>0\tilde c_4>0c~4​>0 in Remark 4.1. The goal's ε0\varepsilon_0ε0​ (a maximum of a set that need not be an interval) is replaced by the inequality defining it.

The goal assumes only the contractive setting, the diffusion's invariance of π\piπ and the parameter choices; it does not assume (H₁)–(H₄), a bias or a variance bound, which appear only as milestones, so it cannot be discharged by restating its hypotheses.

A complete development needs: second-moment and Grönwall estimates for the Euler scheme and the diffusion, existence and uniqueness of invariant laws of Feller chains under a Lyapunov condition, independence and variance decompositions for the multilevel estimator, and the abstract Theorem 2.1. The Gaussian-increment and Wasserstein tools are reusable well beyond this mission. Proofs of any milestone, and a Mathlib-style W2\mathcal W_2W2​ for general laws, are welcome.

Selected references

  • M. Egéa, F. Panloup, Multilevel-Langevin pathwise average for Gibbs approximation, Mathematics of Operations Research, 2025; preprint arXiv:2109.07753v2, 2024. https://arxiv.org/abs/2109.07753
  • M. B. Giles, Multilevel Monte Carlo path simulation, Operations Research 56(3), 2008. https://doi.org/10.1287/opre.1070.0496
  • A. S. Dalalyan, Theoretical guarantees for approximate sampling from smooth and log-concave densities, J. R. Stat. Soc. B 79(3), 2017. https://doi.org/10.1111/rssb.12183
  • A. Durmus, É. Moulines, Nonasymptotic convergence analysis for the unadjusted Langevin algorithm, Ann. Appl. Probab. 27(3), 2017. https://doi.org/10.1214/16-AAP1238
15 thms1 active userReviewed
Numerical AnalysisProbability·Captain: mikedeng1

Multilevel Langevin Pathwise Average for Gibbs Approximation I: Under (H₁)–(H₄) the Multilevel Occupation Measure Has L²-Error ≲ [f]₁ε at Cost 𝔠_𝔞γ₀⁻¹𝔱ε⁻², or ε⁻²R_ε³ When 𝔞 = 1Research Paper

Motivation

Sampling from a Gibbs distribution π(dx)∝e−U(x)dx\pi(dx)\propto e^{-U(x)}dxπ(dx)∝e−U(x)dx on Rd\mathbb R^dRd, or computing integrals π(f)\pi(f)π(f) against it, is a basic task in Bayesian statistics and machine learning. A standard approach simulates an ergodic diffusion whose invariant distribution is π\piπ, such as the overdamped Langevin diffusion dXt=−∇U(Xt)dt+2 dBtdX_t=-\nabla U(X_t)dt+\sqrt2\,dB_tdXt​=−∇U(Xt​)dt+2​dBt​, discretizes it with an Euler scheme of step γ\gammaγ, and averages fff along one long trajectory. Two errors compete: a bias of order γδ\gamma^\deltaγδ, because the scheme's invariant distribution πγ\pi^\gammaπγ differs from π\piπ, and a variance of order 1/T1/T1/T for a time horizon TTT. With a single step, balancing them requires of the order of ε−3\varepsilon^{-3}ε−3 Euler iterations for an ε\varepsilonε-approximation when the bias is of order γ\gammaγ (Egéa–Panloup, p. 3).

Multilevel Monte Carlo (Giles 2008, doi:10.1287/opre.1070.0496) removes this extra factor in finite-horizon problems by combining coarse and fine discretizations whose differences have small variance. Pagès and Panloup (arXiv:1609.00412) and Giles, Majka, Szpruch, Vollmer and Zygalakis (arXiv:1805.10974) adapted the idea to invariant distributions. Egéa and Panloup (arXiv:2109.07753, Math. Oper. Res. 2025) build a multilevel pathwise average, a combination of occupation measures of Euler schemes, and prove an L2L^2L2-error of order ε\varepsilonε at cost ε−2\varepsilon^{-2}ε−2, or ε−2∣log⁡ε∣3\varepsilon^{-2}|\log\varepsilon|^3ε−2∣logε∣3 in a borderline case, under four abstract assumptions. This mission formalizes that general result (their Theorem 2.1 (ii)) and the bias and variance estimates behind it.

Setting

Let b:Rd→Rdb:\mathbb R^d\to\mathbb R^db:Rd→Rd and σ:Rd→Md,d\sigma:\mathbb R^d\to\mathbb M_{d,d}σ:Rd→Md,d​ be Lipschitz. For a step γ>0\gamma>0γ>0, a starting point xxx and a ddd-dimensional standard Brownian motion WWW, the Euler scheme is Xˉ0γ,x=x\bar X^{\gamma,x}_0=xXˉ0γ,x​=x and

Xˉ(n+1)γγ,x=Xˉnγγ,x+γb(Xˉnγγ,x)+σ(Xˉnγγ,x)(W(n+1)γ−Wnγ).\bar X^{\gamma,x}_{(n+1)\gamma}=\bar X^{\gamma,x}_{n\gamma}+\gamma b(\bar X^{\gamma,x}_{n\gamma})+\sigma(\bar X^{\gamma,x}_{n\gamma})(W_{(n+1)\gamma}-W_{n\gamma}).Xˉ(n+1)γγ,x​=Xˉnγγ,x​+γb(Xˉnγγ,x​)+σ(Xˉnγγ,x​)(W(n+1)γ​−Wnγ​).

Write s‾η=η⌊s/η⌋\underline{s}_\eta=\eta\lfloor s/\eta\rfloors​η​=η⌊s/η⌋ and [f]1[f]_1[f]1​ for the Lipschitz constant of f:Rd→Rf:\mathbb R^d\to\mathbb Rf:Rd→R.

Fix x0x_0x0​, a number of levels R≥1R\ge1R≥1, steps γr=γ02−r\gamma_r=\gamma_02^{-r}γr​=γ0​2−r, horizons T0,…,TRT_0,\dots,T_RT0​,…,TR​, a warm start τ\tauτ, and independent Brownian motions B(0),…,B(R)B^{(0)},\dots,B^{(R)}B(0),…,B(R). The multilevel occupation measure is

Y(f)=1T0−τ∫τT0f(Xˉs‾γ0γ0,x0,B(0))ds+∑r=1R1Tr−τ∫τTr(f(Xˉs‾γr−1γr,x0,B(r))−f(Xˉs‾γr−1γr−1,x0,B(r)))ds.\mathcal Y(f)=\frac1{T_0-\tau}\int_\tau^{T_0}f(\bar X^{\gamma_0,x_0,B^{(0)}}_{\underline{s}_{\gamma_0}})ds+\sum_{r=1}^R\frac1{T_r-\tau}\int_\tau^{T_r}\Big(f(\bar X^{\gamma_r,x_0,B^{(r)}}_{\underline{s}_{\gamma_{r-1}}})-f(\bar X^{\gamma_{r-1},x_0,B^{(r)}}_{\underline{s}_{\gamma_{r-1}}})\Big)ds .Y(f)=T0​−τ1​∫τT0​​f(Xˉs​γ0​​γ0​,x0​,B(0)​)ds+r=1∑R​Tr​−τ1​∫τTr​​(f(Xˉs​γr−1​​γr​,x0​,B(r)​)−f(Xˉs​γr−1​​γr−1​,x0​,B(r)​))ds.

Inside level r≥1r\ge1r≥1 both schemes use the same Brownian motion and are read at the coarse times. The cost is C(Y)=T0/γ0+∑r=1R(Tr/γr+Tr/γr−1)\mathcal C(\mathcal Y)=T_0/\gamma_0+\sum_{r=1}^R(T_r/\gamma_r+T_r/\gamma_{r-1})C(Y)=T0​/γ0​+∑r=1R​(Tr​/γr​+Tr​/γr−1​), the number of Euler iterations.

The assumptions involve α>0\alpha>0α>0, a∈[1,2]\mathfrak a\in[1,2]a∈[1,2], η0>0\eta_0>0η0​>0, the invariant distributions π\piπ of the diffusion and πγ\pi^\gammaπγ of the scheme, and ∥Z∥2=E[∣Z∣2]1/2\|Z\|_2=\mathbb E[|Z|^2]^{1/2}∥Z∥2​=E[∣Z∣2]1/2. For every γ∈(0,η0]\gamma\in(0,\eta_0]γ∈(0,η0​]:

  • (H1)(\mathbf H_1)(H1​) ∣Ef(Xˉnγγ,x)−πγ(f)∣≤c1(x)[f]1e−αnγ|\mathbb E f(\bar X^{\gamma,x}_{n\gamma})-\pi^\gamma(f)|\le c_1(x)[f]_1e^{-\alpha n\gamma}∣Ef(Xˉnγγ,x​)−πγ(f)∣≤c1​(x)[f]1​e−αnγ (exponential convergence to equilibrium);
  • (H2)(\mathbf H_2)(H2​) ∥Xˉnγγ,x0−Xˉnγγ/2,x0∥2≤c2γa/2\|\bar X^{\gamma,x_0}_{n\gamma}-\bar X^{\gamma/2,x_0}_{n\gamma}\|_2\le c_2\gamma^{\mathfrak a/2}∥Xˉnγγ,x0​​−Xˉnγγ/2,x0​​∥2​≤c2​γa/2 (L2L^2L2-confluence);
  • (H3)(\mathbf H_3)(H3​) ∣π(f)−πγ(f)∣≤c3[f]1γδ|\pi(f)-\pi^\gamma(f)|\le c_3[f]_1\gamma^\delta∣π(f)−πγ(f)∣≤c3​[f]1​γδ, with δ∈[1/2,1]\delta\in[1/2,1]δ∈[1/2,1] if a=1\mathfrak a=1a=1 and δ∈((1+a)/4,1]\delta\in((1+\mathfrak a)/4,1]δ∈((1+a)/4,1] if a>1\mathfrak a>1a>1;
  • (H4)(\mathbf H_4)(H4​) ∥Xˉtγ,x0−x0∥2+∥c1(Xˉtγ,x0)∥2≤c4\|\bar X^{\gamma,x_0}_{t}-x_0\|_2+\|c_1(\bar X^{\gamma,x_0}_{t})\|_2\le c_4∥Xˉtγ,x0​​−x0​∥2​+∥c1​(Xˉtγ,x0​​)∥2​≤c4​.

Formalization targets

Goal: Theorem 2.1 (ii)

With α≤1\alpha\le1α≤1, η0≤12\eta_0\le\frac12η0​≤21​, γ0≤η0\gamma_0\le\eta_0γ0​≤η0​, r0≥1∨c3γ0δr_0\ge1\vee c_3\gamma_0^\deltar0​≥1∨c3​γ0δ​, t≥t0=daαmax⁡(c22γ0alog⁡γ0−1,c42)\mathfrak t\ge\mathfrak t_0=\frac{\mathfrak d_{\mathfrak a}}\alpha\max(c_2^2\gamma_0^{\mathfrak a}\log\gamma_0^{-1},c_4^2)t≥t0​=αda​​max(c22​γ0a​logγ0−1​,c42​), ε∈(0,1)\varepsilon\in(0,1)ε∈(0,1), Rε=⌈δ−1log⁡2(r0/ε)⌉R_\varepsilon=\lceil\delta^{-1}\log_2(r_0/\varepsilon)\rceilRε​=⌈δ−1log2​(r0​/ε)⌉, Tr=tε−22−1+a2rT_r=\mathfrak t\varepsilon^{-2}2^{-\frac{1+\mathfrak a}2r}Tr​=tε−22−21+a​r (a>1\mathfrak a>1a>1) or tε−2Rε22−r\mathfrak t\varepsilon^{-2}R_\varepsilon^22^{-r}tε−2Rε2​2−r (a=1\mathfrak a=1a=1), and τ∈[τ1∣log⁡ε∣+τ2,12TRε]\tau\in[\tau_1|\log\varepsilon|+\tau_2,\frac12T_{R_\varepsilon}]τ∈[τ1​∣logε∣+τ2​,21​TRε​​]:

∥Y(f)−π(f)∥2≤C1[f]1ε,C(Y)≤caγ0−1t ε−2 (a>1),≤caγ0−1t ε−2Rε3 (a=1),\|\mathcal Y(f)-\pi(f)\|_2\le\mathfrak C_1[f]_1\varepsilon,\qquad \mathcal C(\mathcal Y)\le\mathfrak c_{\mathfrak a}\gamma_0^{-1}\mathfrak t\,\varepsilon^{-2}\ (\mathfrak a>1),\quad\le\mathfrak c_{\mathfrak a}\gamma_0^{-1}\mathfrak t\,\varepsilon^{-2}R_\varepsilon^3\ (\mathfrak a=1),∥Y(f)−π(f)∥2​≤C1​[f]1​ε,C(Y)≤ca​γ0−1​tε−2 (a>1),≤ca​γ0−1​tε−2Rε3​ (a=1),

with C1\mathfrak C_1C1​ a universal constant. The cost constant ca\mathfrak c_{\mathfrak a}ca​ is explicit; the error constant is left existential, which is how the paper states it.

Companion: Theorem 2.1 (i)

With the same steps and horizons but r0≥1r_0\ge1r0​≥1, t>0\mathfrak t>0t>0 arbitrary and τ∈[τ1log⁡(ε−1)∧12TRε,12TRε]\tau\in[\tau_1\log(\varepsilon^{-1})\wedge\frac12T_{R_\varepsilon},\frac12T_{R_\varepsilon}]τ∈[τ1​log(ε−1)∧21​TRε​​,21​TRε​​] for a fixed τ1>1+a−2δ2αδ\tau_1>\frac{1+\mathfrak a-2\delta}{2\alpha\delta}τ1​>2αδ1+a−2δ​, the same two bounds hold with an error constant C1\mathfrak C_1C1​ that depends on the parameters but not on fff, ε\varepsilonε or τ\tauτ. It is a separate theorem item, not a milestone.

Milestones

Lemma 3.1 (long-time bias of one average), Proposition 3.2 (bias of Y\mathcal YY), the variance decomposition (32), Lemma 3.2 (variance of one correcting level), Proposition 3.3 (variance of Y\mathcal YY), the cost bound (28), and Proposition 3.1, the non-asymptotic squared-error bound for given steps and horizons.

Significance

Theorem 2.1 separates the analysis of the multilevel estimator from the analysis of any particular diffusion. Once a diffusion satisfies (H1)(\mathbf H_1)(H1​)–(H4)(\mathbf H_4)(H4​), the complexity follows with explicit parameters. The paper then checks the assumptions for Langevin diffusions with strongly convex potentials and obtains dimension-explicit costs O(dε−2log⁡3(dε−2))\mathcal O(d\varepsilon^{-2}\log^3(d\varepsilon^{-2}))O(dε−2log3(dε−2)) and O(dε−2)\mathcal O(d\varepsilon^{-2})O(dε−2). Those are the companion missions II and III of this series; they rest on the same estimator and cost.

The paper's proofs are on paper only. No machine-checked proof of a multilevel ergodic estimator is known to us. This mission produces a statement-level formalization of the estimator, its cost, the four assumptions and the bias–variance bounds. A complete development would add discrete-time Markov-property arguments for Euler schemes driven by a Brownian motion, covariance bounds for ergodic averages, and the elementary but delicate parameter calculus of §4.

Difficulty

The bias step is close to routine once (H1)(\mathbf H_1)(H1​) and (H3)(\mathbf H_3)(H3​) are given. The variance of a correcting level is not. Bounding it by the second moment alone gives (T−τ)−1⋅Tγa(T-\tau)^{-1}\cdot T\gamma^{\mathfrak a}(T−τ)−1⋅Tγa, which does not decay with TTT. The proof must also use mixing: covariances between times sss and uuu far apart are controlled by (H1)(\mathbf H_1)(H1​) through the Markov property, which needs the conditional law of each scheme given its past and a moment bound on c1c_1c1​ along the path (H4)(\mathbf H_4)(H4​). The two regimes are matched at a separation of order ∣log⁡γ∣/α|\log\gamma|/\alpha∣logγ∣/α, and this is where the factor log⁡(1/γ)\log(1/\gamma)log(1/γ) comes from. At the level of the theorem, the parameters (8) and the warm start τ\tauτ must make three error terms simultaneously ≲ε\lesssim\varepsilon≲ε with a constant independent of ddd, fff, the diffusion and every assumption constant. Getting the uniformity right is what the explicit thresholds t0\mathfrak t_0t0​, τ1\tau_1τ1​, τ2\tau_2τ2​ are for.

Formalization scope

  • Rd\mathbb R^dRd is the published EthierKurtz.SDEState d, and a Brownian motion is the published EthierKurtz.IsStandardBrownian. The levels' Brownian motions form one family B : ℕ → ℝ≥0 → Ω → ℝ^d, mutually independent as path-valued variables. σ(x)\sigma(x)σ(x) is a continuous linear map Rd→Rd\mathbb R^d\to\mathbb R^dRd→Rd.
  • The Euler scheme is evaluated at grid times. (H1)(\mathbf H_1)(H1​)–(H4)(\mathbf H_4)(H4​) are stated at grid times nγn\gammanγ, for every level's Brownian motion. For (H4)(\mathbf H_4)(H4​), whose supremum the paper takes over all real ttt, this is a weaker assumption, so the theorems are stronger.
  • ∥⋅∥2\|\cdot\|_2∥⋅∥2​ is eLpNorm · 2 P and variances are evariance, both valued in [0,∞][0,\infty][0,∞], so an infinite moment is never read as 000. Expectations that are concluded come with an integrability conjunct.
  • π\piπ and πγ\pi^\gammaπγ are arbitrary probability measures satisfying (H1)(\mathbf H_1)(H1​) and (H3)(\mathbf H_3)(H3​), with fff integrable. The SDE and the invariance of π\piπ, πγ\pi^\gammaπγ are not formalized: the results never use invariance, so the statements generalize the paper's.
  • [f]1[f]_1[f]1​ is the least Lipschitz constant. The cost is the formula of p. 6 for all horizons. (a−1)−2∧R2(\mathfrak a-1)^{-2}\wedge R^2(a−1)−2∧R2, da\mathfrak d_{\mathfrak a}da​ and ca\mathfrak c_{\mathfrak a}ca​ are defined case by case, so a=1\mathfrak a=1a=1 never divides by zero.
  • Universal constants (C1\mathfrak C_1C1​, cu\mathfrak c_ucu​) are quantified before the dimension, the probability space and every datum.
  • Added hypotheses: γ≤1/2\gamma\le1/2γ≤1/2 and α≤1\alpha\le1α≤1 in Lemma 3.2, and η0≤1/2\eta_0\le1/2η0​≤1/2 in Proposition 3.3. These are the standing ranges of Theorem 2.1; without them the printed bounds fail for steps ≥1\ge1≥1. The restriction ε<ε0\varepsilon<\varepsilon_0ε<ε0​ of the goal is replaced by quantifying over every τ\tauτ in the admissible interval, which is the same claim.
  • A formalization in which the bias or variance bounds, Proposition 3.1 or any covariance estimate appear as hypotheses of the goal is excluded. The goal assumes only (H1)(\mathbf H_1)(H1​)–(H4)(\mathbf H_4)(H4​) and the parameter choices (8), (39)–(40).

Contributions are welcome on any milestone, on Markov-property and moment lemmas for Euler schemes driven by IsStandardBrownian, and on the Theorem 2.1 (i) companion with non-explicit constants.

Selected references

  • M. Egéa, F. Panloup, Multilevel-Langevin pathwise average for Gibbs approximation, arXiv:2109.07753v2, 2024; Math. Oper. Res., 2025. https://arxiv.org/abs/2109.07753
  • G. Pagès, F. Panloup, Weighted multilevel Langevin simulation of invariant measures, Ann. Appl. Probab. 28(6), 2018. https://arxiv.org/abs/1609.00412
  • M. B. Giles, M. B. Majka, L. Szpruch, S. Vollmer, K. Zygalakis, Multi-level Monte Carlo methods for the approximation of invariant measures of stochastic differential equations, Stat. Comput. 30, 2020. https://arxiv.org/abs/1805.10974
  • M. B. Giles, Multilevel Monte Carlo path simulation, Oper. Res. 56(3), 2008. https://doi.org/10.1287/opre.1070.0496
12 thms1 active userReviewed
Operations ResearchProbabilityStochastic Systems·Captain: mikedeng1

An Approximation to the Invariant Measure of the Limiting Diffusion of G/Ph/n + GI Queues in the Halfin–Whitt Regime and Related Asymptotics 2: CLT for Occupation Measures With Variance µ(|σ′∇f|²)Research Paper

Motivation

In a many-server queue with nnn servers, the Halfin–Whitt (quality-and-efficiency-driven) regime lets n→∞n\to\inftyn→∞ while the arrival rate approaches capacity at rate n\sqrt nn​. In this regime the centred and scaled number of customers in each service phase of a G/Ph/n+GIG/Ph/n+GIG/Ph/n+GI queue (general arrivals, phase-type service, general patience) converges to a ddd-dimensional diffusion with a piecewise-linear drift (Dai, He and Tezcan, Ann. Appl. Probab. 2010). This diffusion is exponentially ergodic with a unique invariant measure μ\muμ (Dieker and Gao, Ann. Appl. Probab. 2013), but for d≥2d\ge2d≥2 no explicit formula for μ\muμ is known.

Since μ\muμ is not explicit, its averages μ(h)\mu(h)μ(h) are estimated by simulation, for example by the time average of hhh along one long trajectory. The second main result of Jin, Pang, Xu and Xu (arXiv:2209.07361) is a central limit theorem for these time averages. It gives the t\sqrt tt​ rate of convergence, for every starting point, and identifies the limiting variance through the solution of a Stein (Poisson) equation.

Setting

A model consists of d≥1d\ge1d≥1, a patience rate α>0\alpha>0α>0, β∈R\beta\in\mathbb Rβ∈R, a probability vector p∈Rdp\in\mathbb R^dp∈Rd, service rates vk>0v_k>0vk​>0, a sub-stochastic matrix PPP with Pii=0P_{ii}=0Pii​=0 and I−PI-PI−P invertible, and a d×dd\times dd×d matrix σ\sigmaσ. Put R=(I−P′)diag(v)R=(I-P')\mathrm{diag}(v)R=(I−P′)diag(v) and e=(1,…,1)′e=(1,\dots,1)'e=(1,…,1)′. The phase distribution has mean one, e′R−1p=1e'R^{-1}p=1e′R−1p=1, and σσ′\sigma\sigma'σσ′ is uniformly elliptic: ξ′σσ′ξ≥c∣ξ∣2\xi'\sigma\sigma'\xi\ge c|\xi|^2ξ′σσ′ξ≥c∣ξ∣2 for some c>0c>0c>0. The drift is

g(x)=−βp−Rx+(R−αI)p (e′x)+,g(x)=-\beta p-Rx+(R-\alpha I)p\,(e'x)^+ ,g(x)=−βp−Rx+(R−αI)p(e′x)+,

and the limiting diffusion solves, for a standard Brownian motion BBB,

Xtx=x+∫0tg(Xsx) ds+σBt.X^x_t=x+\int_0^tg(X^x_s)\,ds+\sigma B_t .Xtx​=x+∫0t​g(Xsx​)ds+σBt​.

Its semigroup is Pth(x)=Eh(Xtx)P_th(x)=\mathbb Eh(X^x_t)Pt​h(x)=Eh(Xtx​) and μ\muμ is its invariant probability measure. For bounded measurable hhh, the empirical average is Etx(h)=1t∫0th(Xsx) ds\mathcal E^x_t(h)=\frac1t\int_0^th(X^x_s)\,dsEtx​(h)=t1​∫0t​h(Xsx​)ds. The Stein solution is

f(x)=−∫0∞(Pth(x)−μ(h)) dt,f(x)=-\int_0^\infty\big(P_th(x)-\mu(h)\big)\,dt ,f(x)=−∫0∞​(Pt​h(x)−μ(h))dt,

which solves Af=h−μ(h)\mathcal Af=h-\mu(h)Af=h−μ(h) for the generator Af=⟨∇f,g⟩+12⟨σσ′,∇2f⟩HS\mathcal Af=\langle\nabla f,g\rangle+\frac12\langle\sigma\sigma',\nabla^2f\rangle_{HS}Af=⟨∇f,g⟩+21​⟨σσ′,∇2f⟩HS​. Here ∥h∥∞\|h\|_\infty∥h∥∞​ is the essential supremum of ∣h∣|h|∣h∣.

Formalization targets

Goal: Theorem 1.3

There is C>0C>0C>0 such that for every bounded measurable hhh, fff is differentiable, μ(∣σ′∇f∣2)≤C∥h∥∞2\mu(|\sigma'\nabla f|^2)\le C\|h\|_\infty^2μ(∣σ′∇f∣2)≤C∥h∥∞2​, and for every xxx

t [Etx(h)−μ(h)]⇒N(0,μ(∣σ′∇f∣2))(t→∞).\sqrt t\,\big[\mathcal E^x_t(h)-\mu(h)\big]\Rightarrow\mathcal N\big(0,\mu(|\sigma'\nabla f|^2)\big)\qquad(t\to\infty).t​[Etx​(h)−μ(h)]⇒N(0,μ(∣σ′∇f∣2))(t→∞).

Milestones (§3 of the paper)

  • Lemma 3.2: the integral defining fff converges, fff solves (3.1) in integrated form, and f(x)=∫0∞e−λtPt[λf−h+μ(h)](x) dtf(x)=\int_0^\infty e^{-\lambda t}P_t[\lambda f-h+\mu(h)](x)\,dtf(x)=∫0∞​e−λtPt​[λf−h+μ(h)](x)dt for every λ>0\lambda>0λ>0.
  • Lemma 3.4: moment bounds E∣Xtx∣m≤eCt(∣x∣m+1)\mathbb E|X^x_t|^m\le e^{Ct}(|x|^m+1)E∣Xtx​∣m≤eCt(∣x∣m+1), uniform over a family of mollified diffusions XεX^{\varepsilon}Xε, and E∣Xtε,x−Xtx∣m→0\mathbb E|X^{\varepsilon,x}_t-X^x_t|^m\to0E∣Xtε,x​−Xtx​∣m→0.
  • Proposition 3.7: the weighted occupation time of e′Xxe'X^xe′Xx in [−ϵ,ϵ][-\epsilon,\epsilon][−ϵ,ϵ] has mean at most Cϵect(1+∣x∣)(1+t)C\epsilon e^{ct}(1+|x|)(1+t)Cϵect(1+∣x∣)(1+t).
  • Lemma 3.5: the Jacobians of the mollified diffusions converge almost surely, simultaneously for all 0≤s≤t0\le s\le t0≤s≤t, to the Jacobian JxJ^xJx of XxX^xXx.
  • Lemma 3.6: LmL^mLm bounds C∣u∣mt−m/2emCoptC|u|^mt^{-m/2}e^{mC_{op}t}C∣u∣mt−m/2emCop​t on the weights Iux(t)=1t∫0t⟨σ−1Jrxu,dBr⟩\mathcal I^x_u(t)=\frac1t\int_0^t\langle\sigma^{-1}J^x_ru,dB_r\rangleIux​(t)=t1​∫0t​⟨σ−1Jrx​u,dBr​⟩, and their convergence as ε→0\varepsilon\to0ε→0.
  • Lemma 3.3: ∇uEψ(Xtx)=E[ψ(Xtx)Iux(t)]\nabla_u\mathbb E\psi(X^x_t)=\mathbb E[\psi(X^x_t)\mathcal I^x_u(t)]∇u​Eψ(Xtx​)=E[ψ(Xtx​)Iux​(t)] for C1C^1C1 functions ψ\psiψ with bounded gradient.
  • Proposition 3.1: ∣f(x)∣,∣∇f(x)∣≤C∥h∥∞(1+∣x∣2)|f(x)|,|\nabla f(x)|\le C\|h\|_\infty(1+|x|^2)∣f(x)∣,∣∇f(x)∣≤C∥h∥∞​(1+∣x∣2).

Significance

The theorem gives the rate t\sqrt tt​ and the asymptotic variance of the standard ergodic-average estimator of μ(h)\mu(h)μ(h), which yields asymptotic confidence intervals for steady-state performance measures of the queue approximated by the diffusion. The function hhh need only be bounded and measurable, so indicator functions are included, for example the probability that the total queue e′Xe'Xe′X exceeds a level. The same variance governs the moderate deviation principle (Theorem 1.4 of the paper), which is a planned follow-up.

The result is proved in the source paper. None of it is formalized. Formalizing it requires Lean infrastructure that does not exist yet and is reusable beyond queueing: Stein/Poisson equations for non-smooth drifts, a Bismut–Elworthy–Li formula, derivative flows of diffusions, and a continuous-time martingale CLT.

Difficulty

The obvious argument applies Itô's formula to the Stein solution f(Xt)f(X_t)f(Xt​): the time average becomes a martingale plus a boundary term, and a martingale CLT finishes the proof. That argument needs f∈C1f\in C^1f∈C1 with polynomial growth of ∇f\nabla f∇f. For bounded measurable hhh, classical regularity results for Stein equations do not apply, and the drift ggg is not differentiable on the hyperplane e′x=0e'x=0e′x=0, so the diffusion cannot be differentiated in its starting point directly. The central difficulty is Proposition 3.1, a gradient bound for fff when neither hhh nor ggg is differentiable.

Formalization scope

  • The state space is EuclideanSpace ℝ (Fin d) with the Euclidean norm. Matrices act through Matrix.toEuclideanCLM, whose norm is the operator norm.
  • Time is ℝ≥0. Brownian motion is the published predicate EthierKurtz.IsStandardBrownian. The diffusion is the pathwise solution of the integral equation above, for every ω\omegaω.
  • Statements take (Ω,P,B,X)(\Omega,\mathbb P,B,X)(Ω,P,B,X) and an invariant probability measure μ\muμ as hypotheses. Both exist, and the conclusions depend only on the law.
  • σ\sigmaσ is a general uniformly elliptic matrix. The paper's explicit formula for σσ′\sigma\sigma'σσ′ (p. 3) is one instance, and Remark 1.1 says the results use only ellipticity.
  • "fff is the solution to (3.1)" means the function (3.2). Its differentiability is part of the conclusions, because in Lean the gradient of a non-differentiable function is 000 and the variance could otherwise vanish silently.
  • Weak convergence is convergence of Eφ(Yt)\mathbb E\varphi(Y_t)Eφ(Yt​) for every bounded continuous φ\varphiφ (§1.4), along real t→∞t\to\inftyt→∞. Expectations of powers are lower Lebesgue integrals.
  • The constant CCC in the goal and in Proposition 3.1 is uniform in hhh.
  • The Jacobians Js,tJ_{s,t}Js,t​ are propagators of the linearized equation, not the paper's exp⁡∫st∇g dr\exp\int_s^t\nabla g\,drexp∫st​∇gdr, which fails to solve the equation when the matrices do not commute (d≥2d\ge2d≥2).
  • Itô integrals are characterized as L2L^2L2 limits of dyadic left-point Riemann sums.
  • Lemma 3.4 states a model-dependent rate CCC in place of Cm=2m2C~opC_m=2m^2\tilde C_{op}Cm​=2m2C~op​, because the explicit rate fails in large dimension.

Restricting hhh to smooth or Lipschitz functions, starting XXX in μ\muμ, taking d=1d=1d=1 or a diagonal σ\sigmaσ, or dropping the differentiability of fff each makes the statement weaker or trivial, and none of them is the theorem.

Welcome contributions include a general Itô integral and Itô formula for additive-noise diffusions, a martingale CLT in continuous time, exponential ergodicity of this diffusion (Lemma A.1 of the paper, posed in the companion mission on the Euler–Maruyama scheme), and any of the milestones.

Selected references

  • X. Jin, G. Pang, L. Xu, X. Xu, An Approximation to the Invariant Measure of the Limiting Diffusion of G/Ph/n+GI Queues in the Halfin–Whitt Regime and Related Asymptotics, arXiv:2209.07361v1, 2022. https://arxiv.org/abs/2209.07361
  • J. G. Dai, S. He, T. Tezcan, Many-server diffusion limits for G/Ph/n+GI queues, Ann. Appl. Probab. 20(5), 2010. https://doi.org/10.1214/09-AAP674
  • A. B. Dieker, X. Gao, Positive recurrence of piecewise Ornstein–Uhlenbeck processes and common quadratic Lyapunov functions, Ann. Appl. Probab. 23(4), 2013. https://doi.org/10.1214/12-AAP870
  • X. Fang, Q.-M. Shao, L. Xu, Multivariate approximations in Wasserstein distance by Stein's method and Bismut's formula, Probab. Theory Relat. Fields 174, 2019. https://doi.org/10.1007/s00440-018-0874-5
16 thms1 active userReviewed
Dynamic ProgrammingOperations ResearchOptimization·Captain: mikedeng1

Robust Markov Decision Processes: Beyond Rectangularity 2: With Polyhedral r-Rectangular Factor Sets, One Deterministic Policy and Factor Matrix Stay Robust-Optimal for All Discount Factors Near 1Research Paper

Motivation

A Markov decision process (MDP) models sequential decisions under uncertainty: in each state a decision maker picks an action, collects a reward, and the system moves to a random next state according to a transition kernel. In applications the kernel is estimated from data, and an optimal policy for the estimated kernel can perform poorly under a slightly different one. Robust MDPs replace the single kernel by an uncertainty set and maximize the worst-case discounted reward over that set (Iyengar 2005; Nilim and El Ghaoui 2005; Wiesemann, Kuhn and Rustem 2013).

Goyal and Grand-Clément (arXiv:1811.00215v8) introduce r-rectangular uncertainty sets, in which transitions are mixtures of a small number of uncertain factors chosen independently by an adversary. Beyond the algorithmic results (the subject of the first mission of this series), they prove two structural properties of optimal robust policies in this model: a robust maximum principle and robust Blackwell optimality.

In a classical MDP, a policy is Blackwell optimal if it is optimal for every discount factor close enough to 111 (Blackwell 1962; Puterman 1994, §10.1.2). Blackwell optimal policies are optimal for the long-run average reward and for every finer nnn-discount criterion, so they connect discounted and average-reward control. For two-player perfect-information zero-sum stochastic games the existence of Blackwell optimal strategies is due to Gaubert and Gunawardena (1998). Goyal and Grand-Clément establish the analogous statement for robust MDPs with r-rectangular uncertainty, under the assumption that the factor sets have finitely many extreme points, as polyhedral factor sets do.

Setting

Let S\mathbb SS and A\mathbb AA be finite sets of states and actions, with rewards rsa≥0r_{sa}\ge0rsa​≥0, an initial distribution p0∈Δ(S)p_0\in\Delta(\mathbb S)p0​∈Δ(S), and a discount factor λ∈(0,1)\lambda\in(0,1)λ∈(0,1). Fix coefficients usai≥0u^i_{sa}\ge0usai​≥0, i∈[r]i\in[r]i∈[r], with ∑iusai=1\sum_i u^i_{sa}=1∑i​usai​=1. A factor matrix W=(w1,…,wr)W=(w_1,\dots,w_r)W=(w1​,…,wr​) has probability vectors wi∈Δ(S)w_i\in\Delta(\mathbb S)wi​∈Δ(S) as columns and induces the kernel

Psas′=∑i=1rusai wi,s′.P_{sas'}=\sum_{i=1}^r u^i_{sa}\,w_{i,s'} .Psas′​=i=1∑r​usai​wi,s′​.

The adversary chooses WWW from a nonempty convex compact set W\mathcal WW. Its projections are the factor sets W1,…,Wr\mathcal W^1,\dots,\mathcal W^rW1,…,Wr, and W\mathcal WW is r-rectangular when W=W1×⋯×Wr\mathcal W=\mathcal W^1\times\dots\times\mathcal W^rW=W1×⋯×Wr.

A stationary policy π∈Π\pi\in\Piπ∈Π picks πs∈Δ(A)\pi_s\in\Delta(\mathbb A)πs​∈Δ(A) in each state; it is deterministic when each πs\pi_sπs​ is a point mass. With rπ=(∑aπsarsa)sr_\pi=(\sum_a\pi_{sa}r_{sa})_srπ​=(∑a​πsa​rsa​)s​ and L(π,W)ss′=∑aπsaPsas′L(\pi,W)_{ss'}=\sum_a\pi_{sa}P_{sas'}L(π,W)ss′​=∑a​πsa​Psas′​, the value function and the expected reward are

vλπ,W=∑t≥0λtL(π,W)trπ,R(π,W,λ)=p0⊤vλπ,W.v^{\pi,W}_\lambda=\sum_{t\ge0}\lambda^tL(\pi,W)^tr_\pi,\qquad R(\pi,W,\lambda)=p_0^\top v^{\pi,W}_\lambda .vλπ,W​=t≥0∑​λtL(π,W)trπ​,R(π,W,λ)=p0⊤​vλπ,W​.

The worst-case value of π\piπ is zλ(π)=min⁡W∈WR(π,W,λ)z_\lambda(\pi)=\min_{W\in\mathcal W}R(\pi,W,\lambda)zλ​(π)=minW∈W​R(π,W,λ). A pair (π∗,W∗)(\pi^*,W^*)(π∗,W∗) is an optimal solution of the robust MDP problem (1.2) at discount λ\lambdaλ when π∗\pi^*π∗ maximizes zλz_\lambdazλ​ over Π\PiΠ and W∗W^*W∗ attains min⁡W∈WR(π∗,W,λ)\min_{W\in\mathcal W}R(\pi^*,W,\lambda)minW∈W​R(π∗,W,λ). The Lean development uses this notation: value, R, z, IsRobustSolution, IsRRect, factorSet, IsDeterministic.

Formalization targets

Goal: robust Blackwell optimality (Proposition 6.2)

If W\mathcal WW is r-rectangular and every factor set Wi\mathcal W^iWi has finitely many extreme points, there exist a deterministic policy π∗\pi^*π∗, a factor matrix W∗∈WW^*\in\mathcal WW∗∈W and λ0∈(0,1)\lambda_0\in(0,1)λ0​∈(0,1) with

∀λ∈(λ0,1):(π∗,W∗) is an optimal solution of (1.2) at discount λ.\forall\lambda\in(\lambda_0,1):\quad(\pi^*,W^*)\ \text{is an optimal solution of (1.2) at discount }\lambda .∀λ∈(λ0​,1):(π∗,W∗) is an optimal solution of (1.2) at discount λ.

The pair is fixed before λ0\lambda_0λ0​ and does not depend on λ\lambdaλ. The statement fixes no constant and no rate.

Milestones

  1. Proposition 6.1.1: a minimizer W1W^1W1 of R(π,⋅,λ)R(\pi,\cdot,\lambda)R(π,⋅,λ) is a worst case from every initial state, vλ,sπ,W1≤vλ,sπ,W0v^{\pi,W^1}_{\lambda,s}\le v^{\pi,W^0}_{\lambda,s}vλ,sπ,W1​≤vλ,sπ,W0​ for all W0∈WW^0\in\mathcal WW0∈W.
  2. Proposition 6.1.2 (robust maximum principle): if (π∗,W∗)(\pi^*,W^*)(π∗,W∗) solves (1.2), then vλ,sπ,W1≤vλ,sπ∗,W∗v^{\pi,W^1}_{\lambda,s}\le v^{\pi^*,W^*}_{\lambda,s}vλ,sπ,W1​≤vλ,sπ∗,W∗​ for every policy π\piπ, every minimizer W1W^1W1 of R(π,⋅,λ)R(\pi,\cdot,\lambda)R(π,⋅,λ) and every state sss.
  3. Appendix G.1, p. 26: for each λ\lambdaλ, (1.2) has an optimal solution (πλ,Wλ)(\pi_\lambda,W_\lambda)(πλ​,Wλ​) with πλ\pi_\lambdaπλ​ deterministic and WλW_\lambdaWλ​ an extreme point of W\mathcal WW.
  4. Appendix G.1, p. 27: vtπ,W=(I−tL(π,W))−1rπv^{\pi,W}_t=(I-tL(\pi,W))^{-1}r_\pivtπ,W​=(I−tL(π,W))−1rπ​ on (0,1)(0,1)(0,1), and each coordinate is a rational function of ttt whose denominator has no zero in (0,1)(0,1)(0,1).

Stronger: finitely many breakpoints (Proposition 6.3, optional)

Under the same hypotheses there are 0=λ0<λ1<⋯<λp=10=\lambda_0<\lambda_1<\dots<\lambda_p=10=λ0​<λ1​<⋯<λp​=1 and, for each jjj, a single deterministic pair (πj,Wj)(\pi_j,W_j)(πj​,Wj​) optimal for every λ∈(λj,λj+1)\lambda\in(\lambda_j,\lambda_{j+1})λ∈(λj​,λj+1​).

Significance

The result. Robust Blackwell optimality says that, for polyhedral r-rectangular uncertainty, a decision maker who is patient enough does not need to know the discount factor exactly: one deterministic policy is robust-optimal for all discounts near 111, and the adversary has one matching worst-case factor matrix. This is the robust counterpart of the classical bridge from discounted to average-reward criteria, and it is the natural entry point for a theory of robust MDPs under the long-run average criterion, for which the paper notes no results were known. The robust maximum principle states that an optimal robust policy has the highest worst-case value from every starting state, a comparison that has been used to derive structural properties of robust policies in a healthcare setting (Grand-Clément et al. 2020, cited on p. 13 of the paper).

Formalizing it. The results are proved in the paper; none has a machine-checked proof. The nominal analogues (Blackwell 1962, Theorem 5, and the rationality of the discounted value) are posed on Prove2Me as open targets on a different, nominal model and are not reused here. The formalization also corrects two printed statements: Proposition 6.1 is false for an initial distribution with a zero entry and is posed with p0>0p_0>0p0​>0, and the claim in Appendix G.1 that the factor matrix of a solution is an extreme point is posed in its existential form, the form the proof uses.

Difficulty

The obvious argument for the goal is a pigeonhole over the finitely many candidate pairs (deterministic policies and extreme factor matrices): some pair is optimal for discounts λn→1\lambda_n\to1λn​→1. The step that fails is passing from this sequence to a whole interval. Optimality of a pair is not a comparison between two fixed rational functions of λ\lambdaλ: the worst-case value zλ(π)z_\lambda(\pi)zλ​(π) is a minimum over W\mathcal WW, and optimality quantifies over all randomized policies. A comparison of worst-case values at one discount does not by itself give a comparison of value functions that can be propagated to other discounts. The paper bridges this gap with the robust maximum principle, which turns a worst-case comparison into a statewise comparison of two value functions, each rational in λ\lambdaλ. The maximum principle itself rests on the saddle-point structure of r-rectangular robust MDPs (Theorem 4.2 and Lemma 4.3 of the paper), which a solver must develop.

Formalization scope

All objects live in the definition FactorRMDP.Blackwell.Model. States and actions are finite types, with A\mathbb AA nonempty. Factors are indexed by Fin r, i.e. 0-based. A factor matrix is a function Fin r → S → ℝ. W\mathcal WW is a nonempty convex compact set of factor matrices with stochastic factors. Its factor sets are its projections, and r-rectangularity is the predicate IsRRect, not built into the structure. Rewards are nonnegative and p0∈Δ(S)p_0\in\Delta(\mathbb S)p0​∈Δ(S). The value function is the discounted series, not a fixed point, so no Bellman identity holds by definition. Minima and maxima are stated with IsLeast/IsGreatest, and statements at a fixed discount assume 0<λ<10<\lambda<10<λ<1. Optimality in (1.2) is over stationary randomized policies, as in the paper's proof, and requires both halves: π∗\pi^*π∗ maximizes zλz_\lambdazλ​ and W∗W^*W∗ attains min⁡WR(π∗,W,λ)\min_W R(\pi^*,W,\lambda)minW​R(π∗,W,λ). "Finitely many extreme points" is Mathlib's Set.extremePoints of each factor set. The second sentence of Assumption 2.4 (an optimization oracle) is not formalized.

Explicit choices relative to the printed text:

  • W\mathcal WW nonempty, A\mathbb AA nonempty and λ∈(0,1)\lambda\in(0,1)λ∈(0,1) are stated explicitly.
  • Proposition 6.1 (both parts) assumes p0(s)>0p_0(s)>0p0​(s)>0 for every state. As printed, it is false when p0p_0p0​ vanishes on a state.
  • The extreme-point claim of Appendix G.1 is posed as the existence of an optimal pair with an extreme factor matrix.
  • The goal (Proposition 6.2) keeps p0p_0p0​ arbitrary.

A goal in which the pair (π∗,W∗)(\pi^*,W^*)(π∗,W∗) may depend on λ\lambdaλ would only restate milestone 3 and is not robust Blackwell optimality; the pair is quantified before λ0\lambda_0λ0​.

A complete development needs the saddle-point theory of r-rectangular robust MDPs (shared with the first mission of this series), Neumann series and Cramer's rule for I−tLI-tLI−tL, extreme points of products of compact convex sets, and the finiteness of zeros of a nonzero polynomial. The rationality of discounted values and the extreme-point lemmas are reusable for nominal Blackwell optimality and for other robust models. Proofs of any milestone, of the optional Proposition 6.3, and auxiliary lemmas are welcome.

Selected references

  • V. Goyal and J. Grand-Clément, Robust Markov Decision Process: Beyond Rectangularity, arXiv:1811.00215v8, 2021; Mathematics of Operations Research, 2023. https://arxiv.org/abs/1811.00215, https://doi.org/10.1287/moor.2022.1259
  • D. Blackwell, Discrete Dynamic Programming, Annals of Mathematical Statistics 33, 1962. https://doi.org/10.1214/aoms/1177704593
  • S. Gaubert and J. Gunawardena, A Non-Linear Hierarchy for Discrete Event Dynamical Systems, 1998 (as cited in Goyal and Grand-Clément, reference list, arXiv:1811.00215v8).
  • G. N. Iyengar, Robust Dynamic Programming, Mathematics of Operations Research 30, 2005. https://doi.org/10.1287/moor.1040.0129
  • A. Nilim and L. El Ghaoui, Robust Control of Markov Decision Processes with Uncertain Transition Matrices, Operations Research 53, 2005. https://doi.org/10.1287/opre.1050.0216
  • W. Wiesemann, D. Kuhn and B. Rustem, Robust Markov Decision Processes, Mathematics of Operations Research 38, 2013. https://doi.org/10.1287/moor.1120.0566
  • M. L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming, Wiley, 1994. https://doi.org/10.1002/9780470316887
7 thms1 active userReviewed
Dynamic ProgrammingOperations ResearchOptimization·Captain: mikedeng1

Robust Markov Decision Processes: Beyond Rectangularity 1: Under r-Rectangular Factor Uncertainty, Robust Value Iteration Converges to the Optimal Worst-Case Value, Attained by a Deterministic PolicyResearch Paper

Motivation

A Markov decision process (MDP) is optimized for a transition kernel that, in practice, is estimated from data, and the optimal policy for the estimated kernel can perform poorly under small perturbations of it. The robust MDP approach replaces the single kernel by an uncertainty set of kernels and asks for the policy with the best worst-case expected reward. It goes back to Satia and Lave (1973).

Whether a robust MDP is tractable depends on the shape of the uncertainty set.

  • 2005. Iyengar and Nilim and El Ghaoui treat (s,a)-rectangular sets, in which every row PsaP_{sa}Psa​ is chosen independently. An optimal robust policy is stationary and deterministic, and robust value iteration computes it.
  • 2013. Wiesemann, Kuhn and Rustem treat s-rectangular sets. Optimal robust policies are stationary but may need randomization. For general uncertainty sets, computing an optimal robust policy is strongly NP-hard.
  • 2018. Goh et al. introduce factor-based uncertainty for a medical cost-effectiveness application and evaluate the worst case of a fixed policy.
  • 2021–2023. Goyal and Grand-Clément compute optimal robust policies for factor uncertainty with independent factors. This mission formalizes that result.

Rectangular sets let the adversary choose the transitions out of different states independently, which is conservative when all transitions depend on a few common factors, such as a patient's physiology across health states. Factor uncertainty couples the states, and it is the first coupled model in which an optimal robust policy can still be computed.

Setting

There are finite sets of states S\mathbb SS and actions A\mathbb AA, with S=∣S∣S = |\mathbb S|S=∣S∣ and A=∣A∣A=|\mathbb A|A=∣A∣. The rewards satisfy rsa≥0r_{sa}\ge 0rsa​≥0, p0p_0p0​ is an initial distribution on S\mathbb SS, and λ∈(0,1)\lambda\in(0,1)λ∈(0,1) is the discount factor. Fix rrr factors indexed by [r][r][r] and coefficients usai≥0u^i_{sa}\ge0usai​≥0 with ∑iusai=1\sum_i u^i_{sa}=1∑i​usai​=1. A factor matrix W=(w1,…,wr)W=(w_1,\dots,w_r)W=(w1​,…,wr​) has probability vectors wiw_iwi​ on S\mathbb SS as columns, and it induces the kernel

Psas′=∑i=1rusai wi,s′.P_{sas'}=\sum_{i=1}^r u^i_{sa}\,w_{i,s'} .Psas′​=i=1∑r​usai​wi,s′​.

The uncertainty set is a nonempty convex compact set W\mathcal WW of factor matrices. It is r-rectangular if W=W1×⋯×Wr\mathcal W=\mathcal W^1\times\cdots\times\mathcal W^rW=W1×⋯×Wr, so that each factor is chosen independently of the others.

A stationary policy π∈Π\pi\in\Piπ∈Π gives a distribution πs\pi_sπs​ on A\mathbb AA at each state. It is deterministic if each πs\pi_sπs​ is a point mass. Write rπ=(∑aπsarsa)sr_\pi=(\sum_a\pi_{sa}r_{sa})_srπ​=(∑a​πsa​rsa​)s​ and Tπ=(∑aπsausai)s,iT_\pi=(\sum_a \pi_{sa}u^i_{sa})_{s,i}Tπ​=(∑a​πsa​usai​)s,i​. The decision maker's value function and the expected discounted reward are

vπ,W=∑t≥0λt(TπW⊤)trπ,R(π,W)=p0⊤vπ,W.v^{\pi,W}=\sum_{t\ge0}\lambda^t (T_\pi W^\top)^t r_\pi, \qquad R(\pi,W)=p_0^\top v^{\pi,W}.vπ,W=t≥0∑​λt(Tπ​W⊤)trπ​,R(π,W)=p0⊤​vπ,W.

The policy evaluation problem computes z(π)=min⁡W∈WR(π,W)z(\pi)=\min_{W\in\mathcal W}R(\pi,W)z(π)=minW∈W​R(π,W). The policy improvement problem computes z∗=max⁡π∈Πz(π)z^*=\max_{\pi\in\Pi}z(\pi)z∗=maxπ∈Π​z(π). Let

F(v)s=max⁡a∈A{rsa+λ∑i=1rusaimin⁡wi∈Wiwi⊤v}.F(v)_s=\max_{a\in\mathbb A}\Big\{r_{sa}+\lambda\sum_{i=1}^r u^i_{sa}\min_{w_i\in\mathcal W^i}w_i^\top v\Big\}.F(v)s​=a∈Amax​{rsa​+λi=1∑r​usai​wi​∈Wimin​wi⊤​v}.

Algorithm VI-PI is the iteration v0=0v^0=0v0=0, vk+1=F(vk)v^{k+1}=F(v^k)vk+1=F(vk).

Formalization targets

Goal: Theorem 5.1, by its mathematical content

Assume W\mathcal WW is r-rectangular. Then FFF has a unique fixed point v∗v^*v∗, and

z∗=max⁡π∈Πmin⁡W∈WR(π,W)=p0⊤v∗,vk→v∗.z^*=\max_{\pi\in\Pi}\min_{W\in\mathcal W}R(\pi,W)=p_0^\top v^*,\qquad v^k\to v^* .z∗=π∈Πmax​W∈Wmin​R(π,W)=p0⊤​v∗,vk→v∗.

Write θ(ε)=ε(1−λ)(2λ)−1\theta(\varepsilon)=\varepsilon(1-\lambda)(2\lambda)^{-1}θ(ε)=ε(1−λ)(2λ)−1. If ∥vk+1−vk∥∞<θ(ε)\|v^{k+1}-v^k\|_\infty<\theta(\varepsilon)∥vk+1−vk∥∞​<θ(ε), then ∥vk+1−v∗∥∞≤ε\|v^{k+1}-v^*\|_\infty\le\varepsilon∥vk+1−v∗∥∞​≤ε, and the deterministic policy that VI-PI computes at that iteration is ε\varepsilonε-optimal. The test is passed as soon as λkmax⁡r<θ(ε)\lambda^k\max r<\theta(\varepsilon)λkmaxr<θ(ε).

Milestones

The milestones follow the paper's route to the goal.

  1. The adversary's Bellman recursion (Proposition 3.1).
  2. The identity W⊤vπ,W=βπ,WW^\top v^{\pi,W}=\beta^{\pi,W}W⊤vπ,W=βπ,W (Lemma 4.3).
  3. The resolvent and linear-programming forms of z(π)z(\pi)z(π) (Lemmas A.2, A.3).
  4. The monotone contraction φ(π,⋅)\varphi(\pi,\cdot)φ(π,⋅) (Appendix A).
  5. The reformulation of z(π)z(\pi)z(π) with constraints βi≤min⁡wiwi⊤(rπ+λTπβ)\beta_i\le\min_{w_i}w_i^\top(r_\pi+\lambda T_\pi\beta)βi​≤minwi​​wi⊤​(rπ​+λTπ​β) (Lemma 3.3).
  6. The existence of a deterministic optimal robust policy (Lemma 4.1).
  7. The saddle point and the min-max duality (Theorem 4.2):
max⁡π∈Πmin⁡W∈WR(π,W)=min⁡W∈Wmax⁡π∈ΠR(π,W).\max_{\pi\in\Pi}\min_{W\in\mathcal W}R(\pi,W)=\min_{W\in\mathcal W}\max_{\pi\in\Pi}R(\pi,W).π∈Πmax​W∈Wmin​R(π,W)=W∈Wmin​π∈Πmax​R(π,W).
  1. The monotone contraction FFF (Lemma E.1).
  2. The fixed-point equation (5.1).

Theorem 3.2, the policy-evaluation analogue of the goal (Algorithm VI-PE), is included as a further item.

Significance

The theorem shows that factor uncertainty, in which transitions out of different states are coupled, is as tractable as (s,a)-rectangular uncertainty, provided the factors are independent. An optimal robust policy exists among deterministic stationary policies, the robust and nominal problems share a saddle point, and value iteration on a λ\lambdaλ-contraction computes the robust value. The structure is also what the companion mission builds on: the robust maximum principle and robust Blackwell optimality are derived from (5.1) and Theorem 4.2.

The results are proved in the paper. None of them is formalized: no factor-matrix robust MDP exists in Lean. The platform's robust-MDP items for (s,a)-rectangular sets (Nilim–El Ghaoui, Iyengar) and s-rectangular sets (Wiesemann et al.) are other uncertainty models and do not specialize to this one. A formal proof would also check a printed statement, since one clause of Theorem 4.2 needs correcting (see below).

Difficulty

The obvious argument writes a robust Bellman equation state by state and appeals to the contraction principle. Here that argument fails. Under factor uncertainty the adversary cannot choose the transitions out of each state separately, because one factor wiw_iwi​ enters every row PsaP_{sa}Psa​ with usai>0u^i_{sa}>0usai​>0. Minimizing state by state therefore yields kernels that need not come from a single feasible WWW. The decision maker's equation for vvv does not close on its own.

The step that requires care links two dynamic programs, the decision maker's over S\mathbb SS and the adversary's over [r][r][r], and shows that factor-wise minimization is feasible. That is where r-rectangularity enters. The paper's Appendix A gives an instance in which the factor-wise minimizers are infeasible once a constraint couples the factors.

Formalization scope

  • States and actions are arbitrary finite types. A\mathbb AA is nonempty, and S\mathbb SS is nonempty because p0p_0p0​ is a distribution. Factors are indexed by Fin r (0-based).
  • W\mathcal WW is a set of maps Fin r → S → ℝ whose values are probability vectors. It is nonempty, convex and compact. Nonemptiness is implicit in the paper and explicit here. Wi\mathcal W^iWi is the projection of W\mathcal WW, and r-rectangularity is a hypothesis (IsRRect). Proposition 3.1, Lemmas 4.3, A.2 and A.3, and both contraction statements are stated without it. Lemma 3.3 is stated with it, since it is false without it.
  • Rewards are nonnegative, as on p. 1. The discount satisfies 0<λ<10<\lambda<10<λ<1 in every statement.
  • vπ,Wv^{\pi,W}vπ,W and the adversary's value βπ,W\beta^{\pi,W}βπ,W are discounted sums, never solutions of a Bellman equation by definition. Defining them as fixed points would make Proposition 3.1, Lemma 4.3 and (5.1) true by definition.
  • Every min and max is attained. Statements use IsLeast/IsGreatest, and Lemma A.2 states the invertibility of I−λW⊤TπI-\lambda W^\top T_\piI−λW⊤Tπ​, because Lean's matrix inverse is 000 on singular matrices.
  • FFF is stated with a maximum over actions. The paper (p. 10) says this is equivalent to its maximum over πs∈Δ\pi_s\in\Deltaπs​∈Δ. Lemma E.1 and the contraction of φ\varphiφ are stated on all of RS\mathbb R^SRS and Rr\mathbb R^rRr.
  • Not formalized: the count of arithmetic operations in Theorems 3.2 and 5.1, and the ε\varepsilonε-optimization oracle of Assumption 2.4 that it counts. The iteration count is stated explicitly instead.
  • Corrections. The paper states (4.6) for every robust solution (π∗,W∗)(\pi^*,W^*)(π∗,W∗) with π∗\pi^*π∗ deterministic. That claim is false: a two-state instance has every policy robust-optimal, yet one robust solution is not nominally optimal against its own W∗W^*W∗. Theorem 4.2 and (5.1) are stated in existence form, which is the form the proof of Theorem 5.1 uses.
  • A goal that kept only the convergence vk→v∗v^k\to v^*vk→v∗ would be a different and weaker theorem. The goal also requires z∗=p0⊤v∗z^*=p_0^\top v^*z∗=p0⊤​v∗, the stopping rule, and the ε\varepsilonε-optimality of the computed deterministic policy.

Reusable infrastructure includes the discounted value of a stationary policy for a stochastic matrix, monotone sup-norm contractions with the standard stopping rule, and minimization of linear forms over compact sets. Contributions to any of these are welcome.

Selected references

  • V. Goyal, J. Grand-Clément, Robust Markov Decision Processes: Beyond Rectangularity, Mathematics of Operations Research, 2023; preprint arXiv:1811.00215v8, 2021 (the version formalized). https://arxiv.org/abs/1811.00215 · https://doi.org/10.1287/moor.2022.1259
  • A. Nilim, L. El Ghaoui, Robust control of Markov decision processes with uncertain transition probabilities, Operations Research 53(5), 2005. https://doi.org/10.1287/opre.1050.0216
  • G. Iyengar, Robust dynamic programming, Mathematics of Operations Research 30(2), 2005. https://doi.org/10.1287/moor.1040.0129
  • W. Wiesemann, D. Kuhn, B. Rustem, Robust Markov decision processes, Mathematics of Operations Research 38(1), 2013. https://doi.org/10.1287/moor.1120.0566
  • J. Goh, M. Bayati, S. A. Zenios, S. Singh, D. Moore, Data uncertainty in Markov chains: application to cost-effectiveness analyses of medical innovations, Operations Research 66(3), 2018. https://doi.org/10.1287/opre.2017.1685
  • J. K. Satia, R. L. Lave, Markov decision processes with uncertain transition probabilities, Operations Research 21(3), 1973. https://doi.org/10.1287/opre.21.3.728
  • M. L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming, Wiley, 1994.
12 thms1 active userReviewed
🏆Completed
ProbabilityReinforcement LearningStatistics+1·Captain: Wenqian Xing

Estimation of Treatment Effects Under Nonstationarity via the Truncated Policy Gradient EstimatorResearch Paper

Why short-horizon outcome windows

Randomized experiments (A/B tests) on ride-sharing marketplaces, recommendation systems and digital-health services assign each arriving user to treatment or control. The assignment changes the system state seen by later users, so observations are not independent: this is temporal interference, or carryover. The difference-in-means estimator ignores it, and in Markovian models its bias can be as large as the treatment effect itself (Farias, Li, Peng, Zheng 2022; Hu, Wager 2022). Corrections based on off-policy evaluation or on differences of QQQ-values assume a stationary environment, while real systems have time-varying dynamics and the experimenter observes a single trajectory.

Johari, Peng and Xing propose the truncated policy gradient (TPG) estimator: each instantaneous outcome in the difference-in-means estimator is replaced by the sum of the next k+1k+1k+1 outcomes. The paper shows that this estimates, without bias, the derivative of a truncated policy value, and derives bias and variance bounds, a central limit theorem and a consistent variance estimator in nonstationary Markovian environments. This mission formalizes those statements.

Setting

Time is t=0,1,…,T−1t=0,1,\dots,T-1t=0,1,…,T−1 (the paper's time t+1t+1t+1). The state space SSS is finite and the action Zt∈{0,1}Z_t\in\{0,1\}Zt​∈{0,1} is control or treatment. For each time ttt and action zzz there is a transition matrix PztP_z^tPzt​ with P(Xt+1=x′∣Xt=x,Zt=z)=Pzt(x,x′)\mathbb P(X_{t+1}=x'\mid X_t=x,Z_t=z)=P_z^t(x,x')P(Xt+1​=x′∣Xt​=x,Zt​=z)=Pzt​(x,x′); the matrices may change arbitrarily with ttt, which is the nonstationarity. The initial state X0X_0X0​ has distribution ρ\rhoρ. The reward YtY_tYt​ is drawn from a distribution R(⋅∣Xt,Zt)R(\cdot\mid X_t,Z_t)R(⋅∣Xt​,Zt​) on R\mathbb RR, independently of everything else given (Xt,Zt)(X_t,Z_t)(Xt​,Zt​), and R([−M,M]∣x,z)=1R([-M,M]\mid x,z)=1R([−M,M]∣x,z)=1 for all x,zx,zx,z. Write r(x,z)=∫y R(dy∣x,z)r(x,z)=\int y\,R(dy\mid x,z)r(x,z)=∫yR(dy∣x,z).

The policy πθ\pi_\thetaπθ​ assigns treatment with probability θ\thetaθ at every time, independently of everything else. π1\pi_1π1​ is all-treatment, π0\pi_0π0​ is all-control, and the experiment runs π1/2\pi_{1/2}π1/2​. Write Pθt=θP1t+(1−θ)P0tP_\theta^t=\theta P_1^t+(1-\theta)P_0^tPθt​=θP1t​+(1−θ)P0t​ and Eθ\mathbb E_\thetaEθ​ for expectation under πθ\pi_\thetaπθ​. The value function and the global average treatment effect (GATE) are

J(θ)=1T∑t=0T−1Eθ[Yt],τ=J(1)−J(0).J(\theta)=\frac1T\sum_{t=0}^{T-1}\mathbb E_\theta[Y_t],\qquad \tau=J(1)-J(0).J(θ)=T1​t=0∑T−1​Eθ​[Yt​],τ=J(1)−J(0).

The TPG estimator with truncation size kkk is

τ^k=1T∑u=0T−1(1{Zu=1}1/2−1{Zu=0}1/2)∑t=umin⁡(u+k, T−1)Yt,\widehat\tau_k=\frac1T\sum_{u=0}^{T-1}\Bigl(\frac{\mathbf 1\{Z_u=1\}}{1/2}-\frac{\mathbf 1\{Z_u=0\}}{1/2}\Bigr)\sum_{t=u}^{\min(u+k,\,T-1)}Y_t ,τk​=T1​u=0∑T−1​(1/21{Zu​=1}​−1/21{Zu​=0}​)t=u∑min(u+k,T−1)​Yt​,

so τ^0\widehat\tau_0τ0​ is the difference-in-means estimator.

Two quantities control the bounds. The mixing assumption (Assumption 1) asks for γ∈(0,1)\gamma\in(0,1)γ∈(0,1) with ∥f′Pzt−fPzt∥TV≤γ ∥f′−f∥TV\|f'P_z^t-fP_z^t\|_{\mathrm{TV}}\le\gamma\,\|f'-f\|_{\mathrm{TV}}∥f′Pzt​−fPzt​∥TV​≤γ∥f′−f∥TV​ for all t<Tt<Tt<T, all zzz and all probability vectors f,f′f,f'f,f′ on SSS, where ∥f−f′∥TV=12∥f−f′∥1\|f-f'\|_{\mathrm{TV}}=\tfrac12\|f-f'\|_1∥f−f′∥TV​=21​∥f−f′∥1​. The kernel deviation is δ=max⁡t<Tmax⁡x∥P1t(x,⋅)−P0t(x,⋅)∥TV\delta=\max_{t<T}\max_{x}\|P_1^t(x,\cdot)-P_0^t(x,\cdot)\|_{\mathrm{TV}}δ=maxt<T​maxx​∥P1t​(x,⋅)−P0t​(x,⋅)∥TV​.

Formalization targets

Goal: Theorem 1, bias and variance of the TPG estimator

There are absolute constants C1,C2>0C_1,C_2>0C1​,C2​>0 such that for every environment, every T≥1T\ge1T≥1, every 0≤k≤T0\le k\le T0≤k≤T and every γ∈(0,1)\gamma\in(0,1)γ∈(0,1) and MMM as above,

∣E1/2[τ^k]−τ∣≤C1(1−γk(1−γ)2 δ2M+γk1−γ δM),Var⁡1/2(τ^k)≤C2((k+1)2M2T+γ (k+1)M2T(1−γ)).\bigl|\mathbb E_{1/2}[\widehat\tau_k]-\tau\bigr|\le C_1\Bigl(\frac{1-\gamma^k}{(1-\gamma)^2}\,\delta^2M+\frac{\gamma^k}{1-\gamma}\,\delta M\Bigr),\qquad \operatorname{Var}_{1/2}(\widehat\tau_k)\le C_2\Bigl(\frac{(k+1)^2M^2}{T}+\frac{\gamma\,(k+1)M^2}{T(1-\gamma)}\Bigr).​E1/2​[τk​]−τ​≤C1​((1−γ)21−γk​δ2M+1−γγk​δM),Var1/2​(τk​)≤C2​(T(k+1)2M2​+T(1−γ)γ(k+1)M2​).

The constants are left unspecified, as in the paper's O(⋅)O(\cdot)O(⋅) statement, so an improvement of the constants does not invalidate the goal.

Explicit constants: Propositions 5 and 6

For data collected under πθ\pi_\thetaπθ​ with θ∈(0,1)\theta\in(0,1)θ∈(0,1), the paper gives both bounds with explicit constants, for the TPG estimator and for a reward-corrected variant. At θ=1/2\theta=1/2θ=1/2, Proposition 5 reads

∣E[τ^k]−τ∣≤2δMγk1−γ+2δ2M Ψk(γ),Var⁡(τ^k)≤4M2(k+1)T(2k+3+4γ1−γ),\bigl|\mathbb E[\widehat\tau_k]-\tau\bigr|\le\frac{2\delta M\gamma^k}{1-\gamma}+2\delta^2M\,\Psi_k(\gamma),\qquad \operatorname{Var}(\widehat\tau_k)\le\frac{4M^2(k+1)}{T}\Bigl(2k+3+\frac{4\gamma}{1-\gamma}\Bigr),​E[τk​]−τ​≤1−γ2δMγk​+2δ2MΨk​(γ),Var(τk​)≤T4M2(k+1)​(2k+3+1−γ4γ​),

with Ψk(γ)=∑j=0k−2(j+1)γj\Psi_k(\gamma)=\sum_{j=0}^{k-2}(j+1)\gamma^jΨk​(γ)=∑j=0k−2​(j+1)γj.

Policy-gradient identities: Propositions 1–4

JJJ and its truncated analogue JkJ_kJk​ are differentiable at 1/21/21/2, with derivative equal to the horizon average of differences of (kkk-truncated) QQQ-values, and Eθ[τ^θ,kTPG]\mathbb E_\theta[\widehat\tau^{\mathrm{TPG}}_{\theta,k}]Eθ​[τθ,kTPG​] equals the corresponding derivative for every θ∈(0,1)\theta\in(0,1)θ∈(0,1). Without the mixing assumption, the untruncated estimator satisfies ∣E[τ^T]−τ∣≤C T2δ2M|\mathbb E[\widehat\tau_T]-\tau|\le C\,T^2\delta^2M∣E[τT​]−τ∣≤CT2δ2M and Var⁡(τ^T)≤2M2(T+1)\operatorname{Var}(\widehat\tau_T)\le2M^2(T+1)Var(τT​)≤2M2(T+1).

Asymptotics: Theorem 2, Lemmas 1–5 and Corollary 1

For fixed kkk, under mixing at all times, a positive limit σk2\sigma_k^2σk2​ of the normalized variance (Assumption 2) and a Cesàro mean-stability condition (Assumption 3),

T (τ^k−E[τ^k])⇒N(0,σk2),Ω^T→pσk2,\sqrt T\,\bigl(\widehat\tau_k-\mathbb E[\widehat\tau_k]\bigr)\Rightarrow\mathcal N(0,\sigma_k^2),\qquad \widehat\Omega_T\xrightarrow{p}\sigma_k^2,T​(τk​−E[τk​])⇒N(0,σk2​),ΩT​p​σk2​,

where Ω^T\widehat\Omega_TΩT​ is a Newey–West estimator with bandwidth LT→∞L_T\to\inftyLT​→∞, LT=O(T1/3)L_T=O(T^{1/3})LT​=O(T1/3). The supporting statements are a covariance inequality under strong mixing, the bound α(h)≤γh−1\alpha(h)\le\gamma^{h-1}α(h)≤γh−1 on the mixing coefficients of the experimental process and its consequence for the summands of the estimator, the central limit theorem alone, and consistency of the ideal and of the feasible variance estimators.

What the results give

Theorem 1 separates two error sources: a term of second order in δ\deltaδ that does not grow with TTT, and a term that decays geometrically in kkk. Together with Proposition 2 it shows what truncation buys: a variance bound of order (k+1)2/T(k+1)^2/T(k+1)2/T instead of order TTT, at the price of a bias term of order γk\gamma^kγk. Theorem 2 supplies standard errors for the estimator from a single nonstationary trajectory.

All results of the paper are proved on paper; none has a machine-checked proof. The work of this mission is the formal proofs of the stated results. The definition layer (a finite-state nonstationary Markov decision process with real rewards, its trajectory law under time-varying randomization, total-variation contraction, strong-mixing windows, a Newey–West estimator) can be reused for other results on experiments with Markovian interference.

Where the difficulty lies

The first idea is to bound ∣J′(1/2)−(J(1)−J(0))∣|J'(1/2)-(J(1)-J(0))|∣J′(1/2)−(J(1)−J(0))∣ by the second derivative of JJJ on [0,1][0,1][0,1]. Every pair of times contributes to that second derivative, so this route gives the T2δ2MT^2\delta^2MT2δ2M of Proposition 2 and nothing uniform in the horizon. Theorem 1 is uniform in TTT, and its bound has no term of first order in δ\deltaδ beyond the truncation term; Proposition 5 shows that such a term is present at every allocation θ≠1/2\theta\ne1/2θ=1/2. Formal proofs have to handle ordered products of time-varying stochastic matrices and signed measures of total mass zero, with a contraction that is assumed one step at a time.

The variance bound and Theorem 2 concern a dependent, non-identically distributed sequence observed once. Mathlib has no central limit theorem for nonhomogeneous Markov chains and no covariance inequality for strongly mixing sequences; the paper relies on Arlotto and Steele (2016) for the former and on an inequality from Doukhan (1994) for the latter.

Formalization scope

TPG.Env S packages (P,ρ,R)(P,\rho,R)(P,ρ,R) for a finite type S. Kernels are real matrices with nonnegative entries and rows summing to one, indexed by all t∈Nt\in\mathbb Nt∈N; a statement with horizon TTT uses only the times below TTT. A trajectory is a function Fin T → S × Bool × ℝ, and Env.trajLaw is its law for an arbitrary sequence of treatment probabilities, built as a finite mixture over state and action paths of product reward laws. JJJ, τ\tauτ, JkJ_kJk​ and the QQQ-values are defined from this law, and expectations given {Zu=z}\{Z_u=z\}{Zu​=z} are taken under the conditional measure. Rewards are bounded through the hypothesis E.RewardBound M; the standing assumptions of the paper enter each theorem as hypotheses, and S carries the discrete measurable structure.

Statements require T≥1T\ge1T≥1 and 0≤k≤T0\le k\le T0≤k≤T, as in the paper. Asymptotic statements compute the law of each statistic under the horizon-TTT measure: convergence in distribution is tested against bounded continuous functions, and convergence in probability is stated for each ε>0\varepsilon>0ε>0. The limit σk2\sigma_k^2σk2​ is a parameter constrained by Assumption 2. For θ∉[0,1]\theta\notin[0,1]θ∈/[0,1] the law of πθ\pi_\thetaπθ​ is not a probability law and no statement depends on it. The hypotheses are satisfiable for every TTT, kkk and γ\gammaγ: an environment with one state and a nonzero constant reward satisfies all of them, including Assumptions 2 and 3, so no statement is vacuous.

Not included: Definition 1 and Lemma 6 of the paper (Dobrushin coefficients and the Arlotto–Steele theorem on general Borel state spaces), the stationary difference-in-QQQ discussion of Appendix C, the constant-reward example showing that the variance order is attained, and the numerical studies. Formal statements of those, and auxiliary lemmas about Env.trajLaw, ordered matrix products and total-variation contraction, are welcome contributions.

Selected references

  • R. Johari, T. Peng, W. Xing, Estimation of Treatment Effects Under Nonstationarity via the Truncated Policy Gradient Estimator, arXiv preprint, 2025–2026. https://arxiv.org/abs/2506.05308
  • V. F. Farias, A. A. Li, T. Peng, A. Zheng, Markovian Interference in Experiments, Advances in Neural Information Processing Systems 35, 2022. https://arxiv.org/abs/2206.02371
  • Y. Hu, S. Wager, Switchback Experiments under Geometric Mixing, arXiv preprint, 2022. https://arxiv.org/abs/2209.00197
  • A. Arlotto, J. M. Steele, A Central Limit Theorem for Temporally Nonhomogeneous Markov Chains with Applications to Dynamic Programming, Mathematics of Operations Research 41(4), 2016. https://doi.org/10.1287/moor.2016.0784
  • P. Doukhan, Mixing: Properties and Examples, Lecture Notes in Statistics 85, Springer, 1994. https://doi.org/10.1007/978-1-4612-2642-0
17 thms1 active userReviewed
Control TheoryStochastic Systems·Captain: mikedeng1

Strong and Weak Equilibria for Time-Inconsistent Stochastic Control in Continuous Time 2: Under Pseudo-Exponential Discounting, a Two-State Generator Is a Weak but Not a Strong EquilibriumResearch Paper

Motivation

An agent whose preferences change over time cannot simply optimize once: a plan that is optimal today may be abandoned tomorrow by the agent's own "future selves". This is time inconsistency, and its standard source is non-exponential discounting, studied since Strotz (1955). The usual response, going back to Strotz and made rigorous in continuous time by Ekeland and Lazrak (2006), is to look for an equilibrium strategy: a strategy from which no future self wants to deviate. In continuous time "no incentive to deviate" is encoded through a first-order condition, which Huang and Zhou call a weak equilibrium.

Björk, Khapko and Murgoci (2017, Remark 3.5) observed that this first-order condition need not express the absence of an incentive to deviate: it only says that a derivative vanishes or is nonnegative, and a stationary point need not be a maximum. Huang and Zhou (Math. Oper. Res. 46(2), 2021; read here as arXiv:1809.09243v3) introduce a strong equilibrium, which requires the agent's payoff to be no better after any short deviation, and study the two notions for continuous-time Markov chains with controlled generators. This mission formalizes their main example (Example 4.3): a concrete two-state model in which a weak equilibrium fails to be strong, so that an agent in one state strictly gains from deviating on every short interval.

Setting

The state space is S={1,…,N}S=\{1,\dots,N\}S={1,…,N}. A generator is a matrix Q∈RN×NQ\in\mathbb R^{N\times N}Q∈RN×N whose iii-th row QiQ_iQi​ lies in Ei={q:qj≥0 (j≠i), qi=−∑j≠iqj}E_i=\{q: q_j\ge0\ (j\ne i),\ q_i=-\sum_{j\ne i}q_j\}Ei​={q:qj​≥0 (j=i), qi​=−∑j=i​qj​}. Admissible rows form sets Di⊆EiD_i\subseteq E_iDi​⊆Ei​, and the control space is Q={Q:Qi∈Di ∀i}\mathcal Q=\{Q: Q_i\in D_i\ \forall i\}Q={Q:Qi​∈Di​ ∀i}. A payoff function f(t,i,q)f(t,i,q)f(t,i,q) gives the payment rate at time difference ttt in state iii when row qqq is in force. The expected payoff of using QQQ from state iii is

F(i,Q)=Ei,Q[∫0∞f(t,Xt,QXt) dt]=∫0∞∑j(etQ)ij f(t,j,Qj) dt.F(i,Q)=\mathbb E_{i,Q}\Big[\int_0^\infty f(t,X_t,Q_{X_t})\,dt\Big]=\int_0^\infty\sum_j (e^{tQ})_{ij}\,f(t,j,Q_j)\,dt .F(i,Q)=Ei,Q​[∫0∞​f(t,Xt​,QXt​​)dt]=∫0∞​j∑​(etQ)ij​f(t,j,Qj​)dt.

The concatenation Q⊗εQ∗Q\otimes_\varepsilon Q^*Q⊗ε​Q∗ uses QQQ on [0,ε][0,\varepsilon][0,ε] and Q∗Q^*Q∗ afterwards. A generator Q∗∈QQ^*\in\mathcal QQ∗∈Q is a weak equilibrium if lim inf⁡ε↓0[F(i,Q∗)−F(i,Q⊗εQ∗)]/ε≥0\liminf_{\varepsilon\downarrow0}[F(i,Q^*)-F(i,Q\otimes_\varepsilon Q^*)]/\varepsilon\ge0liminfε↓0​[F(i,Q∗)−F(i,Q⊗ε​Q∗)]/ε≥0 for all i∈Si\in Si∈S, Q∈QQ\in\mathcal QQ∈Q, and a strong equilibrium if for all i,Qi,Qi,Q there is ε>0\varepsilon>0ε>0 with F(i,Q∗)≥F(i,Q⊗ε′Q∗)F(i,Q^*)\ge F(i,Q\otimes_{\varepsilon'}Q^*)F(i,Q∗)≥F(i,Q⊗ε′​Q∗) for every ε′∈(0,ε]\varepsilon'\in(0,\varepsilon]ε′∈(0,ε].

The paper's first-order and second-order quantities are

ΓQ∗(Qi)=f(0,i,Qi)+Qi⋅F(Q∗),ΛQ∗(i,Q)=ft(0,i,Qi)+Qi⋅(2G(Q∗)+ΓQ∗(Q)),\Gamma^{Q^*}(Q_i)=f(0,i,Q_i)+Q_i\cdot F(Q^*),\qquad \Lambda^{Q^*}(i,Q)=f_t(0,i,Q_i)+Q_i\cdot\big(2G(Q^*)+\Gamma^{Q^*}(Q)\big),ΓQ∗(Qi​)=f(0,i,Qi​)+Qi​⋅F(Q∗),ΛQ∗(i,Q)=ft​(0,i,Qi​)+Qi​⋅(2G(Q∗)+ΓQ∗(Q)),

with GGG the payoff computed with the time derivative ftf_tft​ in place of fff.

In the two-state model of §4, S={1,2}S=\{1,2\}S={1,2}, Di=EiD_i=E_iDi​=Ei​, every generator is Q∼(a,b)=[−aab−b]Q\sim(a,b)=\begin{bmatrix}-a&a\\ b&-b\end{bmatrix}Q∼(a,b)=[−ab​a−b​] with a,b≥0a,b\ge0a,b≥0, and f(t,1,(−a,a))=δ(t)g1(a)f(t,1,(-a,a))=\delta(t)g_1(a)f(t,1,(−a,a))=δ(t)g1​(a), f(t,2,(b,−b))=δ(t)g2(b)f(t,2,(b,-b))=\delta(t)g_2(b)f(t,2,(b,−b))=δ(t)g2​(b) with the pseudo-exponential discount δ(t)=λe−ρt+(1−λ)e−ρ′t\delta(t)=\lambda e^{-\rho t}+(1-\lambda)e^{-\rho't}δ(t)=λe−ρt+(1−λ)e−ρ′t. Example 4.3 takes λ=12\lambda=\tfrac12λ=21​, ρ=1\rho=1ρ=1, ρ′=2\rho'=2ρ′=2, g1(a)=−a2g_1(a)=-a^2g1​(a)=−a2, and g2(b)=193144+56bg_2(b)=\tfrac{193}{144}+\tfrac56bg2​(b)=144193​+65​b for b<712b<\tfrac7{12}b<127​, g2(b)=2−(1−b)2g_2(b)=2-(1-b)^2g2​(b)=2−(1−b)2 for b≥712b\ge\tfrac7{12}b≥127​.

Formalization targets

Goal: Example 4.3

Q∗∼(512,712) is a weak equilibrium and not a strong equilibrium.Q^*\sim\Big(\frac5{12},\frac7{12}\Big)\ \text{is a weak equilibrium and not a strong equilibrium.}Q∗∼(125​,127​) is a weak equilibrium and not a strong equilibrium.

The statement is fully explicit and has no hypotheses.

Milestones

  1. (4.3)–(4.4): closed forms of F1−F2F_1-F_2F1​−F2​ and G1−G2G_1-G_2G1​−G2​ for Q∼(a,b)Q\sim(a,b)Q∼(a,b) under pseudo-exponential discounting.
  2. (4.5), (4.6), (4.8): ΓQ∗(Q1)\Gamma^{Q^*}(Q_1)ΓQ∗(Q1​), ΓQ∗(Q2)\Gamma^{Q^*}(Q_2)ΓQ∗(Q2​) and ΛQ∗(2,Q)\Lambda^{Q^*}(2,Q)ΛQ∗(2,Q) in the two-state model.
  3. Theorem 3.1: under (2.3) and (3.3), Q∗Q^*Q∗ is weak iff ΓQ∗(Qi∗)≥ΓQ∗(Qi)\Gamma^{Q^*}(Q^*_i)\ge\Gamma^{Q^*}(Q_i)ΓQ∗(Qi∗​)≥ΓQ∗(Qi​) for all (i,Q)(i,Q)(i,Q).
  4. (4.11): at Q∗Q^*Q∗, Γ1\Gamma_1Γ1​ is maximized uniquely at a∗a^*a∗ and arg max⁡b≥0Γ2(b)=[0,7/12]\operatorname{arg\,max}_{b\ge0}\Gamma_2(b)=[0,7/12]argmaxb≥0​Γ2​(b)=[0,7/12].
  5. Proposition 3.3: the second-order test on the set RRR of equality cases.
  6. (4.12): Λ2(a∗,b)=−b/12−579/288\Lambda_2(a^*,b)=-b/12-579/288Λ2​(a∗,b)=−b/12−579/288 on [0,7/12][0,7/12][0,7/12], so Λ2(a∗,b∗)<Λ2(a∗,b)\Lambda_2(a^*,b^*)<\Lambda_2(a^*,b)Λ2​(a∗,b∗)<Λ2​(a∗,b) for b∈[0,7/12)b\in[0,7/12)b∈[0,7/12).

Further statements

Remark 4.1: for Q∼(5/12,b)Q\sim(5/12,b)Q∼(5/12,b), b∈[0,7/12)b\in[0,7/12)b∈[0,7/12), F(2,Q∗)<F(2,Q⊗εQ∗)F(2,Q^*)<F(2,Q\otimes_\varepsilon Q^*)F(2,Q∗)<F(2,Q⊗ε​Q∗) for all small ε>0\varepsilon>0ε>0. And (4.13): the equation 56≤2a=12(11+a+12+a)(a2+193144)\tfrac56\le2a=\tfrac12(\tfrac1{1+a}+\tfrac1{2+a})(a^2+\tfrac{193}{144})65​≤2a=21​(1+a1​+2+a1​)(a2+144193​) has a unique solution aˉ≥0\bar a\ge0aˉ≥0, and Qˉ∗∼(aˉ,0)\bar Q^*\sim(\bar a,0)Qˉ​∗∼(aˉ,0) is a strong equilibrium.

Significance

The example shows that the weak equilibrium concept, the standard one in the time-inconsistency literature since Ekeland and Lazrak, admits strategies that an agent would abandon: Remark 4.1 exhibits a deviation that strictly increases the payoff on every short interval. Together with (4.13), it shows that in one and the same model a weak-but-not-strong equilibrium and a strong equilibrium coexist, so the choice of equilibrium concept changes which strategies are predicted. The example is the paper's justification for introducing strong equilibria.

The result is proved in the paper, through the first-order characterization (Theorem 3.1), the second-order test (Proposition 3.3, resting on the second-order expansion Lemma 3.2) and explicit computations in the two-state model. None of it has been machine-checked. A formal proof would provide checked versions of Theorem 3.1 and Proposition 3.3 for finite-state chains, reusable for any other example, and an independent check of the arithmetic of Example 4.3, where the source contains at least one misprinted formula.

Difficulty

Both conjuncts of the goal concern the small-ε\varepsilonε behaviour of F(2,Q⊗εQ∗)F(2,Q\otimes_\varepsilon Q^*)F(2,Q⊗ε​Q∗), an integral of products of matrix exponentials. Weakness reduces to a first-order comparison by Theorem 3.1, but its proof needs the expansion (3.6) with an o(ε)o(\varepsilon)o(ε) remainder controlled uniformly enough to pass under the integral over [0,∞)[0,\infty)[0,∞). Failure of strong equilibrium cannot be read off the first-order term: at b∗=7/12b^*=7/12b∗=7/12 the maximizer of Γ2\Gamma_2Γ2​ is not unique, every b∈[0,7/12)b\in[0,7/12)b∈[0,7/12) ties at first order, and the decision moves to the second-order coefficient Λ\LambdaΛ of Lemma 3.2, whose derivation needs a second-order expansion of the concatenated payoff. The regularity hypotheses of Lemma 3.2 and of (3.3) must also be verified for the example's discount function.

Formalization scope

States are Fin N; the paper's state kkk is index k−1k-1k−1, so in the two-state model state 1 is 0 : Fin 2 and state 2 is 1 : Fin 2. Expectations are written through the transition matrices etQe^{tQ}etQ (NormedSpace.exp (t • Q)) and Fubini, exactly as the paper computes on p. 23; no Markov process is constructed. Payoffs are Bochner integrals over (0,∞)(0,\infty)(0,∞). The weak-equilibrium condition is written as "for every η>0\eta>0η>0, eventually as ε↓0\varepsilon\downarrow0ε↓0 the quotient is ≥−η\ge-\eta≥−η", which is lim inf⁡≥0\liminf\ge0liminf≥0 without a real-valued liminf. The norm in (2.3) is the sup norm, equivalent to the paper's Euclidean norm because (2.3) quantifies over every c>0c>0c>0.

Standing assumptions: Di⊆EiD_i\subseteq E_iDi​⊆Ei​, continuity of fff in ttt (2.2) and the integrability condition (2.3). Theorem 3.1 adds the shift bound (3.3)–(3.5); Proposition 3.3 assumes the conditions of Lemma 3.2, with the derivative ftf_tft​ tied to fff. The closed forms (4.3)–(4.4) assume ρ,ρ′>0\rho,\rho'>0ρ,ρ′>0 (the page allows ρ,ρ′≥0\rho,\rho'\ge0ρ,ρ′≥0, but with a zero rate the payoff integral diverges); the algebraic identities (4.5)–(4.8) keep the page's ρ,ρ′≥0\rho,\rho'\ge0ρ,ρ′≥0; Example 4.3 has ρ=1\rho=1ρ=1, ρ′=2\rho'=2ρ′=2. Statement (4.11) gives Γ2(b)=g2(b)−56b\Gamma_2(b)=g_2(b)-\tfrac56bΓ2​(b)=g2​(b)−65​b and the argmax; the page's case formula for b≥7/12b\ge7/12b≥7/12 contains a misprint and is not used.

No trivializing reading is available: the goal has no hypotheses, FFF is built from etQe^{tQ}etQ and the concatenation uses QQQ before and Q∗Q^*Q∗ after ε\varepsilonε, so neither conjunct holds by a junk value. Integrability of every payoff follows from (2.3) and is never assumed.

Theorem 3.1 and the second-order package are also drafted in the companion mission on existence of equilibria; they are restated here in the sub-namespace StrongWeakEq.TwoState. The goal needs the verification of (2.3), (3.3) and the Lemma 3.2 conditions for the example (convexity of δ\deltaδ gives (3.15)), the closed forms of etQe^{tQ}etQ for two-state generators, and the expansions of the paper. Contributions of proofs of the general results (Theorem 3.1, Proposition 3.3) are welcome independently of the example.

Selected references

  • Y.-J. Huang and Z. Zhou, Strong and Weak Equilibria for Time-Inconsistent Stochastic Control in Continuous Time, Math. Oper. Res. 46(2), 2021; arXiv:1809.09243v3. https://arxiv.org/abs/1809.09243, https://doi.org/10.1287/moor.2020.1066
  • I. Ekeland and A. Lazrak, Being serious about non-commitment: subgame perfect equilibrium in continuous time, 2006. https://arxiv.org/abs/math/0604264
  • T. Björk, M. Khapko and A. Murgoci, On time-inconsistent stochastic control in continuous time, Finance Stoch. 21 (2017), pp. 331–360.
  • R. H. Strotz, Myopia and inconsistency in dynamic utility maximization, Rev. Econ. Stud. 23 (1955), pp. 165–180.
11 thms1 active userReviewed
Control TheoryDynamic ProgrammingStochastic Systems·Captain: mikedeng1

Strong and Weak Equilibria for Time-Inconsistent Stochastic Control in Continuous Time 4: Limits of Equilibria of the Time-Discretized Problems Are Weak Equilibria in Continuous TimeResearch Paper

Motivation

A decision maker is time-inconsistent when the plan that is optimal today stops being optimal tomorrow. The standard source is non-exponential discounting: a reward ttt units of time ahead is weighted by δ(t)\delta(t)δ(t), and unless δ\deltaδ is exponential the relative weight of two future dates changes as time passes. Hyperbolic and pseudo-exponential discount functions, which fit experimental and group-decision data better than exponential ones, all have this property. For such an agent "optimal control" is not a meaningful target, and the literature instead looks for equilibrium policies: plans that no current self wants to deviate from, given that all future selves follow the plan.

In discrete time the definition is unambiguous: a policy is an equilibrium if deviating at the current step alone does not help. In continuous time there is no "current step", and the usual definition, due to Ekeland and Lazrak (2006), compares the payoff of the plan with that of a deviation on a short interval [0,ε][0,\varepsilon][0,ε] through a first-order condition as ε↓0\varepsilon\downarrow0ε↓0. Huang and Zhou (arXiv:1809.09243, Math. Oper. Res. 2021) call this a weak equilibrium, introduce a strong equilibrium that requires the deviation to be unprofitable for every small ε\varepsilonε, and study both for a controlled finite-state Markov chain.

A natural question is whether continuous-time equilibria are the limits of the discrete-time ones as the time step shrinks. This mission formalizes the paper's answer (§5.1): limits of discrete equilibria are always weak equilibria. The paper's Example 5.1 shows they need not be strong.

Setting

The state space is S={1,…,N}S=\{1,\dots,N\}S={1,…,N}. A continuous-time Markov chain on SSS is controlled through its generator Q=(qij)Q=(q_{ij})Q=(qij​): in state iii the row QiQ_iQi​ is chosen from an admissible set Di⊆Ei={q:qj≥0 (j≠i), qi=−∑j≠iqj}D_i\subseteq E_i=\{q: q_j\ge0\ (j\ne i),\ q_i=-\sum_{j\ne i}q_j\}Di​⊆Ei​={q:qj​≥0 (j=i), qi​=−∑j=i​qj​}, and the control space is Q={Q:Qi∈Di ∀i}\mathcal Q=\{Q: Q_i\in D_i\ \forall i\}Q={Q:Qi​∈Di​ ∀i}. A payoff function f(t,i,q)f(t,i,q)f(t,i,q) gives the reward at time-distance ttt from now, in state iii, under row qqq. The standing assumptions are continuity of t↦f(t,i,q)t\mapsto f(t,i,q)t↦f(t,i,q) on [0,∞)[0,\infty)[0,∞) and, for each c>0c>0c>0, integrability over [0,∞)[0,\infty)[0,∞) of sup⁡i, q∈Di, ∥q∥≤c∣f(t,i,q)∣\sup_{i,\,q\in D_i,\,\|q\|\le c}|f(t,i,q)|supi,q∈Di​,∥q∥≤c​∣f(t,i,q)∣ (condition (2.3)). The expected payoff from state iii is

F(i,Q)=Ei,Q[∫0∞f(t,Xt,QXt) dt]=∫0∞∑j(etQ)ijf(t,j,Qj) dt,F(i,Q)=\mathbb E_{i,Q}\Big[\int_0^\infty f(t,X_t,Q_{X_t})\,dt\Big]=\int_0^\infty\sum_j(e^{tQ})_{ij}f(t,j,Q_j)\,dt,F(i,Q)=Ei,Q​[∫0∞​f(t,Xt​,QXt​​)dt]=∫0∞​j∑​(etQ)ij​f(t,j,Qj​)dt,

and F(Q)=(F(1,Q),…,F(N,Q))F(Q)=(F(1,Q),\dots,F(N,Q))F(Q)=(F(1,Q),…,F(N,Q)). With Q⊗εQ∗Q\otimes_\varepsilon Q^*Q⊗ε​Q∗ the generator that uses QQQ on [0,ε][0,\varepsilon][0,ε] and Q∗Q^*Q∗ afterwards, Q∗∈QQ^*\in\mathcal QQ∗∈Q is a weak equilibrium if lim inf⁡ε↓0(F(i,Q∗)−F(i,Q⊗εQ∗))/ε≥0\liminf_{\varepsilon\downarrow0}\big(F(i,Q^*)-F(i,Q\otimes_\varepsilon Q^*)\big)/\varepsilon\ge0liminfε↓0​(F(i,Q∗)−F(i,Q⊗ε​Q∗))/ε≥0 for all iii and Q∈QQ\in\mathcal QQ∈Q. The first-order quantity is ΓQ∗(q)=f(0,i,q)+q⋅F(Q∗)\Gamma^{Q^*}(q)=f(0,i,q)+q\cdot F(Q^*)ΓQ∗(q)=f(0,i,q)+q⋅F(Q∗).

Discretization. Fix a mesh δn>0\delta_n>0δn​>0. A transition matrix uuu (rows in the probability simplex P\mathfrak PP) has generator Qu,n=(u−I)/δnQ^{u,n}=(u-I)/\delta_nQu,n=(u−I)/δn​, the admissible matrices are An={u:Qu,n∈Q}\mathcal A^n=\{u: Q^{u,n}\in\mathcal Q\}An={u:Qu,n∈Q}, and the discrete payoff is κn(k,i,α)=f(kδn,i,(α−ei)/δn) δn\kappa^n(k,i,\alpha)=f\big(k\delta_n,i,(\alpha-e_i)/\delta_n\big)\,\delta_nκn(k,i,α)=f(kδn​,i,(α−ei​)/δn​)δn​. The discrete value is Vn(i,u)=Ei,u[∑k≥0κn(k,Xk,uXk)]V^n(i,u)=\mathbb E_{i,u}\big[\sum_{k\ge0}\kappa^n(k,X_k,u_{X_k})\big]Vn(i,u)=Ei,u​[∑k≥0​κn(k,Xk​,uXk​​)], and un∈Anu^n\in\mathcal A^nun∈An is a discrete equilibrium if Vn(i,un)≥Vn(i,u⊗1un)V^n(i,u^n)\ge V^n(i,u\otimes_1u^n)Vn(i,un)≥Vn(i,u⊗1​un) for all iii and u∈Anu\in\mathcal A^nu∈An, where u⊗1unu\otimes_1u^nu⊗1​un uses uuu at time 000 and unu^nun afterwards. The continuation values are Hin(un)=Ei,un[∑k≥0κn(k+1,Xk,uXkn)]H^n_i(u^n)=\mathbb E_{i,u^n}\big[\sum_{k\ge0}\kappa^n(k+1,X_k,u^n_{X_k})\big]Hin​(un)=Ei,un​[∑k≥0​κn(k+1,Xk​,uXk​n​)]. Theorem 5.2 additionally assumes (5.8): for some T>0T>0T>0, t↦∣f(t,i,q)∣t\mapsto|f(t,i,q)|t↦∣f(t,i,q)∣ is nonincreasing on [T,∞)[T,\infty)[T,∞).

Formalization targets

Goal: Theorem 5.2

Let δn↓0\delta_n\downarrow0δn​↓0, let unu^nun be an equilibrium of the problem with mesh δn\delta_nδn​, Qn=Qun,nQ^n=Q^{u^n,n}Qn=Qun,n, and suppose f(⋅,i,⋅)f(\cdot,i,\cdot)f(⋅,i,⋅) is continuous on [0,∞)×Di[0,\infty)\times D_i[0,∞)×Di​. If Qnk→Q∗∈QQ^{n_k}\to Q^*\in\mathcal QQnk​→Q∗∈Q along a subsequence, then

ΓQ∗(Qi∗)≥ΓQ∗(Qi)for all (i,Q)∈S×Q,and Q∗ is a weak equilibrium.\Gamma^{Q^*}(Q^*_i)\ge\Gamma^{Q^*}(Q_i)\quad\text{for all }(i,Q)\in S\times\mathcal Q,\qquad\text{and } Q^* \text{ is a weak equilibrium.}ΓQ∗(Qi∗​)≥ΓQ∗(Qi​)for all (i,Q)∈S×Q,and Q∗ is a weak equilibrium.

Milestones

  1. (5.12): an equilibrium unu^nun satisfies κn(0,i,uin)+Hn(un)⋅uin≥κn(0,i,ui)+Hn(un)⋅ui\kappa^n(0,i,u^n_i)+H^n(u^n)\cdot u^n_i\ge\kappa^n(0,i,u_i)+H^n(u^n)\cdot u_iκn(0,i,uin​)+Hn(un)⋅uin​≥κn(0,i,ui​)+Hn(un)⋅ui​ for all u∈Anu\in\mathcal A^nu∈An.
  2. Generator form (proof of Theorem 5.2): dividing by δn\delta_nδn​, the same inequality reads f(0,i,Qin)+H⋅Qin≥f(0,i,Qiu,n)+H⋅Qiu,nf(0,i,Q^n_i)+H\cdot Q^n_i\ge f(0,i,Q^{u,n}_i)+H\cdot Q^{u,n}_if(0,i,Qin​)+H⋅Qin​≥f(0,i,Qiu,n​)+H⋅Qiu,n​.
  3. Continuity of Q↦F(Q)Q\mapsto F(Q)Q↦F(Q) on Q\mathcal QQ (proof of Theorem 3.3).
  4. Lemma 5.1: Hink(unk)→F(i,Q∗)H^{n_k}_i(u^{n_k})\to F(i,Q^*)Hink​​(unk​)→F(i,Q∗) along the subsequence.

Significance

The theorem justifies the weak-equilibrium notion from discrete time: whatever a time-inconsistent agent does on a fine time grid, the continuous-time limit satisfies the first-order condition (3.10). Combined with the paper's discrete existence result (Theorem 5.1) and compactness of Q\mathcal QQ, it yields weak equilibria by approximation (Remark 5.3). It also locates the gap between the two notions: the limit is weak but in general not strong.

The result is proved in the paper; nothing about it is open. No part of it has a machine-checked proof. A formal development adds a checked Euler-scheme comparison between (I+δQ)k(I+\delta Q)^k(I+δQ)k and ekδQe^{k\delta Q}ekδQ for generator matrices, a dominated passage to the limit for Riemann sums of the payoff, and continuity of the payoff in the generator, each reusable for other finite-state continuous-time control problems.

Difficulty

The obvious argument is to divide the discrete one-step inequality (5.12) by δn\delta_nδn​ and let n→∞n\to\inftyn→∞. Its algebraic part is immediate. The difficulty is the convergence of the discrete continuation values Hn(un)H^n(u^n)Hn(un), which are infinite sums along a discrete chain with transition matrix unu^nun, to the continuous payoff F(Q∗)F(Q^*)F(Q∗), an integral along the continuous chain with generator Q∗Q^*Q∗. Three errors must be controlled at once: the discrete chain against the chain sampled from etQne^{tQ^n}etQn (the difference of kkk-th powers grows with k≈M/δnk\approx M/\delta_nk≈M/δn​), the Riemann sum against the integral (needs uniform continuity of fff jointly in ttt and qqq), and F(Qn)F(Q^n)F(Qn) against F(Q∗)F(Q^*)F(Q∗). The tail of the infinite sums is not controlled by (2.3) alone: integrability of a majorant does not make its samples on a grid summable, which is why (5.8) is assumed. A second subtlety concerns the last step: the paper passes from (3.10) to the weak-equilibrium property through Theorem 3.1, which assumes a condition (3.3) that Theorem 5.2 does not. The conclusion is still true without (3.3), but no numbered statement of the paper provides that step.

Formalization scope

  • States are Fin N; rows are Fin N → ℝ with the sup norm, generators and transition matrices are Matrix (Fin N) (Fin N) ℝ with the product topology.
  • Expectations are written through marginals: F(i,Q)=∫0∞∑j(etQ)ijf(t,j,Qj) dtF(i,Q)=\int_0^\infty\sum_j(e^{tQ})_{ij}f(t,j,Q_j)\,dtF(i,Q)=∫0∞​∑j​(etQ)ij​f(t,j,Qj​)dt (Bochner integral) and Hin(u)=∑k∑j(uk)ijκn(k+1,j,uj)H^n_i(u)=\sum_k\sum_j(u^k)_{ij}\kappa^n(k+1,j,u_j)Hin​(u)=∑k​∑j​(uk)ij​κn(k+1,j,uj​) (tsum). No Markov process is constructed.
  • The weak-equilibrium liminf is encoded as "for every η>0\eta>0η>0, eventually as ε↓0\varepsilon\downarrow0ε↓0 the quotient is ≥−η\ge-\eta≥−η".
  • "f(⋅,i,⋅)f(\cdot,i,\cdot)f(⋅,i,⋅) is continuous" is joint continuity on [0,∞)×Di[0,\infty)\times D_i[0,∞)×Di​; (5.8) is required for q∈Diq\in D_iq∈Di​; δn↓0\delta_n\downarrow0δn​↓0 is positivity, monotonicity and convergence to 000; the subsequence is a strictly increasing φ:N→N\varphi:\mathbb N\to\mathbb Nφ:N→N.
  • Not assumed: integrability of FFF, summability of HnH^nHn or (5.2) for κn\kappa^nκn (they follow from (2.3) and (5.8)), compactness of DiD_iDi​, and condition (3.3).
  • The equilibria unu^nun, the generators Qn=(un−I)/δnQ^n=(u^n-I)/\delta_nQn=(un−I)/δn​ and the subsequence are part of the goal's data. A version that only assumes "Qn→Q∗Q^n\to Q^*Qn→Q∗" for arbitrary Qn∈QQ^n\in\mathcal QQn∈Q would be false and is ruled out.
  • The continuity milestone is proved in the paper for compact DiD_iDi​; it is stated here without compactness, since only convergent sequences in Q\mathcal QQ matter.
  • Needed infrastructure: bounds on ∥(I+δQ)k−ekδQ∥\|(I+\delta Q)^k-e^{k\delta Q}\|∥(I+δQ)k−ekδQ∥ for generator matrices, stochasticity of etQe^{tQ}etQ for generator QQQ, dominated convergence for the payoff integral and for the discrete series. Contributions of these general lemmas are welcome.

Selected references

  • Y.-J. Huang and Z. Zhou, Strong and Weak Equilibria for Time-Inconsistent Stochastic Control in Continuous Time, Math. Oper. Res. 46(2), 2021; read as arXiv:1809.09243v3. https://arxiv.org/abs/1809.09243 , https://doi.org/10.1287/moor.2020.1066
  • I. Ekeland and A. Lazrak, Being serious about non-commitment: subgame perfect equilibrium in continuous time, tech. rep., University of British Columbia, 2006. http://arxiv.org/abs/math/0604264
  • S. N. Ethier and T. G. Kurtz, Markov Processes: Characterization and Convergence, Wiley, 1986. https://doi.org/10.1002/9780470316658
8 thms1 active userReviewed
ProbabilityStatisticsStochastic Systems·Captain: mikedeng1

Can Local Particle Filters Beat the Curse of Dimensionality?: The Block Particle Filter Has a Local Error Bound Uniform in Time and in the Model DimensionResearch Paper

Motivation

Nonlinear filtering asks for the conditional law of an unobserved Markov process given noisy partial observations of it. It is the computational core of data assimilation in weather forecasting, oceanography and geophysics, of target tracking, and of inference in large networked systems. The standard numerical method, the particle filter, replaces the conditional law by a cloud of NNN weighted samples. Its error is well understood in fixed dimension, but in practice it degrades exponentially in the dimension of the state: the number of particles needed for a fixed accuracy grows exponentially with the number of variables. This phenomenon, the curse of dimensionality of particle filters, has been documented empirically and analysed by Bickel, Li and Bengtsson and by Snyder et al. (Bickel–Li–Bengtsson 2008; Snyder et al. 2008), and it is a main obstacle to their use in high-dimensional data assimilation.

Rebeschini and van Handel (arXiv:1301.6585v2, Ann. Appl. Probab. 25(5), 2015) gave the first rigorous proof that a local variant of the particle filter, the block particle filter, has an approximation error that does not depend on the model dimension, for models with local interactions above a mixing threshold. The result is the standard theoretical reference for localization in sequential Monte Carlo.

Setting

Let G=(V,E)G=(V,E)G=(V,E) be a finite graph with graph distance ddd, and fix a neighbourhood size r≥1r\ge1r≥1. The rrr-neighbourhood of a vertex is N(v)={v′:d(v,v′)≤r}N(v)=\{v':d(v,v')\le r\}N(v)={v′:d(v,v′)≤r}, and the rrr-inner boundary of J⊆VJ\subseteq VJ⊆V is ∂J={v∈J:N(v)⊈J}\partial J=\{v\in J:N(v)\not\subseteq J\}∂J={v∈J:N(v)⊆J}. Each vertex carries a Polish state space Xv\mathbb X^vXv with a reference measure ψv\psi^vψv, and an observation space Yv\mathbb Y^vYv; write X=∏vXv\mathbb X=\prod_v\mathbb X^vX=∏v​Xv and ψ=⨂vψv\psi=\bigotimes_v\psi^vψ=⨂v​ψv.

The hidden process (Xn)n≥0(X_n)_{n\ge0}(Xn​)n≥0​ is a Markov chain on X\mathbb XX with transition density p(x,z)=∏vpv(x,zv)p(x,z)=\prod_{v}p^v(x,z^v)p(x,z)=∏v​pv(x,zv) with respect to ψ\psiψ, where each pv(x,zv)p^v(x,z^v)pv(x,zv) depends on xxx only through xN(v)x^{N(v)}xN(v) (locality). The observation YnY_nYn​ has density g(x,y)=∏vgv(xv,yv)g(x,y)=\prod_v g^v(x^v,y^v)g(x,y)=∏v​gv(xv,yv) given Xn=xX_n=xXn​=x. For a fixed observation sequence, the filter is the recursion

π0=δx,πn=CnPπn−1,\pi_0=\delta_x,\qquad \pi_n=\mathsf C_n\mathsf P\pi_{n-1},π0​=δx​,πn​=Cn​Pπn−1​,

where P\mathsf PP applies the transition kernel and Cn\mathsf C_nCn​ reweights by g(⋅,Yn)g(\cdot,Y_n)g(⋅,Yn​) and renormalizes (the Bayes formula).

A partition K\mathcal KK of VVV into blocks defines the blocking operator Bρ=⨂K∈KBKρ\mathsf B\rho=\bigotimes_{K\in\mathcal K}\mathsf B^K\rhoBρ=⨂K∈K​BKρ, which replaces a measure by the product of its block marginals. The block particle filter with NNN particles is

π^n=CnBSNPπ^n−1,\hat\pi_n=\mathsf C_n\mathsf B\mathsf S^N\mathsf P\hat\pi_{n-1},π^n​=Cn​BSNPπ^n−1​,

where SN\mathsf S^NSN replaces a measure by the empirical measure of NNN i.i.d. samples from it. The local quantities are ∣K∣∞=max⁡Kcard⁡K|\mathcal K|_\infty=\max_K\operatorname{card}K∣K∣∞​=maxK​cardK, Δ=max⁡vcard⁡N(v)\Delta=\max_v\operatorname{card}N(v)Δ=maxv​cardN(v), and ΔK=max⁡Kcard⁡{K′:d(K,K′)≤r}\Delta_{\mathcal K}=\max_K\operatorname{card}\{K':d(K,K')\le r\}ΔK​=maxK​card{K′:d(K,K′)≤r}. The error is measured locally:

∣∣∣ρ−ρ′∣∣∣J=sup⁡f∈XJ, ∣f∣≤1E[∣ρ(f)−ρ′(f)∣2]1/2,|||\rho-\rho'|||_J=\sup_{f\in\mathbb X^J,\,|f|\le1}\mathbf E\big[|\rho(f)-\rho'(f)|^2\big]^{1/2},∣∣∣ρ−ρ′∣∣∣J​=f∈XJ,∣f∣≤1sup​E[∣ρ(f)−ρ′(f)∣2]1/2,

where XJ\mathbb X^JXJ is the class of measurable functions depending only on the coordinates in JJJ.

Formalization targets

Goal: Theorem 2.1

There is 0<ε0<10<\varepsilon_0<10<ε0​<1 depending only on (Δ,ΔK)(\Delta,\Delta_{\mathcal K})(Δ,ΔK​) such that, if ε0<ε<1\varepsilon_0<\varepsilon<1ε0​<ε<1, 0<κ<10<\kappa<10<κ<1, ε≤pv≤ε−1\varepsilon\le p^v\le\varepsilon^{-1}ε≤pv≤ε−1 and κ≤gv≤κ−1\kappa\le g^v\le\kappa^{-1}κ≤gv≤κ−1, then for all n≥0n\ge0n≥0, x∈Xx\in\mathbb Xx∈X, K∈KK\in\mathcal KK∈K, J⊆KJ\subseteq KJ⊆K,

∣∣∣πnx−π^nx∣∣∣J≤αcard⁡J[e−β1d(J,∂K)+eβ2∣K∣∞N],|||\pi^x_n-\hat\pi^x_n|||_J\le\alpha\operatorname{card}J\Big[e^{-\beta_1d(J,\partial K)}+\frac{e^{\beta_2|\mathcal K|_\infty}}{\sqrt N}\Big],∣∣∣πnx​−π^nx​∣∣∣J​≤αcardJ[e−β1​d(J,∂K)+N​eβ2​∣K∣∞​​],

with 0<α,β1,β2<∞0<\alpha,\beta_1,\beta_2<\infty0<α,β1​,β2​<∞ depending only on (ε,κ,r,Δ,ΔK)(\varepsilon,\kappa,r,\Delta,\Delta_{\mathcal K})(ε,κ,r,Δ,ΔK​). The goal states the constants existentially, as the paper does, so that it survives any improvement of them.

Milestones

The proof splits the error through the block filter π~n=CnBPπ~n−1\tilde\pi_n=\mathsf C_n\mathsf B\mathsf P\tilde\pi_{n-1}π~n​=Cn​BPπ~n−1​:

  1. elementary lemmas: Lemma 4.1 (minorization), Lemma 4.2 (reweighting), Lemma 4.3 (weighted Neumann series);
  2. Theorem 3.1, the Dobrushin comparison theorem;
  3. Theorem 4.14, the bias term ∥πnx−π~nx∥J≤8e−β1−e−β(1−ε2Δ)card⁡J e−βd(J,∂K)\|\pi^x_n-\tilde\pi^x_n\|_J\le\frac{8e^{-\beta}}{1-e^{-\beta}}(1-\varepsilon^{2\Delta})\operatorname{card}J\,e^{-\beta d(J,\partial K)}∥πnx​−π~nx​∥J​≤1−e−β8e−β​(1−ε2Δ)cardJe−βd(J,∂K);
  4. Proposition 4.17, stability of the block filter from product initial laws;
  5. Lemma 4.19, Lemma 4.20 (products of empirical measures) and Proposition 4.22, the one-step sampling errors;
  6. Theorem 4.23, the variance term ∣∣∣π~nx−π^nx∣∣∣J≤card⁡J 64ΔKeβ1−e−βε−4∣K∣∞κ−4∣K∣∞ΔKN|||\tilde\pi^x_n-\hat\pi^x_n|||_J\le\operatorname{card}J\,\frac{64\Delta_{\mathcal K}e^\beta}{1-e^{-\beta}}\frac{\varepsilon^{-4|\mathcal K|_\infty}\kappa^{-4|\mathcal K|_\infty\Delta_{\mathcal K}}}{\sqrt N}∣∣∣π~nx​−π^nx​∣∣∣J​≤cardJ1−e−β64ΔK​eβ​N​ε−4∣K∣∞​κ−4∣K∣∞​ΔK​​.

Theorems 4.14 and 4.23 together give Theorem 2.1 by the triangle inequality.

Significance

The theorem shows that the error of a local particle filter can be controlled by quantities that are local in the graph: neither the assumptions nor the bound depend on the time horizon or on card⁡V\operatorname{card}VcardV. On a square lattice, choosing blocks of side length of order log⁡N\log NlogN balances the two terms and gives a local error, at most locations, that tends to zero as N→∞N\to\inftyN→∞ uniformly in time and in the lattice size (the paper's Corollary 2.5, which is not part of this mission): the number of particles needed for a given local accuracy does not grow with the dimension. The result identifies decay of correlations of the filter, established through the Dobrushin comparison theorem, as the mechanism that makes localization work, and it underlies later work on local particle filters and their spatial bias.

The result is proved on paper. No part of it is formalized, and the formal library has no nonlinear filter, no particle filter and no Dobrushin comparison theorem. The mission produces machine-checked statements of the main theorem and of the intermediate results of its proof, with explicit constants for the two halves; proofs of any milestone, or of the goal from Theorems 4.14 and 4.23, are the remaining work.

Difficulty

The obvious argument fails in two places. First, the standard particle filter error bound is uniform in time only through filter stability, and its constants are exponential in the dimension, because the reweighting step compares likelihoods over all of VVV at once. Localizing the sampling (blocking) removes this, but introduces a bias at block boundaries that must not accumulate over time. Controlling that bias requires showing that the filter has decay of correlations uniformly in time, which is a statement about conditional distributions of a high-dimensional measure, not about its marginals. Second, the sampling error must be propagated through the block filter dynamics in a norm strong enough to compose: the weak ∣∣∣⋅∣∣∣|||\cdot|||∣∣∣⋅∣∣∣ norm does not contract under the block filter, which is why the paper measures one-step errors in total variation after one further step of dynamics.

Formalization scope

The Lean development is in the namespace LocalPF.Block. Committed conventions:

  • the graph distance takes values in N∪{∞}\mathbb N\cup\{\infty\}N∪{∞}, and e−βd=0e^{-\beta d}=0e−βd=0 for d=∞d=\inftyd=∞; the partition is a surjective block label map;
  • the filter, block filter and block particle filter are defined by their recursions for a fixed observation sequence, and every statement holds for every observation sequence (Remark 2.4); the reference measure of the observations is omitted;
  • the block particle filter is represented by the law of its particle array at each step, built with the Giry monad bind, and the step-nnn estimate as a function of that array;
  • total variation norms ∥⋅∥\|\cdot\|∥⋅∥, ∥⋅∥J\|\cdot\|_J∥⋅∥J​ are suprema in [0,∞][0,\infty][0,∞]; ∣∣∣⋅∣∣∣J≤B|||\cdot|||_J\le B∣∣∣⋅∣∣∣J​≤B is stated as E[∣ρ(f)−ρ′(f)∣2]≤B2\mathbf E[|\rho(f)-\rho'(f)|^2]\le B^2E[∣ρ(f)−ρ′(f)∣2]≤B2 for every test function, with lower Lebesgue integrals;
  • r≥1r\ge1r≥1 and N≥1N\ge1N≥1; reference measures are σ\sigmaσ-finite.

Added or changed hypotheses, each disclosed in the item: strict positivity and integrability of gvg^vgv in Theorem 4.14 and Proposition 4.17 (the paper assumes no bounds on ggg there; without them the Bayes formula is undefined); explicit bounds 0<a≤Λ≤b0<a\le\Lambda\le b0<a≤Λ≤b in Lemma 4.2, whose ∣∣∣⋅∣∣∣|||\cdot|||∣∣∣⋅∣∣∣ clause is stated for Giry-measurable random probability measures; ε<1\varepsilon<1ε<1 in Theorem 4.23, excluding only the case where the paper's bound is +∞+\infty+∞; Lemma 4.19 and Proposition 4.22 are stated for every deterministic probability measure in place of the random previous estimate, which is stronger.

The statement admits no trivializing reading: the constants of Theorem 2.1 are quantified before the graph and the model, so they cannot depend on card⁡V\operatorname{card}VcardV; the goal and Theorem 4.23 assert that the particle-array law is a probability measure, so a zero law cannot make the expectation vanish; test functions are measurable and bounded, so the integrals are not junk values; N≥1N\ge1N≥1 excludes division by 0\sqrt00​; and the correction step's normalizer is positive and finite under the stated hypotheses.

Needed infrastructure: product measures over finite index sets and their marginals, the Giry monad and measurability of the maps it composes, conditional distributions on products of Polish spaces, and matrix Neumann series. The Dobrushin comparison theorem and the V-statistic bound (Lemma 4.20) are reusable beyond filtering. Proofs of any milestone are welcome, and so are formal statements of the remaining results of §4 (Propositions 4.4, 4.8, 4.9, Corollary 4.13).

Selected references

  • P. Rebeschini and R. van Handel, Can local particle filters beat the curse of dimensionality?, Ann. Appl. Probab. 25(5), 2809–2866, 2015. arXiv:1301.6585, doi:10.1214/14-AAP1061
  • P. Bickel, B. Li and T. Bengtsson, Sharp failure rates for the bootstrap particle filter in high dimensions, IMS Collections 3, 318–329, 2008. doi:10.1214/074921708000000228
  • C. Snyder, T. Bengtsson, P. Bickel and J. Anderson, Obstacles to high-dimensional particle filtering, Monthly Weather Review 136(12), 4629–4640, 2008. doi:10.1175/2008MWR2529.1
  • H.-O. Georgii, Gibbs Measures and Phase Transitions, de Gruyter, 2nd ed., 2011 (Theorem 8.20, the Dobrushin comparison theorem). doi:10.1515/9783110250329
13 thms1 active userReviewed
CombinatoricsLinear algebraTheoretical Computer Science·Captain: mikedeng1

High Order Random Walks: Beyond Spectral Gap 2: On a Two-Sided λ-Local Spectral Expander, Spec(M⁺_k) ⊆ {1} ∪ ⋃_j [(k+1−j)/(k+2) ± √(k+1)·ε_k/(k+2)] and Eigenvectors Lie Near U^j_kResearch Paper

Motivation

High-dimensional expanders are simplicial complexes whose local structure forces global mixing. Random walks on their kkk-dimensional faces, the high order random walks, are used to analyse sampling algorithms: the bases-exchange walk on the bases of a matroid is such a walk, and its spectral analysis by Anari, Liu, Oveis Gharan and Vinzant (arXiv:1811.01816) settled the Mihail–Vazirani conjecture on counting matroid bases. Agreement tests and PCP constructions use the same walks (Dinur–Kaufman, FOCS 2017).

Most of these applications use only the second eigenvalue of the walk. Kaufman and Oppenheim (arXiv:1707.02799v3) observe that this eigenvalue cannot be very small, because of natural obstructions such as cochains pulled up from lower dimensions, and ask instead for the whole spectrum and the structure of the eigenvectors. For complexes whose links have both small second eigenvalue and bounded-below smallest eigenvalue (two-sided local spectral expanders), they show that the spectrum of the walk on kkk-faces clusters around k+1k+1k+1 explicit values, each cluster corresponding to cochains built from a lower dimension.

Timeline. Garland (1973, doi:10.2307/1970833) introduced the localization method relating global cohomology to spectra of links. Kaufman and Mass (ITCS 2017, arXiv:1604.02947) defined high order random walks and bounded their second eigenvalue. Oppenheim (Discrete Comput. Geom. 2018, arXiv:1709.04431) developed the local spectral expansion framework and its trickle-down theorem. Kaufman and Oppenheim gave the decomposition theorems this mission formalizes; arXiv v3 (2021) corrects an error of the published version. Alev and Lau (arXiv:2001.02827) later improved the one-sided bounds.

Setting

Let XXX be a pure nnn-dimensional finite simplicial complex on a vertex set VVV: a finite family of finite subsets of VVV, closed under subsets, in which every face lies in a face with n+1n+1n+1 vertices. X(k)X(k)X(k) is the set of faces with k+1k+1k+1 vertices, and X(−1)={∅}X(-1)=\{\emptyset\}X(−1)={∅}. A weight is a function m:X→(0,∞)m:X\to(0,\infty)m:X→(0,∞) with m(τ)=∑σ∈X(k+1), τ⊆σm(σ)m(\tau)=\sum_{\sigma\in X(k+1),\,\tau\subseteq\sigma}m(\sigma)m(τ)=∑σ∈X(k+1),τ⊆σ​m(σ) for τ∈X(k)\tau\in X(k)τ∈X(k), k≤n−1k\le n-1k≤n−1.

A kkk-cochain is a function ϕ:X(k)→R\phi:X(k)\to\mathbb Rϕ:X(k)→R; the space Ck(X)C^k(X)Ck(X) carries the inner product ⟨ϕ,ψ⟩=∑σ∈X(k)m(σ)ϕ(σ)ψ(σ)\langle\phi,\psi\rangle=\sum_{\sigma\in X(k)}m(\sigma)\phi(\sigma)\psi(\sigma)⟨ϕ,ψ⟩=∑σ∈X(k)​m(σ)ϕ(σ)ψ(σ). C0k(X)C^k_0(X)C0k​(X) is the subspace of cochains orthogonal to the constants.

  • The signless differential dk:Ck→Ck+1d_k:C^k\to C^{k+1}dk​:Ck→Ck+1 is dkϕ(σ)=∑τ⊂σ, τ∈X(k)ϕ(τ)d_k\phi(\sigma)=\sum_{\tau\subset\sigma,\,\tau\in X(k)}\phi(\tau)dk​ϕ(σ)=∑τ⊂σ,τ∈X(k)​ϕ(τ), and dk∗d_k^*dk∗​ is its adjoint.
  • The upper random walk Mk+M^+_kMk+​ on X(k)X(k)X(k) stays at τ\tauτ with probability 1k+2\frac1{k+2}k+21​ and moves to τ′\tau'τ′ with probability m(τ∪τ′)(k+2)m(τ)\frac{m(\tau\cup\tau')}{(k+2)m(\tau)}(k+2)m(τ)m(τ∪τ′)​ when τ∪τ′∈X(k+1)\tau\cup\tau'\in X(k+1)τ∪τ′∈X(k+1); equivalently dk∗dk=(k+2)Mk+d_k^*d_k=(k+2)M^+_kdk∗​dk​=(k+2)Mk+​.
  • The link of τ\tauτ is Xτ={η: η∩τ=∅, τ∪η∈X}X_\tau=\{\eta:\ \eta\cap\tau=\emptyset,\ \tau\cup\eta\in X\}Xτ​={η: η∩τ=∅, τ∪η∈X} with weight mτ(η)=m(τ∪η)m_\tau(\eta)=m(\tau\cup\eta)mτ​(η)=m(τ∪η). On its vertices, (M′)τ,0+(M')^+_{\tau,0}(M′)τ,0+​ is the non-lazy walk; μτ\mu_\tauμτ​ and ντ\nu_\tauντ​ are its second largest and smallest eigenvalues.
  • XXX is a two-sided λ\lambdaλ-local spectral expander if −λ≤ντ-\lambda\le\nu_\tau−λ≤ντ​ and μτ≤λ\mu_\tau\le\lambdaμτ​≤λ for every face τ\tauτ of dimension −1,…,n−2-1,\dots,n-2−1,…,n−2.

For 0≤k≤n−10\le k\le n-10≤k≤n−1 the paper defines subspaces of C0k(X)C^k_0(X)C0k​(X): V00=C00V^0_0=C^0_0V00​=C00​; for k≥1k\ge1k≥1, Vkj=dk−1⋯dj(C0j)V^j_k=d_{k-1}\cdots d_j(C^j_0)Vkj​=dk−1​⋯dj​(C0j​) for j<kj<kj<k and Vkk=ker⁡dk−1∗V^k_k=\ker d^*_{k-1}Vkk​=kerdk−1∗​; and Uk0=Vk0U^0_k=V^0_kUk0​=Vk0​, Ukj=Vkj∩(Vkj−1)⊥U^j_k=V^j_k\cap(V^{j-1}_k)^\perpUkj​=Vkj​∩(Vkj−1​)⊥ for 0<j<k0<j<k0<j<k, Ukk=VkkU^k_k=V^k_kUkk​=Vkk​. With ε0=λ\varepsilon_0=\lambdaε0​=λ and εk=2k(1+2kk)εk−1+(k+1)λ\varepsilon_k=2k(1+2k\sqrt k)\varepsilon_{k-1}+(k+1)\lambdaεk​=2k(1+2kk​)εk−1​+(k+1)λ, PUP_{U}PU​ denotes the weighted orthogonal projection onto UUU.

Formalization targets

Goal: Theorem 5.9 (p. 22)

If εk≤12(1+2(k+1)k+1)\varepsilon_k\le\frac{1}{2(1+2(k+1)\sqrt{k+1})}εk​≤2(1+2(k+1)k+1​)1​ for 0≤k≤n−20\le k\le n-20≤k≤n−2 and εn−1<12n\varepsilon_{n-1}<\frac1{2\sqrt n}εn−1​<2n​1​, then for 0≤k≤n−10\le k\le n-10≤k≤n−1

Spec⁡(Mk+)⊆{1}∪⋃j=0k[k+1−jk+2−k+1k+2εk, k+1−jk+2+k+1k+2εk],\operatorname{Spec}(M^+_k)\subseteq\{1\}\cup\bigcup_{j=0}^{k}\Big[\tfrac{k+1-j}{k+2}-\tfrac{\sqrt{k+1}}{k+2}\varepsilon_k,\ \tfrac{k+1-j}{k+2}+\tfrac{\sqrt{k+1}}{k+2}\varepsilon_k\Big],Spec(Mk+​)⊆{1}∪j=0⋃k​[k+2k+1−j​−k+2k+1​​εk​, k+2k+1−j​+k+2k+1​​εk​],

and an eigenvector ϕ∈C0k\phi\in C^k_0ϕ∈C0k​ whose eigenvalue lies in the jjj-th interval satisfies ∥ϕ−PUkjϕ∥≤k+1εk1−k+1εk∥ϕ∥\|\phi-P_{U^j_k}\phi\|\le\frac{\sqrt{k+1}\varepsilon_k}{1-\sqrt{k+1}\varepsilon_k}\|\phi\|∥ϕ−PUkj​​ϕ∥≤1−k+1​εk​k+1​εk​​∥ϕ∥.

Milestones

  • the localization identities (Lemma 3.6, Corollary 3.7, Propositions 4.1, 4.2) and the two-sided bound of Lemma 4.3;
  • Lemma 5.1 and the orthogonal decomposition C0k(X)=Ukk⊕⋯⊕Uk0C^k_0(X)=U^k_k\oplus\cdots\oplus U^0_kC0k​(X)=Ukk​⊕⋯⊕Uk0​ (§5.3);
  • Theorem 5.6: ∥dk∗dkPUkjϕ−(k+1−j)PUkjϕ∥≤εk∥PUkjϕ∥\|d^*_kd_kP_{U^j_k}\phi-(k+1-j)P_{U^j_k}\phi\|\le\varepsilon_k\|P_{U^j_k}\phi\|∥dk∗​dk​PUkj​​ϕ−(k+1−j)PUkj​​ϕ∥≤εk​∥PUkj​​ϕ∥;
  • Corollary 5.8: the same for Mk+M^+_kMk+​, with k+1−jk+2\frac{k+1-j}{k+2}k+2k+1−j​ and εkk+2\frac{\varepsilon_k}{k+2}k+2εk​​.

Significance

The result. Theorem 5.9 describes the spectrum of Mk+M^+_kMk+​, not only its second eigenvalue. The eigenvalue k+1−jk+2\frac{k+1-j}{k+2}k+2k+1−j​ is the one carried by cochains pulled up from dimension jjj, so it explains which cochains are shrunk slowly by the walk and which quickly. It is the two-sided counterpart of the paper's one-sided mixing bound (Theorem 5.4, the subject of the first mission in this series). It gives approximate eigenspaces explicitly, which a mixing-time analysis or an agreement-test analysis can use in place of a bound on a single eigenvalue.

Formalizing it. No machine-checked development of simplicial-complex random walks, Garland's method, or local spectral expansion is known to exist; Mathlib has simplicial complexes in a geometric sense but no weighted cochain calculus. The mission produces the cochain calculus of §2–§4 together with the spectral statements of §5.3. The statements are those of the arXiv v3 preprint. They are published results but have not been formalized.

Difficulty

The obvious route would show that the spaces UkjU^j_kUkj​ are exact eigenspaces. That holds for the complete complex but not in general: d∗dd^*dd∗d maps UkjU^j_kUkj​ only approximately into itself, and the error has to be controlled through every lower dimension at once. Theorem 5.6 is an induction on kkk in which the error εk−1\varepsilon_{k-1}εk−1​ of dimension k−1k-1k−1 enters dimension kkk multiplied by norms of dk−1∗dk−1d^*_{k-1}d_{k-1}dk−1∗​dk−1​ and its square root. Controlling these factors on the correct subspaces, rather than on all of Ck−1C^{k-1}Ck−1, is the delicate step: dk−1∗dk−1=(k+1)Mk−1+d^*_{k-1}d_{k-1}=(k+1)M^+_{k-1}dk−1∗​dk−1​=(k+1)Mk−1+​ has norm k+1k+1k+1 on Ck−1(X)C^{k-1}(X)Ck−1(X), attained on the constants, and the bound needed is on the complement of the constants. Passing from Corollary 5.8 to the spectral statement of Theorem 5.9 needs the intervals to be disjoint and the eigenvector to be split along the orthogonal decomposition.

Formalization scope

  • Encoding. Faces are Finset V and XXX is a Finset (Finset V); faces are indexed by their number of vertices (cells X (k + 1) is X(k)X(k)X(k)). Cochains are functions Finset V → ℝ, and every operator vanishes off XXX.
  • Supports. Subspaces, eigenvectors and orthogonal complements consist of functions supported on X(k)X(k)X(k) (Supp X (k + 1)). Without this, a function living off the faces would have norm 000, would be an eigenvector with eigenvalue 000, and would make the spectral statement false; a formalization without supports is ruled out for this reason.
  • Weights and adjoints. The weight is strictly positive on faces. d∗d^*d∗ is defined by its explicit formula, and Lemma 3.6 states that this formula is the adjoint.
  • Expansion. The eigenvalue conditions on links are stated in Rayleigh form on the weighted complement of the constants, ∣⟨(M′)τ,0+g,g⟩∣≤λ∥g∥2|\langle(M')^+_{\tau,0}g,g\rangle|\le\lambda\|g\|^2∣⟨(M′)τ,0+​g,g⟩∣≤λ∥g∥2; this encodes −λ≤ντ-\lambda\le\nu_\tau−λ≤ντ​ and μτ≤λ\mu_\tau\le\lambdaμτ​≤λ and needs no connectivity. Definition 1.2 is followed where Definition 4.4 misprints the two-sided clause.
  • Projections. PUϕP_U\phiPU​ϕ is characterised relationally (IsWProj: u∈Uu\in Uu∈U and ϕ−u⊥U\phi-u\perp Uϕ−u⊥U) instead of by an operator. On these finite-dimensional subspaces it exists and is unique.
  • Indices and constants. εk\varepsilon_kεk​ is eps lam k, with the recursion shifted by one index. kkk is quantified inside each conclusion, and εn−1\varepsilon_{n-1}εn−1​ is written without natural subtraction. The statements follow the printed constants of arXiv v3.

A complete development needs finite-dimensional weighted inner-product spaces of cochains, Garland localization, Courant–Fischer for self-adjoint operators on these spaces, and orthogonal projections onto images of iterated differentials. The cochain calculus and the localization identities are reusable for any weighted simplicial complex, and the first mission of this series needs the same objects. Proofs of the localization identities, of the decomposition, and of any milestone are welcome, as are proofs of the results with weaker hypotheses.

Selected references

  • T. Kaufman, I. Oppenheim, High Order Random Walks: Beyond Spectral Gap, arXiv:1707.02799v3, 2021. https://arxiv.org/abs/1707.02799
  • H. Garland, p-adic curvature and the cohomology of discrete subgroups of p-adic groups, Annals of Mathematics 97, 1973. https://doi.org/10.2307/1970833
  • T. Kaufman, D. Mass, High dimensional random walks and colorful expansion, ITCS 2017. https://arxiv.org/abs/1604.02947
  • I. Oppenheim, Local spectral expansion approach to high dimensional expanders Part I: Descent of spectral gaps, Discrete & Computational Geometry 59, 2018. https://arxiv.org/abs/1709.04431
  • N. Anari, K. Liu, S. Oveis Gharan, C. Vinzant, Log-concave polynomials II: high-dimensional walks and an FPRAS for counting bases of a matroid, STOC 2019. https://arxiv.org/abs/1811.01816
  • V. L. Alev, L. C. Lau, Improved analysis of higher order random walks and applications, STOC 2020. https://arxiv.org/abs/2001.02827
15 thms1 active userReviewed
CombinatoricsLinear algebraTheoretical Computer Science·Captain: mikedeng1

High Order Random Walks: Beyond Spectral Gap 1: On a One-Sided λ-Local Spectral Expander, ‖M⁺_k φ‖ ≤ ((k+1)/(k+2) + (k+1)λ/2)‖φ‖ for Every k-Cochain φ Orthogonal to the ConstantsResearch Paper

Motivation

A high dimensional expander is a simplicial complex whose local structure forces global expansion. The high order random walks on such a complex move from a kkk-face to a kkk-face through a common (k+1)(k+1)(k+1)-face. They were studied for their connection to agreement tests in probabilistically checkable proofs (Dinur–Kaufman 2017; Kaufman–Mass 2017), and they later became a tool for sampling. Anari, Liu, Oveis Gharan and Vinzant (arXiv:1811.01816) used the mixing bound of this mission to prove the Mihail–Vazirani conjecture: the bases-exchange walk of every matroid mixes rapidly.

Timeline:

  • 1973. Garland (Ann. of Math. 97) shows that spectral gaps in the links of a complex control its cohomology. Inner products are localized to links; this is now called Garland's method.
  • 2017. Kaufman and Mass (arXiv:1604.02947) bound the second eigenvalue of the high order walks on complexes with well-expanding links. Dinur and Kaufman obtain the optimal bound under a two-sided spectral gap in the links, as Kaufman–Oppenheim explain.
  • 2018. Oppenheim (Discrete Comput. Geom. 59) shows how spectral gaps descend from top-dimensional links to lower ones ("trickling down").
  • 2017–2021. Kaufman and Oppenheim (arXiv:1707.02799v3) obtain the optimal mixing bound under a one-sided gap only: negative eigenvalues of the links do not matter. This is the target of this mission.

Setting

Let VVV be a type of vertices. A pure nnn-dimensional finite simplicial complex XXX is a finite family of finite subsets of VVV that is closed under subsets, has faces of at most n+1n+1n+1 elements, and has every face contained in a face of exactly n+1n+1n+1 elements. A face with k+1k+1k+1 elements is a kkk-face; X(k)X(k)X(k) is the set of kkk-faces, and X(−1)={∅}X(-1)=\{\emptyset\}X(−1)={∅}.

A weight function is a map mmm from the faces of XXX to (0,∞)(0,\infty)(0,∞) that is balanced: for −1≤k≤n−1-1\le k\le n-1−1≤k≤n−1 and τ∈X(k)\tau\in X(k)τ∈X(k),

m(τ)=∑σ∈X(k+1), τ⊆σm(σ).m(\tau)=\sum_{\sigma\in X(k+1),\ \tau\subseteq\sigma}m(\sigma).m(τ)=σ∈X(k+1), τ⊆σ∑​m(σ).

A kkk-cochain is a function ϕ:X(k)→R\phi:X(k)\to\mathbb Rϕ:X(k)→R. The cochains carry the inner product ⟨ϕ,ψ⟩=∑σ∈X(k)m(σ)ϕ(σ)ψ(σ)\langle\phi,\psi\rangle=\sum_{\sigma\in X(k)}m(\sigma)\phi(\sigma)\psi(\sigma)⟨ϕ,ψ⟩=∑σ∈X(k)​m(σ)ϕ(σ)ψ(σ) and the norm ∥⋅∥\|\cdot\|∥⋅∥. The cochains orthogonal to the constants form C0k(X,R)={ϕ:∑σ∈X(k)m(σ)ϕ(σ)=0}C^k_0(X,\mathbb R)=\{\phi:\sum_{\sigma\in X(k)}m(\sigma)\phi(\sigma)=0\}C0k​(X,R)={ϕ:∑σ∈X(k)​m(σ)ϕ(σ)=0}.

The upper random walk Mk+M^+_kMk+​ acts on kkk-cochains by

Mk+ϕ(τ)=1k+2 ϕ(τ)+∑τ′∈X(k), τ∪τ′∈X(k+1)m(τ∪τ′)(k+2) m(τ) ϕ(τ′).M^+_k\phi(\tau)=\frac{1}{k+2}\,\phi(\tau)+\sum_{\tau'\in X(k),\ \tau\cup\tau'\in X(k+1)}\frac{m(\tau\cup\tau')}{(k+2)\,m(\tau)}\,\phi(\tau').Mk+​ϕ(τ)=k+21​ϕ(τ)+τ′∈X(k), τ∪τ′∈X(k+1)∑​(k+2)m(τ)m(τ∪τ′)​ϕ(τ′).

It stays put with probability 1/(k+2)1/(k+2)1/(k+2), and otherwise moves to a neighbouring kkk-face with probability proportional to the weight of their union.

The link of a face τ\tauτ is Xτ={η∈X:η∩τ=∅, τ∪η∈X}X_\tau=\{\eta\in X:\eta\cap\tau=\emptyset,\ \tau\cup\eta\in X\}Xτ​={η∈X:η∩τ=∅, τ∪η∈X}, weighted by mτ(η)=m(τ∪η)m_\tau(\eta)=m(\tau\cup\eta)mτ​(η)=m(τ∪η). On the vertices of XτX_\tauXτ​, the non-lazy upper walk (M′)τ,0+=2Mτ,0+−I(M')^+_{\tau,0}=2M^+_{\tau,0}-I(M′)τ,0+​=2Mτ,0+​−I is the weighted random walk on the 1-skeleton of the link. Let μτ\mu_\tauμτ​ be its second largest eigenvalue. XXX is a one-sided λ\lambdaλ-local spectral expander if μτ≤λ\mu_\tau\le\lambdaμτ​≤λ for every face τ\tauτ of dimension −1,…,n−2-1,\dots,n-2−1,…,n−2. The empty face is included, and its link is XXX itself.

Formalization targets

Goal: Theorem 5.4 (p. 17)

Let 0≤λ≤10\le\lambda\le10≤λ≤1 and let XXX be a one-sided λ\lambdaλ-local spectral expander. Then for every 0≤k≤n−10\le k\le n-10≤k≤n−1 and every ϕ∈C0k(X,R)\phi\in C^k_0(X,\mathbb R)ϕ∈C0k​(X,R),

∥Mk+ϕ∥≤(k+1k+2+k+12 λ)∥ϕ∥.\|M^+_k\phi\|\le\Big(\frac{k+1}{k+2}+\frac{k+1}{2}\,\lambda\Big)\|\phi\|.∥Mk+​ϕ∥≤(k+2k+1​+2k+1​λ)∥ϕ∥.

Milestones

They follow the paper's proof order:

  • Lemma 3.6 and Corollary 3.7: the adjoint d∗d^*d∗ of the signless differential ddd, and the factorizations d∗d=(k+2)Mk+d^*d=(k+2)M^+_kd∗d=(k+2)Mk+​, dd∗=(k+1)Mk−dd^*=(k+1)M^-_kdd∗=(k+1)Mk−​.
  • Proposition 4.1 (1)–(3), Garland's localization identities, and Proposition 4.2:
∥dϕ∥2=∥d∗ϕ∥2+∥ϕ∥2+∑τ∈X(k−1)⟨(M′)τ,0+(I−Mτ,0−)ϕτ,ϕτ⟩.\|d\phi\|^2=\|d^*\phi\|^2+\|\phi\|^2+\sum_{\tau\in X(k-1)}\langle (M')^+_{\tau,0}(I-M^-_{\tau,0})\phi_\tau,\phi_\tau\rangle .∥dϕ∥2=∥d∗ϕ∥2+∥ϕ∥2+τ∈X(k−1)∑​⟨(M′)τ,0+​(I−Mτ,0−​)ϕτ​,ϕτ​⟩.
  • Lemma 4.3, which bounds the link terms by (k+1)μ∥ϕ∥2(k+1)\mu\|\phi\|^2(k+1)μ∥ϕ∥2.
  • Lemma 5.1.
  • Theorem 5.2 (the Decomposition Theorem).
  • Corollary 5.3, in its general and its one-sided form:
∥dϕ∥2≤∑j=0k(k+1−j+(k+j+2)(k+1−j)2λ)∥ϕj∥2,∥ϕ∥2=∑j=0k∥ϕj∥2.\|d\phi\|^2\le\sum_{j=0}^{k}\Big(k+1-j+\frac{(k+j+2)(k+1-j)}{2}\lambda\Big)\|\phi^j\|^2,\qquad \|\phi\|^2=\sum_{j=0}^k\|\phi^j\|^2.∥dϕ∥2≤j=0∑k​(k+1−j+2(k+j+2)(k+1−j)​λ)∥ϕj∥2,∥ϕ∥2=j=0∑k​∥ϕj∥2.

Significance

The result. The coefficient k+1k+2\frac{k+1}{k+2}k+2k+1​ is the baseline in the paper's mixing bound, and the bound is optimal up to the O(λ)O(\lambda)O(λ) term. It needs a gap on one side only: links may have eigenvalues near −1-1−1. This matters in two places. The links of a matroid's independence complex are only one-sided expanders, so the matroid application needs exactly this case. And the links of Ramanujan complexes are one-sided expanders, so this bound gives optimal mixing on them; the two-sided bound of Dinur–Kaufman does not apply to them.

Formalizing it. The theorem is proved on paper; the Lean statements in this mission are proof targets. Completing the mission would give machine-checked proofs of the mixing bound, Garland's localization identities, and the Decomposition Theorem, built on definitions of weighted simplicial complexes, links, and their random walks. Lemma 4.3 and Corollary 5.3 are stated with a correction: as printed they are false when a link eigenvalue bound is negative (see Formalization scope).

Difficulty

The obvious argument fails. The tempting route bounds the second eigenvalue of Mk+M^+_kMk+​ directly from the link spectra, by induction on kkk. But Mk+M^+_kMk+​ has large eigenvalues near k+1−jk+2\frac{k+1-j}{k+2}k+2k+1−j​ for every jjj, coming from cochains of the form dψd\psidψ with ψ\psiψ of lower degree. A single spectral-gap estimate cannot separate these levels from one another. The paper instead splits ϕ\phiϕ into contributions from every degree j≤kj\le kj≤k; this is the Decomposition Theorem, and each piece is controlled separately. Formally, the main costs are:

  • the double-counting behind Proposition 4.1 (Appendix A, pp. 28–29);
  • a square root of a positive self-adjoint operator on a weighted finite-dimensional space (Theorem 5.2);
  • the bookkeeping of degrees and links.

Formalization scope

Faces are Finset V. XXX is a Finset (Finset V), and X(k)X(k)X(k) is cells X (k + 1): faces are indexed by their number of vertices. Cochains are functions Finset V → ℝ that are read only on the relevant faces. Every operator is zero off XXX. The weight is positive on faces; the paper writes R+\mathbb R^+R+, and mmm appears in denominators. Every goal-level statement carries the standing assumption of §2: XXX is pure nnn-dimensional and mmm is a weight. Orthogonality to the constants is always weighted by mmm (or by mτm_\taumτ​ in a link).

The adjoint d∗d^*d∗ is defined by its explicit formula; that it is the adjoint is Lemma 3.6. The eigenvalue condition μτ≤λ\mu_\tau\le\lambdaμτ​≤λ is stated in Rayleigh form: ⟨(M′)τ,0+g,g⟩≤λ∥g∥2\langle (M')^+_{\tau,0}g,g\rangle\le\lambda\|g\|^2⟨(M′)τ,0+​g,g⟩≤λ∥g∥2 for every ggg on the vertices of XτX_\tauXτ​ that is mτm_\taumτ​-orthogonal to the constants. It is equivalent to the eigenvalue form even for disconnected links. A Rayleigh bound over all ggg, constants included, would be unsatisfiable for λ<1\lambda<1λ<1 and would make the goal vacuous; it is ruled out. The predicate quantifies over every face of dimension −1-1−1 to n−2n-2n−2, including ∅\emptyset∅.

Corrections, all disclosed in the items:

  • Lemma 4.3 is stated with a bound μ≥max⁡(μk,0)\mu\ge\max(\mu_k,0)μ≥max(μk​,0). The printed version with μk<0\mu_k<0μk​<0 fails already for the complete graph on three vertices.
  • Corollary 5.3 accordingly takes μi≥0\mu_i\ge0μi​≥0 and λ≥0\lambda\ge0λ≥0. Its two formal statements retain the page's assumption that every link of dimension at least one is connected, including XXX itself.
  • The proof of Theorem 5.4 ends with (k+1)λ(k+1)\lambda(k+1)λ; the statement's k+12λ\frac{k+1}{2}\lambda2k+1​λ is formalized.

Welcome contributions: proofs of any milestone. Reusable lemmas on finite weighted inner products, on double counting over faces of a Finset (Finset V), and on links are useful beyond this mission; the companion mission on two-sided expanders uses the same objects.

Selected references

  • T. Kaufman, I. Oppenheim, High Order Random Walks: Beyond Spectral Gap, arXiv:1707.02799v3, 2021. https://arxiv.org/abs/1707.02799
  • N. Anari, K. Liu, S. Oveis Gharan, C. Vinzant, Log-concave polynomials II: High-dimensional walks and an FPRAS for counting bases of a matroid, STOC 2019. https://arxiv.org/abs/1811.01816
  • H. Garland, p-adic curvature and the cohomology of discrete subgroups of p-adic groups, Ann. of Math. 97:375–423, 1973. https://doi.org/10.2307/1970781
  • T. Kaufman, D. Mass, High dimensional random walks and colorful expansion, ITCS 2017. https://arxiv.org/abs/1604.02947
  • I. Oppenheim, Local spectral expansion approach to high dimensional expanders Part I: Descent of spectral gaps, Discrete Comput. Geom. 59(2):293–330, 2018. https://arxiv.org/abs/1709.04431
13 thms1 active userReviewed
ProbabilityStatistics·Captain: mikedeng1

Spectral Gaps for a Metropolis–Hastings Algorithm in Infinite Dimensions II: The pCN Algorithm Has a Wasserstein Spectral Gap Uniform in the Dimension When the Potential Is Locally LipschitzResearch Paper

Motivation

Bayesian inverse problems for functions — the initial condition of a PDE, a permeability field, a diffusion coefficient — lead to posterior measures μ\muμ on an infinite-dimensional Hilbert space HHH, given by a density exp⁡(−Φ)\exp(-\Phi)exp(−Φ) with respect to a Gaussian prior γ\gammaγ. In practice μ\muμ is sampled by Markov chain Monte Carlo on an mmm-dimensional discretisation μm\mu_mμm​, and the question that matters is whether the sampler's convergence rate survives the limit m→∞m\to\inftym→∞. For the standard random-walk Metropolis algorithm it does not (mission III of this series). The preconditioned Crank–Nicolson (pCN) algorithm, whose proposal preserves γ\gammaγ, was introduced to avoid this (Beskos, Roberts, Stuart and Voss 2008; Cotter, Roberts, Stuart and White 2013).

Hairer, Stuart and Vollmer (2014) prove that pCN has a Wasserstein spectral gap that is uniform in the dimension. Mission I of this series covers their result for globally Lipschitz potentials Φ\PhiΦ (Theorem 2.12). This mission covers Theorem 2.14, which needs only that Φ\PhiΦ be locally Lipschitz, with a local Lipschitz constant growing slower than any exponential. That class contains potentials of polynomial growth, which arise for nonlinear forward maps.

Setting

HHH is a separable real Hilbert space with a centred Gaussian measure γ\gammaγ, and (ei)i≥0(e_i)_{i\ge0}(ei​)i≥0​ is an orthonormal basis of eigenvectors of the covariance of γ\gammaγ. PmHP_mHPm​H is the span of e0,…,em−1e_0,\dots,e_{m-1}e0​,…,em−1​, and γm\gamma_mγm​ is the image of γ\gammaγ under the orthogonal projection onto PmHP_mHPm​H. For a potential Φ:H→R\Phi:H\to\mathbb RΦ:H→R the targets are

μ(dx)∝e−Φ(x)γ(dx),μm(dx)∝e−Φ(x)γm(dx).\mu(dx)\propto e^{-\Phi(x)}\gamma(dx),\qquad \mu_m(dx)\propto e^{-\Phi(x)}\gamma_m(dx).μ(dx)∝e−Φ(x)γ(dx),μm​(dx)∝e−Φ(x)γm​(dx).

Fix a step size δ∈(0,12]\delta\in(0,\tfrac12]δ∈(0,21​] and put ρ=1−(1−2δ)1/2\rho=1-(1-2\delta)^{1/2}ρ=1−(1−2δ)1/2. One pCN step from xxx draws ξ∼γ\xi\sim\gammaξ∼γ and proposes px=(1−2δ)1/2x+2δ ξp_x=(1-2\delta)^{1/2}x+\sqrt{2\delta}\,\xipx​=(1−2δ)1/2x+2δ​ξ. The proposal is accepted with probability α(x,px)=1∧exp⁡(Φ(x)−Φ(px))\alpha(x,p_x)=1\wedge\exp(\Phi(x)-\Phi(p_x))α(x,px​)=1∧exp(Φ(x)−Φ(px​)); otherwise the chain stays at xxx. Its transition kernel is P\mathcal PP. The kernel Pm\mathcal P_mPm​ is the same algorithm on PmHP_mHPm​H with ξ∼γm\xi\sim\gamma_mξ∼γm​.

Distances between probability measures are Wasserstein lifts of a cost ccc: c(ν1,ν2)=inf⁡π∫c dπc(\nu_1,\nu_2)=\inf_\pi\int c\,d\pic(ν1​,ν2​)=infπ​∫cdπ over couplings π\piπ of ν1,ν2\nu_1,\nu_2ν1​,ν2​. A kernel is ddd-contracting if d(x,y)<1d(x,y)<1d(x,y)<1 implies d(P(x,⋅),P(y,⋅))≤c d(x,y)d(\mathcal P(x,\cdot),\mathcal P(y,\cdot))\le c\,d(x,y)d(P(x,⋅),P(y,⋅))≤cd(x,y) for a fixed c<1c<1c<1. A set SSS is ddd-small if d(P(x,⋅),P(y,⋅))≤s<1d(\mathcal P(x,\cdot),\mathcal P(y,\cdot))\le s<1d(P(x,⋅),P(y,⋅))≤s<1 for all x,y∈Sx,y\in Sx,y∈S.

The distance of this mission is a weighted path length (3.6). For unit-speed C1C^1C1 paths ψ:[0,T]→H\psi:[0,T]\to Hψ:[0,T]→H from xxx to yyy,

dˉ(x,y)=inf⁡T,ψ1ε∫0Teη∥ψ(t)∥ dt,d=1∧dˉ,d~(x,y)=d(x,y) (1+V(x)+V(y)),\bar d(x,y)=\inf_{T,\psi}\frac1\varepsilon\int_0^T e^{\eta\|\psi(t)\|}\,dt,\qquad d=1\wedge\bar d,\qquad \tilde d(x,y)=\sqrt{d(x,y)\,(1+V(x)+V(y))},dˉ(x,y)=T,ψinf​ε1​∫0T​eη∥ψ(t)∥dt,d=1∧dˉ,d~(x,y)=d(x,y)(1+V(x)+V(y))​,

with V(x)=∥x∥iV(x)=\|x\|^iV(x)=∥x∥i (i≥1i\ge1i≥1) or V(x)=ev∥x∥V(x)=e^{v\|x\|}V(x)=ev∥x∥ (v>0v>0v>0).

There are two assumptions. Assumption 2.10 says that for ∥x∥≥R\|x\|\ge R∥x∥≥R every proposal within distance r(∥x∥)r(\|x\|)r(∥x∥) of the proposal mean (1−ρ)x(1-\rho)x(1−ρ)x is accepted with probability at least eαle^{\alpha_l}eαl​, where r(s)≤ρ2sr(s)\le\frac\rho2 sr(s)≤2ρ​s. Assumption 2.13 says that e−Φe^{-\Phi}e−Φ is γ\gammaγ-integrable and that φ(r)=sup⁡x≠y∈Br(0)∣Φ(x)−Φ(y)∣/∥x−y∥≤Mκeκr\varphi(r)=\sup_{x\ne y\in B_r(0)}|\Phi(x)-\Phi(y)|/\|x-y\|\le M_\kappa e^{\kappa r}φ(r)=supx=y∈Br​(0)​∣Φ(x)−Φ(y)∣/∥x−y∥≤Mκ​eκr for every κ>0\kappa>0κ>0.

Formalization targets

Goal: Theorem 2.14

Under Assumption 2.10 with r(s)=rsar(s)=r s^ar(s)=rsa (r>0r>0r>0, a∈(12,1)a\in(\frac12,1)a∈(21​,1)) and Assumption 2.13:

  1. μ\muμ and every μm\mu_mμm​ are the unique invariant probability measures of P\mathcal PP and Pm\mathcal P_mPm​.
  2. For η\etaη small enough and then ε\varepsilonε small enough, there is one n~\tilde nn~ such that for every mmm and all probability measures ν1,ν2\nu_1,\nu_2ν1​,ν2​
d~(ν1Pn~,ν2Pn~)≤12d~(ν1,ν2),d~(ν1Pmn~,ν2Pmn~)≤12d~(ν1,ν2).\tilde d(\nu_1\mathcal P^{\tilde n},\nu_2\mathcal P^{\tilde n})\le\tfrac12\tilde d(\nu_1,\nu_2),\qquad \tilde d(\nu_1\mathcal P_m^{\tilde n},\nu_2\mathcal P_m^{\tilde n})\le\tfrac12\tilde d(\nu_1,\nu_2).d~(ν1​Pn~,ν2​Pn~)≤21​d~(ν1​,ν2​),d~(ν1​Pmn~​,ν2​Pmn~​)≤21​d~(ν1​,ν2​).

The essential point is that n~\tilde nn~ does not depend on mmm.

Milestones

  • Lemma 3.5 (1)–(3): the comparison of dˉ\bar ddˉ with ∥x−y∥eη(∥x∥∨∥y∥)/ε\|x-y\|e^{\eta(\|x\|\vee\|y\|)}/\varepsilon∥x−y∥eη(∥x∥∨∥y∥)/ε, and the contraction of dˉ\bar ddˉ under a common-noise pCN step,
dˉ(px,py)≤(1−2δ)1/2eη[−ρ(∥x∥∨∥y∥)+2δ∥ξ∥+J]dˉ(x,y).\bar d(p_x,p_y)\le(1-2\delta)^{1/2}e^{\eta[-\rho(\|x\|\vee\|y\|)+\sqrt{2\delta}\|\xi\|+J]}\bar d(x,y).dˉ(px​,py​)≤(1−2δ)1/2eη[−ρ(∥x∥∨∥y∥)+2δ​∥ξ∥+J]dˉ(x,y).
  • Lemma 3.6: P\mathcal PP and every Pm\mathcal P_mPm​ are ddd-contracting with one constant.
  • Lemma 3.7: bounded sets are ddd-small for Pn\mathcal P^nPn and every Pmn\mathcal P_m^nPmn​, with nnn and sss independent of mmm.

The paper derives Theorem 2.14 from these, from Lemma 3.2 (Lyapunov functions) and from Proposition 2.6 (the weak Harris theorem of Hairer, Mattingly and Scheutzow 2011). The last two are posed in mission I of this series.

Significance

A Wasserstein spectral gap uniform in mmm means that the number of pCN steps needed to reach a given accuracy does not grow with the discretisation level. In §4 of the paper this yields a strong law of large numbers and a central limit theorem for pCN ergodic averages, and, combined with Theorem 2.15, an Lμ2L^2_\muLμ2​-spectral gap and mean-square error bounds, all uniform in the dimension. The locally Lipschitz case is the one that covers potentials of polynomial growth.

The result is proved in the paper. It has not been formalized. A formalization would produce a reusable weak-Harris toolkit for Markov kernels on Polish spaces, and a machine-checked account of the path-metric construction, which the paper adapts from Hairer and Majda (2010) and Hairer, Mattingly and Scheutzow (2011). It would also check the paper's proof, whose printed Lemma 3.5 (3) contains a typographical slip in the exponent; the formalized bound is the one the proof derives.

Difficulty

With Φ\PhiΦ only locally Lipschitz, no single ε\varepsilonε makes 1∧∥x−y∥/ε1\wedge\|x-y\|/\varepsilon1∧∥x−y∥/ε contracting. The probability that exactly one of two coupled chains accepts is controlled by ∣Φ(x)−Φ(y)∣|\Phi(x)-\Phi(y)|∣Φ(x)−Φ(y)∣, and far from the origin this exceeds any fixed multiple of ∥x−y∥\|x-y\|∥x−y∥. The weighted path distance makes far-out points "close" only when they are very close in norm. But then even the "both accept" move can increase the distance, because the weight eη∥⋅∥e^{\eta\|\cdot\|}eη∥⋅∥ changes along the move. Showing that the shrinkage (1−ρ)(1-\rho)(1−ρ) of the proposal mean beats this, while the Gaussian noise term stays integrable uniformly in mmm, is the content of Lemmas 3.5 and 3.6. The order of the choices (RRR, then η\etaη, then ε\varepsilonε) is essential.

Formalization scope

  • HHH is a real Hilbert space that is complete, separable and Borel. γ\gammaγ is Mathlib's IsGaussian with mean zero. (ei)(e_i)(ei​) is a HilbertBasis ℕ whose vectors are pairwise uncorrelated under the covariance bilinear form.
  • PmHP_mHPm​H is a submodule. γm\gamma_mγm​ is the pushforward under the orthogonal projection. Pm\mathcal P_mPm​ lives on the subtype PmHP_mHPm​H, with Φ\PhiΦ and VVV restricted to it and ddd restricted from HHH.
  • μ\muμ and μm\mu_mμm​ are Mathlib's normalised tilted measures. The goal concludes that they are probability measures.
  • The Metropolis–Hastings kernel is built from Mathlib's Kernel.withDensity. Kernel powers are composition powers, and νPn\nu\mathcal P^nνPn is Measure.bind.
  • Wasserstein lifts and dˉ\bar ddˉ are infima in [0,∞][0,\infty][0,∞], so no junk zero arises from an empty or unbounded set. Unit speed is imposed on (0,T)(0,T)(0,T), so dˉ(x,x)=0\bar d(x,x)=0dˉ(x,x)=0 through the constant path with T=0T=0T=0.
  • Contraction, smallness and Lyapunov constants are explicit arguments. "Uniform in mmm" is "one constant for all mmm".
  • "Small enough" is ∃η0>0 ∀η∈(0,η0] ∃ε0>0 ∀ε∈(0,ε0]\exists\eta_0>0\ \forall\eta\in(0,\eta_0]\ \exists\varepsilon_0>0\ \forall\varepsilon\in(0,\varepsilon_0]∃η0​>0 ∀η∈(0,η0​] ∃ε0​>0 ∀ε∈(0,ε0​].
  • Several readings are disclosed. "r∈Rr\in\mathbb Rr∈R" is read as r>0r>0r>0, i∈Ni\in\mathbb Ni∈N as i≥1i\ge1i≥1, and v>0v>0v>0. Assumption 2.10 is stated for s≥0s\ge0s≥0, the domain of rrr, and requires r(s)>0r(s)>0r(s)>0 for s>0s>0s>0 so that the acceptance balls are nonempty. Its infimum and the supremum φ(r)\varphi(r)φ(r) of Assumption 2.13 are stated pointwise.
  • The general form of Lemma 3.6 has a disclosed radius-floor hypothesis: r(s)r(s)r(s) stays bounded away from zero for large sss. The printed proof needs a uniform positive probability for the proposal ball; this hypothesis supplies it. Theorem 2.14's power-law radius already meets it.
  • The formalization rules out trivialising readings. The tilted measure cannot be the zero measure, because the goal asserts it is a probability measure. Uniqueness is among probability measures. V≡1V\equiv1V≡1 is excluded. A n~\tilde nn~ chosen after mmm would give a per-dimension gap, not this theorem.
  • A complete development needs:
    • the weak Harris theorem (Proposition 2.6, mission I);
    • Gaussian tail bounds of Fernique type, with the dimension-monotonicity of Lemma 3.1 (mission I);
    • the Lyapunov drift Lemma 3.2 (mission I);
    • calculus of C1C^1C1 paths in Hilbert space;
    • synchronous couplings of Metropolis–Hastings kernels.
  • The Harris and coupling layers are reusable well beyond this paper. Contributions to any of them are welcome.

Selected references

  • M. Hairer, A. M. Stuart and S. J. Vollmer, Spectral gaps for a Metropolis–Hastings algorithm in infinite dimensions, arXiv:1112.1392v4 (2014), published in Ann. Appl. Probab. 24(6) (2014) 2455–2490, https://doi.org/10.1214/13-AAP982. https://arxiv.org/abs/1112.1392v4
  • M. Hairer, J. C. Mattingly and M. Scheutzow, Asymptotic coupling and a general form of Harris' theorem with applications to stochastic delay equations, Probab. Theory Related Fields 149 (2011) 223–259. MR2773030, https://mathscinet.ams.org/mathscinet-getitem?mr=2773030
  • M. Hairer and A. J. Majda, A simple framework to justify linear response theory, Nonlinearity 23 (2010) 909–922. MR2602020, https://mathscinet.ams.org/mathscinet-getitem?mr=2602020
  • A. Beskos, G. O. Roberts, A. M. Stuart and J. Voss, MCMC methods for diffusion bridges, Stoch. Dyn. 8 (2008) 319–350. MR2444507, https://mathscinet.ams.org/mathscinet-getitem?mr=2444507
  • S. L. Cotter, G. O. Roberts, A. M. Stuart and D. White, MCMC methods for functions: modifying old algorithms to make them faster, Statist. Sci. 28 (2013) 424–446. https://doi.org/10.1214/13-STS421
13 thms1 active userReviewed
ProbabilityStatistics·Captain: mikedeng1

Spectral Gaps for a Metropolis–Hastings Algorithm in Infinite Dimensions III: The L²-Spectral Gap of Random Walk Metropolis for a Gaussian Target Vanishes Faster Than Any Power of the DimensionResearch Paper

Motivation

Bayesian inverse problems and conditioned diffusions lead to probability measures on function spaces, and these are sampled in practice through finite-dimensional approximations on Rm\mathbb R^mRm with mmm large. A Markov chain Monte Carlo method is useful in this setting only if its convergence rate does not deteriorate as the discretisation is refined. Random walk Metropolis (RWM), the default Metropolis–Hastings algorithm, is known from diffusion-limit analyses to need step sizes of order 1/m1/m1/m for Gaussian-type targets (Roberts, Gelman and Gilks 1997; Beskos, Roberts and Stuart 2009). Those results describe a scaling limit of the chain in stationarity; they do not bound the speed of convergence of the chain itself.

Hairer, Stuart and Vollmer (arXiv:1112.1392v4, Ann. Appl. Probab. 2014) prove that the preconditioned Crank–Nicolson algorithm has a spectral gap bounded below uniformly in mmm, and in §2.4 they show by an explicit example that RWM does not: its L2L^2L2-spectral gap vanishes as m→∞m\to\inftym→∞, faster than any power of mmm unless the step size decays at least like 1/m1/m1/m. This mission formalizes that negative result.

Setting

Fix m≥1m\ge1m≥1. The target is the Gaussian measure

γm=L(∑i=1m1i ξiei),ξi i.i.d. N(0,1),\gamma_m=\mathcal L\Bigl(\sum_{i=1}^m\tfrac1i\,\xi_i e_i\Bigr),\qquad \xi_i\ \text{i.i.d.}\ \mathcal N(0,1),γm​=L(i=1∑m​i1​ξi​ei​),ξi​ i.i.d. N(0,1),

on Rm\mathbb R^mRm: the coordinates x1,…,xmx_1,\dots,x_mx1​,…,xm​ are independent and xi∼N(0,1/i2)x_i\sim\mathcal N(0,1/i^2)xi​∼N(0,1/i2). Its covariance CCC satisfies ⟨x,C−1x⟩=∑i=1mi2xi2\langle x,C^{-1}x\rangle=\sum_{i=1}^m i^2x_i^2⟨x,C−1x⟩=∑i=1m​i2xi2​.

A Metropolis–Hastings kernel with proposal kernel QQQ and acceptance probability α(x,y)∈[0,1]\alpha(x,y)\in[0,1]α(x,y)∈[0,1] is

P(x,dz)=Q(x,dz) α(x,z)+δx(dz)∫(1−α(x,u)) Q(x,du).P(x,dz)=Q(x,dz)\,\alpha(x,z)+\delta_x(dz)\int\bigl(1-\alpha(x,u)\bigr)\,Q(x,du).P(x,dz)=Q(x,dz)α(x,z)+δx​(dz)∫(1−α(x,u))Q(x,du).

RWM with step size δ>0\delta>0δ>0 proposes y=x+2δ ξy=x+\sqrt{2\delta}\,\xiy=x+2δ​ξ with ξ∼γm\xi\sim\gamma_mξ∼γm​ and accepts with probability α(x,y)=1∧exp⁡(∑i=1mi22(xi2−yi2))\alpha(x,y)=1\wedge\exp\bigl(\sum_{i=1}^m\frac{i^2}{2}(x_i^2-y_i^2)\bigr)α(x,y)=1∧exp(∑i=1m​2i2​(xi2​−yi2​)). The resulting kernel PmP_mPm​ is reversible with respect to γm\gamma_mγm​. The expected acceptance probability from xxx is α(x)=∫α(x,y) Q(x,dy)\alpha(x)=\int\alpha(x,y)\,Q(x,dy)α(x)=∫α(x,y)Q(x,dy).

For a Markov kernel PPP with invariant probability measure μ\muμ, write (Pf)(x)=∫f(y) P(x,dy)(Pf)(x)=\int f(y)\,P(x,dy)(Pf)(x)=∫f(y)P(x,dy), μ(f)=∫f dμ\mu(f)=\int f\,d\muμ(f)=∫fdμ and

β=∥P∥L02→L02=sup⁡f∈Lμ2∥Pf−μ(f)∥2∥f−μ(f)∥2;\beta=\|P\|_{L^2_0\to L^2_0}=\sup_{f\in L^2_\mu}\frac{\|Pf-\mu(f)\|_2}{\|f-\mu(f)\|_2};β=∥P∥L02​→L02​​=f∈Lμ2​sup​∥f−μ(f)∥2​∥Pf−μ(f)∥2​​;

1−β1-\beta1−β is the Lμ2L^2_\muLμ2​-spectral gap. The conductance is

C=inf⁡0<μ(A)≤1/2∫AP(x,Ac) dμ(x)μ(A).\mathsf C=\inf_{0<\mu(A)\le1/2}\frac{\int_A P(x,A^c)\,d\mu(x)}{\mu(A)}.C=0<μ(A)≤1/2inf​μ(A)∫A​P(x,Ac)dμ(x)​.

The step size is scaled with the dimension as δm=s m−a\delta_m=s\,m^{-a}δm​=sm−a with s>0s>0s>0 and a≥0a\ge0a≥0; βm\beta_mβm​ is β\betaβ for PmP_mPm​ and μ=γm\mu=\gamma_mμ=γm​.

Formalization targets

Goal: Theorem 2.17 (1)

For a∈[0,1)a\in[0,1)a∈[0,1), s>0s>0s>0 and every ppp, there is a constant KKK, independent of mmm, such that

1−βm≤K m−pfor all m≥1.1-\beta_m\le K\,m^{-p}\qquad\text{for all } m\ge1 .1−βm​≤Km−pfor all m≥1.

Companion: Theorem 2.17 (2)

For a≥1a\ge1a≥1 and s>0s>0s>0 there is a constant KKK, independent of mmm, such that 1−βm≤K m−a/21-\beta_m\le K\,m^{-a/2}1−βm​≤Km−a/2 for all m≥1m\ge1m≥1.

Milestones

  1. Cheeger's inequality (2.7): 1−β≤2C1-\beta\le2\mathsf C1−β≤2C for a μ\muμ-reversible Markov kernel (cited by the paper).
  2. The conductance displays of p. 15: C≤inf⁡A∫Aα(x) μ(dx)/μ(A)\mathsf C\le\inf_A\int_A\alpha(x)\,\mu(dx)/\mu(A)C≤infA​∫A​α(x)μ(dx)/μ(A) and C≤sup⁡x∈Bα(x)\mathsf C\le\sup_{x\in B}\alpha(x)C≤supx∈B​α(x) for 0<μ(B)≤120<\mu(B)\le\frac120<μ(B)≤21​.
  3. Proposition 2.16: 1−β≤2C≤2sup⁡x∈Bα(x)1-\beta\le2\mathsf C\le2\sup_{x\in B}\alpha(x)1−β≤2C≤2supx∈B​α(x) (2.8) and 1−β≤2C≤2inf⁡A∫AQ(x,Ac) dμ/μ(A)1-\beta\le2\mathsf C\le2\inf_A\int_AQ(x,A^c)\,d\mu/\mu(A)1−β≤2C≤2infA​∫A​Q(x,Ac)dμ/μ(A) (2.9) for any Metropolis–Hastings kernel.
  4. Proposition 2.16, second branch of (2.8): 1−β≤2C≤2 Eμα(x)1-\beta\le2\mathsf C\le2\,\mathbb E_\mu\alpha(x)1−β≤2C≤2Eμ​α(x), for atomless μ\muμ.
  5. The acceptance bound of the proof of Theorem 2.17: for λ∈[0,1]\lambda\in[0,1]λ∈[0,1], α(x)≤(1+2λδ)−m/2exp⁡(∑i=1mδλ2i2xi2/(2δλ+1))\alpha(x)\le(1+2\lambda\delta)^{-m/2}\exp\bigl(\sum_{i=1}^m\delta\lambda^2i^2x_i^2/(2\delta\lambda+1)\bigr)α(x)≤(1+2λδ)−m/2exp(∑i=1m​δλ2i2xi2​/(2δλ+1)).
  6. The half-space estimate for part (2): A={x1≥0}A=\{x_1\ge0\}A={x1​≥0} has γm(A)=12\gamma_m(A)=\frac12γm​(A)=21​ and ∫AQ(x,Ac) γm(dx)≤cδ/(δ+1)\int_AQ(x,A^c)\,\gamma_m(dx)\le c\sqrt{\delta/(\delta+1)}∫A​Q(x,Ac)γm​(dx)≤cδ/(δ+1)​ with ccc independent of mmm and δ\deltaδ.
  7. Theorem 2.17 (2).

Significance

The theorem separates two algorithms that differ only in their proposal. For the same Gaussian targets (the case Φ=0\Phi=0Φ=0 of the paper), the preconditioned Crank–Nicolson algorithm has an L2L^2L2-spectral gap bounded below uniformly in mmm, while RWM's gap collapses. For a reversible chain the spectral gap determines the worst-case asymptotic variance of ergodic averages over Lμ2L^2_\muLμ2​ observables, so for RWM at a fixed step-size decay a<1a<1a<1 that worst case grows faster than any polynomial in mmm. Proposition 2.16 is reusable on its own: it bounds the spectral gap of every Metropolis–Hastings kernel by the acceptance probability or by the proposal's flow, on any measurable space. It also turns diffusion-limit statements about acceptance rates into bounds on the spectral gap.

The results are proved in the paper (Cheeger's inequality is cited from Lawler and Sokal 1988 and Sinclair and Jerrum 1989). No part has a machine-checked proof. The prove2me library has Cheeger's inequality for finite-state reversible chains (MarkovMixing.cheeger_upper), not for kernels on general measurable spaces with the L02L^2_0L02​ operator norm. A complete development would contain the first formal Cheeger upper bound in that generality, the Metropolis–Hastings conductance bounds, and an explicit Gaussian computation for RWM.

Difficulty

Three steps resist a direct transcription. First, Cheeger's inequality has to be stated and proved for the Markov operator on Lμ2L^2_\muLμ2​ of a general measurable space: PfPfPf is a Bochner integral that is finite only μ\muμ-almost everywhere, and every identity relating β\betaβ to the flow between sets passes through invariance and centring. Second, the acceptance bound is an exact Gaussian computation in mmm dimensions with explicit constants, where an off-by-one in the coordinate weights changes the target. Third, the asymptotic step: the acceptance probability is small only on sets where the weighted quantity ∑ii2xi2\sum_i i^2x_i^2∑i​i2xi2​ is controlled, but a typical draw from γm\gamma_mγm​ has ∑ii2xi2\sum_i i^2x_i^2∑i​i2xi2​ of order mmm. A bound on α(x)\alpha(x)α(x) over all of Rm\mathbb R^mRm, or over a Euclidean ball, gives no decay, and the set used in Proposition 2.16 must keep γm\gamma_mγm​-mass bounded away from 000 uniformly in mmm while α\alphaα decays on it faster than every power of mmm.

Formalization scope

  • State space Fin m → ℝ. The Lean coordinate i : Fin m is the paper's xi+1x_{i+1}xi+1​, with variance 1/(i+1)21/(i+1)^21/(i+1)2 and weight (i+1)2(i+1)^2(i+1)2. The target is Measure.pi of gaussianReal 0 (1/(i+1)^2).
  • Kernels. The Metropolis–Hastings kernel is Q.withDensity α + Kernel.id.withDensity (rejection mass), with α : X → X → ℝ≥0∞. Every statement assumes α ≤ 1 and joint measurability (without it withDensity is the zero kernel). The RWM acceptance probability is Algorithm 2's with Φ=0\Phi=0Φ=0, with the exponential (display (1.4) omits it, a misprint).
  • β\betaβ is the real supremum of the ratios over measurable f∈Lμ2f\in L^2_\muf∈Lμ2​ with ∥f−μ(f)∥2≠0\|f-\mu(f)\|_2\ne0∥f−μ(f)∥2​=0, norms via eLpNorm. For a μ\muμ-invariant Markov kernel the ratio set is bounded by 111, and for γm\gamma_mγm​ with m≥1m\ge1m≥1 it is nonempty, so βm\beta_mβm​ is not a junk value.
  • Conductance is in [0,∞][0,\infty][0,∞] with the infimum over measurable AAA with 0<μ(A)≤120<\mu(A)\le\frac120<μ(A)≤21​. The condition 0<μ(A)0<\mu(A)0<μ(A) is added: with 0/0=00/0=00/0=0 it would otherwise force C=0\mathsf C=0C=0. Cheeger's inequality and Proposition 2.16 are stated in [0,∞][0,\infty][0,∞] with max⁡(1−β,0)\max(1-\beta,0)max(1−β,0) on the left.
  • Added hypotheses and readings. "δm∼m−a\delta_m\sim m^{-a}δm​∼m−a" is read as δm=s m−a\delta_m=s\,m^{-a}δm​=sm−a for a fixed s>0s>0s>0 (p. 14), and KKK may depend on sss. The bounds hold for all m≥1m\ge1m≥1, not only for large mmm. "Any ppp" is every real ppp. Proposition 2.16's "Metropolis–Hastings kernel for a target μ\muμ" is formalized as reversibility of PPP with respect to μ\muμ. Its undefined middle term 1−Λ1-\Lambda1−Λ is dropped. Its Eμα\mathbb E_\mu\alphaEμ​α branch carries an added atomlessness hypothesis on μ\muμ.
  • Trivializations ruled out. No statement involves a Bochner integral of a possibly non-integrable function or a real supremum over a possibly empty or unbounded set at the relevant parameters. KKK is quantified before mmm. The target is a probability measure by construction.
  • Infrastructure. The development needs the Markov operator on Lμ2L^2_\muLμ2​, Jensen's inequality for kernels, Gaussian integrals on Rm\mathbb R^mRm with explicit constants, and tail estimates for γm\gamma_mγm​ in weighted norms. Cheeger's inequality and Proposition 2.16 are reusable beyond this mission. Contributions of any of the milestones, or of alternative proofs of the goal, are welcome.

Selected references

  • M. Hairer, A. M. Stuart and S. J. Vollmer, Spectral gaps for a Metropolis–Hastings algorithm in infinite dimensions, Ann. Appl. Probab. 24(6) (2014) 2455–2490; arXiv:1112.1392v4. https://arxiv.org/abs/1112.1392
  • G. F. Lawler and A. D. Sokal, Bounds on the L2L^2L2 spectrum for Markov chains and Markov processes: a generalization of Cheeger's inequality, Trans. Amer. Math. Soc. 309 (1988) 557–580. https://doi.org/10.1090/S0002-9947-1988-0930082-9
  • A. Sinclair and M. Jerrum, Approximate counting, uniform generation and rapidly mixing Markov chains, Inform. and Comput. 82 (1989) 93–133. https://doi.org/10.1016/0890-5401(89)90067-9
  • A. Beskos, G. Roberts and A. Stuart, Optimal scalings for local Metropolis–Hastings chains on nonproduct targets in high dimensions, Ann. Appl. Probab. 19(3) (2009) 863–898. https://doi.org/10.1214/08-AAP563
  • G. O. Roberts, A. Gelman and W. R. Gilks, Weak convergence and optimal scaling of random walk Metropolis algorithms, Ann. Appl. Probab. 7(1) (1997) 110–120. https://doi.org/10.1214/aoap/1034625254
12 thms1 active userReviewed
Dynamic ProgrammingOperations ResearchProbability·Captain: mikedeng1

Partially Observable Total-Cost Markov Decision Processes with Weakly Continuous Transition Probabilities 2: Setwise-Continuous Observations Do Not Make the Belief Transition Weakly ContinuousResearch Paper

Motivation

A partially observable Markov decision process describes a controlled system whose state is hidden but whose observations are available to a decision maker. The distribution of the hidden state conditional on the observations is the belief state. Replacing the hidden state by this distribution turns the model into a fully observed decision process, provided the resulting belief transition has the regularity needed by the usual existence and dynamic programming results. Feinberg, Kasyanov, and Zgurovsky give sufficient conditions for that regularity in their study of total-cost POMDPs, Theorems 3.6 and 3.7.

Their observation-kernel condition is continuity in total variation. Example 4.1 of the same paper tests its sharpness: the authors construct a two-state model with setwise-continuous observations for which the belief transition fails to be weakly continuous. This matters when selecting assumptions for a POMDP theorem. Setwise convergence already requires convergence of the probabilities of every fixed observation event, so it can look adequate for belief updates. The example shows that it is still too weak for the stated conclusion.

Setting

The hidden state space is X={1,2}X=\{1,2\}X={1,2} and the observation space is Y=[0,1]Y=[0,1]Y=[0,1] with Lebesgue probability measure mmm. The action space is A={0}∪{1/n:n=1,2,…}A=\{0\}\cup\{1/n:n=1,2,\ldots\}A={0}∪{1/n:n=1,2,…}, equipped with the topology inherited from R\mathbb RR. The state does not move: its transition law P(⋅∣x,a)P(\cdot\mid x,a)P(⋅∣x,a) is the point mass at xxx for every action. Its continuity in total variation is immediate from this fixed-state behavior.

At hidden state 1 the observation law is always mmm. At hidden state 2, action zero also produces mmm. Under action 1/n1/n1/n, state 2 produces a law m(n)=f(n)mm^{(n)}=f^{(n)}mm(n)=f(n)m. The density f(n)f^{(n)}f(n) equals zero on the open left half of each cell in the 2n2^n2n-part dyadic partition and equals two elsewhere, including the cell endpoints. Equation (4.1) gives the exact intervals. These laws oscillate more rapidly as nnn grows, and the paper establishes m(n)(C)→m(C)m^{(n)}(C)\to m(C)m(n)(C)→m(C) for every fixed Borel set CCC. Thus the observation kernel Q(⋅∣a,x)Q(\cdot\mid a,x)Q(⋅∣a,x) is setwise continuous even at the accumulation action zero. Example 4.1, pp. 13–14.

A belief z∈P(X)z\in\mathcal P(X)z∈P(X) assigns a probability to each hidden state. For a current belief zzz and an action aaa, the joint law R(⋅∣z,a)R(\cdot\mid z,a)R(⋅∣z,a) covers the next state and its observation, and R′(⋅∣z,a)R'(\cdot\mid z,a)R′(⋅∣z,a) is the observation marginal. A filter H(z,a,y)H(z,a,y)H(z,a,y) is a conditional law of the next state given observation yyy: it satisfies the disintegration identity (3.3). The belief transition qH(⋅∣z,a)q_H(\cdot\mid z,a)qH​(⋅∣z,a) is the law of the random posterior H(z,a,y)H(z,a,y)H(z,a,y) when yyy has law R′(⋅∣z,a)R'(\cdot\mid z,a)R′(⋅∣z,a). These objects are defined in (3.1)–(3.5) of the paper. Different versions of HHH agree only almost everywhere for each current belief and action, yet they yield the same belief transition.

Formalization targets

Example 4.1: failure of weak belief continuity

At the uniform belief z=(1/2,1/2)z=(1/2,1/2)z=(1/2,1/2), action zero reveals nothing, while every positive-index action produces the same two-point posterior law:

qH(⋅∣z,0)=δz,qH(⋅∣z,1/n)=34δ(1/3,2/3)+14δ(1,0)(n≥1).q_H(\cdot\mid z,0)=\delta_z,\qquad q_H(\cdot\mid z,1/n)=\tfrac34\delta_{(1/3,2/3)}+\tfrac14\delta_{(1,0)} \quad(n\ge1).qH​(⋅∣z,0)=δz​,qH​(⋅∣z,1/n)=43​δ(1/3,2/3)​+41​δ(1,0)​(n≥1).

The mission's goal asserts that PPP is total-variation continuous, QQQ is a setwise-continuous probability kernel, a valid filter exists, and the positive-index laws fail to converge weakly to the zero-action law as 1/n→01/n\to01/n→0. It states the failure for every filter satisfying (3.3), so it is a property of the POMDP rather than of a selected version of its conditional distribution. The milestone list follows the paper's assertions: setwise convergence of the dyadic laws, setwise continuity of QQQ, the two posterior values almost everywhere, and the explicit laws of qHq_HqH​.

Significance

The example identifies a precise boundary of the sufficient condition in Theorems 3.6 and 3.7. Those theorems use total-variation continuity of the observation law to obtain a weakly continuous belief transition; replacing that premise by setwise continuity makes their continuity conclusion false in this model. The explicit law also shows the size of the discrepancy: the positive-index belief transition has two fixed atoms, while the zero-action transition is concentrated at the original belief. Example 4.1, pp. 13–14.

A machine-checked development would add reusable definitions of sequential kernel continuity, the joint state-observation law, conditional filters, and the belief transition on the Borel space of probability measures. It would also formalize a concrete setwise-convergent sequence of densities whose induced posterior laws do not converge weakly. The paper proves the example; the Lean statements in this proposal are open proof targets, with no completed machine-checked proof claimed here.

Difficulty

Setwise continuity controls Q(C∣a,x)Q(C\mid a,x)Q(C∣a,x) for each fixed Borel set CCC. A posterior is a probability measure determined by the observation, and the belief transition tests the distribution of those posterior measures. The observation regions distinguishing the states change with nnn, so convergence on fixed observation sets does not imply convergence of their posterior distributions. This is the gap in the immediate argument that tries to pass from setwise convergence of QQQ to weak convergence of qHq_HqH​.

The conditional filter adds a second precision issue. Equation (3.3) determines its values only almost surely under the observation marginal, and different versions may disagree on null observations. Any claim about a posterior value at every yyy would exceed the source; the posterior milestone therefore has an almost-everywhere qualifier, while the goal quantifies over every valid version of HHH.

Formalization scope

Lean represents XXX by Fin 2, with 0 for the paper's state 1 and 1 for state 2; YYY by unitInterval; and AAA by the specified subtype of R\mathbb RR. Positive integers are a subtype of N\mathbb NN, so the density and the action 1/n1/n1/n are never evaluated at n=0n=0n=0. The measure mmm is volume on unitInterval, which has total mass one. The density follows (4.1) literally, with open zero intervals and density two at their endpoints. The general transition PPP and observation law QQQ use Mathlib kernels; the example instantiates them directly. The goal includes the probability-kernel property of the concrete QQQ and existence of a valid HHH, guarding the universal filter clause against vacuity.

Weak, setwise, and total-variation convergence are sequential, as on page 4. The weak test class consists of bounded continuous real functions; the setwise test class consists of all Borel sets. The target space of qHq_HqH​ carries the Borel σ-algebra of the weak topology on P(X)\mathcal P(X)P(X), stated explicitly because Mathlib's default measurable structure on that type is the Giry σ-algebra. The filter predicate requires measurability in yyy for fixed (z,a)(z,a)(z,a) in both structures. Joint measurability in (z,a,y)(z,a,y)(z,a,y) is described in the paper but is not encoded in this local predicate; only the fixed-parameter property is used by (3.3) and (3.5). This choice and its quantifier effect are recorded in the moderation notes.

No cost, initial observation kernel, initial prior, or discount factor enters Example 4.1. Contributions on the dyadic law, kernel continuity, conditional posterior values, and the resulting belief-law identity are all within scope. A filter hard-coded to return the prior belief, a discrete topology on AAA, or a claim for only one version of HHH would change the counterexample and is excluded by the definitions and goal.

Selected references

  • E. A. Feinberg, P. O. Kasyanov, and M. Z. Zgurovsky, Partially Observable Total-Cost Markov Decision Processes with Weakly Continuous Transition Probabilities, Mathematics of Operations Research 41(2), 2016; arXiv:1401.2168v2. This mission follows the pinned preprint's numbering and pages.
6 thms1 active userReviewed
Dynamic ProgrammingOperations ResearchProbability·Captain: mikedeng1

Partially Observable Total-Cost Markov Decision Processes with Weakly Continuous Transition Probabilities 1: The Belief MDP Satisfies Assumption (W*) Under Total-Variation-Continuous ObservationsResearch Paper

Motivation

A partially observable Markov decision process (POMDP) models a controller that cannot see the state of the system it steers, only noisy observations of it. Inventory control with unrecorded demand, machine maintenance with imperfect inspection, and target tracking from sensor readings all have this form. The standard way to solve a POMDP is to replace it by its belief MDP, also called the completely observable MDP (COMDP): the state becomes the posterior distribution of the hidden state given everything observed so far. The belief MDP is fully observed, so the theory of Markov decision processes applies to it, but only if the belief MDP satisfies that theory's hypotheses. For MDPs with Borel state and action spaces, unbounded costs and possibly noncompact action sets, the hypothesis that yields optimality equations, convergence of value iterations and stationary optimal policies is Assumption (W*) of Feinberg, Kasyanov and Zadoianchuk (Math. Oper. Res. 2012): the cost is K\mathbb KK-inf-compact and the transition probability is weakly continuous.

Before Feinberg, Kasyanov and Zgurovsky (arXiv:1401.2168v2, published in Math. Oper. Res. 41(2), 2016), sufficient conditions for the belief MDP to be weakly continuous required a weakly continuous filter (Hernández-Lerma, 1989, as recalled on p. 11 of arXiv:1401.2168v2), a condition that fails in many natural models. The paper replaces it by a condition on the primitive data of the POMDP.

Setting

Let X\mathbb XX (states), Y\mathbb YY (observations) and A\mathbb AA (actions) be Borel subsets of Polish spaces. Let P(dx′∣x,a)P(dx' \mid x, a)P(dx′∣x,a) be the transition kernel on X\mathbb XX given X×A\mathbb X \times \mathbb AX×A, Q(dy∣a,x)Q(dy \mid a, x)Q(dy∣a,x) the observation kernel on Y\mathbb YY given A×X\mathbb A \times \mathbb XA×X, and c:X×A→R∪{+∞}c : \mathbb X \times \mathbb A \to \mathbb R \cup \{+\infty\}c:X×A→R∪{+∞} a Borel one-step cost bounded below. P(X)\mathbb P(\mathbb X)P(X) is the set of probability measures on X\mathbb XX with the topology of weak convergence (∫f dμn→∫f dμ\int f\,d\mu_n \to \int f\,d\mu∫fdμn​→∫fdμ for bounded continuous fff). A sequence μn\mu_nμn​ converges setwise if μn(C)→μ(C)\mu_n(C) \to \mu(C)μn​(C)→μ(C) for every Borel CCC, and in the total variation if sup⁡f∣∫f dμn−∫f dμ∣→0\sup_f |\int f\,d\mu_n - \int f\,d\mu| \to 0supf​∣∫fdμn​−∫fdμ∣→0 over Borel fff with values in [−1,1][-1, 1][−1,1]. A kernel is weakly (setwise, total-variation) continuous if xn→xx_n \to xxn​→x implies the corresponding convergence of the measures.

For a belief z∈P(X)z \in \mathbb P(\mathbb X)z∈P(X) and an action aaa, let R(B×C∣z,a)=∫X∫BQ(C∣a,x′) P(dx′∣x,a) z(dx)R(B \times C \mid z, a) = \int_{\mathbb X} \int_B Q(C \mid a, x')\,P(dx' \mid x, a)\,z(dx)R(B×C∣z,a)=∫X​∫B​Q(C∣a,x′)P(dx′∣x,a)z(dx) be the joint law of the next state and observation (3.1), and R′(C∣z,a)=R(X×C∣z,a)R'(C \mid z, a) = R(\mathbb X \times C \mid z, a)R′(C∣z,a)=R(X×C∣z,a) its observation marginal (3.2). A filter is a kernel H(dx∣z,a,y)H(dx \mid z, a, y)H(dx∣z,a,y) with R(B×C∣z,a)=∫CH(B∣z,a,y) R′(dy∣z,a)R(B \times C \mid z, a) = \int_C H(B \mid z, a, y)\,R'(dy \mid z, a)R(B×C∣z,a)=∫C​H(B∣z,a,y)R′(dy∣z,a) (3.3): H(z,a,y)H(z, a, y)H(z,a,y) is the posterior after observing yyy. The belief MDP has state space P(X)\mathbb P(\mathbb X)P(X), action set A\mathbb AA, cost cˉ(z,a)=∫c(x,a) z(dx)\bar c(z, a) = \int c(x, a)\,z(dx)cˉ(z,a)=∫c(x,a)z(dx) (3.8), and transition q(⋅∣z,a)q(\cdot \mid z, a)q(⋅∣z,a), the law of H(z,a,y)H(z, a, y)H(z,a,y) when y∼R′(⋅∣z,a)y \sim R'(\cdot \mid z, a)y∼R′(⋅∣z,a) (3.5). A cost is K\mathbb KK-inf-compact on S1×S2\mathbb S_1 \times \mathbb S_2S1​×S2​ if its restriction to K×S2K \times \mathbb S_2K×S2​ has compact level sets for every compact K⊆S1K \subseteq \mathbb S_1K⊆S1​.

Formalization targets

Goal: Theorem 3.6

If either Assumption (D) (0≤α<10 \le \alpha < 10≤α<1) or Assumption (P) (c≥0c \ge 0c≥0 and 0≤α≤10 \le \alpha \le 10≤α≤1) holds, ccc is bounded below and K\mathbb KK-inf-compact on X×A\mathbb X \times \mathbb AX×A, PPP is weakly continuous and QQQ is continuous in the total variation, then the belief MDP satisfies Assumption (W*):

cˉ is bounded below and K-inf-compact on P(X)×A,(zn,an)→(z,a)  ⟹  q(⋅∣zn,an)→q(⋅∣z,a) weakly.\bar c \text{ is bounded below and } \mathbb K\text{-inf-compact on } \mathbb P(\mathbb X) \times \mathbb A, \qquad (z_n, a_n) \to (z, a) \implies q(\cdot \mid z_n, a_n) \to q(\cdot \mid z, a) \text{ weakly}.cˉ is bounded below and K-inf-compact on P(X)×A,(zn​,an​)→(z,a)⟹q(⋅∣zn​,an​)→q(⋅∣z,a) weakly.

The two halves

  • Cost half (Theorem 3.4, through Lemma 6.1 and Lemma 5.1(ii)): cˉ\bar ccˉ is bounded below by the same constant as ccc and K\mathbb KK-inf-compact.
  • Transition half (Theorem 3.7): under the hypotheses on PPP and QQQ, R′R'R′ is setwise continuous and Assumption (H) holds: whenever zn→zz_n \to zzn​→z and an→aa_n \to aan​→a, along a subsequence H(znk,ank,y)→H(z,a,y)H(z_{n_k}, a_{n_k}, y) \to H(z, a, y)H(znk​​,ank​​,y)→H(z,a,y) weakly for R′(⋅∣z,a)R'(\cdot \mid z, a)R′(⋅∣z,a)-almost every yyy. Theorem 3.5 then gives weak continuity of qqq. The route runs through Theorem 5.2, Lemma 5.3, Corollary 5.4, Theorem 5.5 and Lemma 5.6.

Significance

Theorem 3.6 supplies the hypothesis under which the cited general MDP theory gives stationary optimal policies, optimality equations and convergence of value iterations for total-cost POMDPs in terms of PPP, QQQ and ccc alone, without compactness of the action set, boundedness of the cost or continuity of the filter. It covers, for example, observation kernels with densities continuous in (a,x)(a, x)(a,x), and the paper applies it to inventory control with incomplete records and to the Kalman filter. Example 4.1 of the same paper shows that total-variation continuity of QQQ cannot be weakened to setwise continuity.

The result is proved in the paper; this proposal poses its formal statement and the supporting results as proof targets. Completing them would provide a checked bridge from POMDPs to the general MDP theory under weak continuity, together with reusable facts about weak, setwise and total-variation convergence: equicontinuity of integrals against weakly continuous kernels (Theorem 5.2), the convergence-in-probability criterion (Theorem 5.5), and the generalized Fatou lemmas.

Difficulty

The posterior H(z,a,y)H(z, a, y)H(z,a,y) is defined only for R′(⋅∣z,a)R'(\cdot \mid z, a)R′(⋅∣z,a)-almost every yyy and has no continuous version in general, so weak continuity of qqq cannot be read off from continuity of HHH. The tempting argument, that continuous PPP and QQQ make the Bayes posterior move continuously, fails: Y\mathbb YY need not be countable and QQQ need not have densities. What has to be controlled is the joint law RRR on sets (O1∖O2)×C(\mathcal O_1 \setminus \mathcal O_2) \times C(O1​∖O2​)×C uniformly in the observation set CCC; convergence for each fixed CCC is not enough. Turning this uniform control into almost-sure convergence of posteriors along a subsequence requires a diagonal argument over a countable base of X\mathbb XX. On the cost side, the difficulty is that ccc may take the value +∞+\infty+∞ and is not continuous; K\mathbb KK-inf-compactness has to be transported through integration against a weakly converging sequence of measures.

Formalization scope

All definitions live in FeinbergPOMDP.WStar.Model. X,Y,A\mathbb X, \mathbb Y, \mathbb AX,Y,A are separable metrizable spaces with their Borel σ-algebras (PolishSpace is not assumed, since Borel subsets of Polish spaces need not be Polish); X\mathbb XX is additionally nonempty and standard Borel in Theorems 3.6 and 3.7, where a filter must exist. P(X)\mathbb P(\mathbb X)P(X) is Mathlib's ProbabilityMeasure with the weak topology. All continuity notions are the paper's sequential ones; total variation is the supremum over Borel functions with values in [−1,1][-1, 1][−1,1], written in ε\varepsilonε–NNN form, while the set-indexed suprema of (5.15) and (5.17) keep their set form. The cost is written c=ℓ+fc = \ell + fc=ℓ+f with ℓ∈R\ell \in \mathbb Rℓ∈R and fff taking values in [0,∞][0, \infty][0,∞], so cˉ=ℓ+∫f dz\bar c = \ell + \int f\,dzcˉ=ℓ+∫fdz, exact even when c=+∞c = +\inftyc=+∞ on a set of positive measure. K\mathbb KK-inf-compactness is the published FeinbergLiang.ACOE.KInfCompact. A filter is a map H:P(X)→A→Y→P(X)H : \mathbb P(\mathbb X) \to \mathbb A \to \mathbb Y \to \mathbb P(\mathbb X)H:P(X)→A→Y→P(X) that satisfies (3.3) and is jointly Borel measurable in (z,a,y)(z,a,y)(z,a,y), as the paper's stochastic kernel is; qqq is the image of R′R'R′ under H(z,a,⋅)H(z, a, \cdot)H(z,a,⋅) for the Borel σ-algebra of the weak topology on P(X)\mathbb P(\mathbb X)P(X).

Since qqq does not depend on the choice of filter, every statement about qqq is made for every filter satisfying (3.3). A fixed filter that ignores the observation, such as H(z,a,y)=zH(z, a, y) = zH(z,a,y)=z, would make qqq a point mass and the weak continuity trivial; it satisfies (3.3) only when observations carry no information, and is excluded by quantifying over all filters.

The goal states the Assumption (W*) conclusion of Theorem 3.6. Its tail, "statements (i)–(vi) of Theorem 3.1 hold", invokes Theorem 2.1 of Feinberg et al. (2012) applied to the belief MDP; it is cited rather than proved in this paper. Generalized Fatou's lemmas 5.1(i) and 5.1(ii) are referenced from the published Schäl (1993) items. A complete development needs weak and setwise convergence on ProbabilityMeasure, portmanteau-type arguments, disintegration (Measure.condKernel) and the identification of the Borel and Giry σ-algebras on P(X)\mathbb P(\mathbb X)P(X); the last is missing from Mathlib and is a welcome contribution in its own right. Proofs of any milestone, and alternative arguments for Theorem 3.7, are welcome.

Selected references

  • E. A. Feinberg, P. O. Kasyanov, M. Z. Zgurovsky, Partially Observable Total-Cost Markov Decision Processes with Weakly Continuous Transition Probabilities, arXiv:1401.2168v2 (2014); Mathematics of Operations Research 41(2), 2016. https://arxiv.org/abs/1401.2168v2
  • E. A. Feinberg, P. O. Kasyanov, N. V. Zadoianchuk, Average-cost Markov decision processes with weakly continuous transition probabilities, Mathematics of Operations Research 37(4), 2012. https://doi.org/10.1287/moor.1120.0555
  • D. P. Bertsekas, S. E. Shreve, Stochastic Optimal Control: The Discrete-Time Case, Academic Press, 1978. https://web.mit.edu/dimitrib/www/soc.html
  • M. Schäl, Average optimality in dynamic programming with general state space, Mathematics of Operations Research 18(1), 1993. https://doi.org/10.1287/moor.18.1.163
14 thms1 active userReviewed
Machine LearningProbabilityStatistics·Captain: mikedeng1

Nonasymptotic Convergence Analysis for the Unadjusted Langevin Algorithm: Explicit Total-Variation Bound for ULA with Nonincreasing Step Sizes When the Potential Is Strongly Convex Outside a BallResearch Paper

Why sampling from e−Ue^{-U}e−U needs non-asymptotic guarantees

Bayesian inference, statistical physics and machine learning all ask for samples from a probability distribution π\piπ on Rd\mathbb R^dRd whose density is known only up to a constant, π(dx)∝e−U(x)dx\pi(dx)\propto e^{-U(x)}dxπ(dx)∝e−U(x)dx. The unadjusted Langevin algorithm (ULA) is the Euler–Maruyama discretisation of the Langevin diffusion, whose invariant law is π\piπ. It needs only gradients of UUU, has no accept/reject step, and is therefore cheap in high dimension. Without the Metropolis correction, however, the chain does not leave π\piπ invariant, so its usefulness rests on quantitative bounds: how far, in total variation, the law of the ppp-th iterate is from π\piπ, as an explicit function of the dimension ddd, the step sizes and the regularity of UUU.

Durmus and Moulines (arXiv:1507.05021, Ann. Appl. Probab. 27(3), 2017) gave such bounds for constant and decreasing step sizes. Earlier work of Dalalyan (arXiv:1412.7392, 2017) treated strongly log-concave targets with constant steps. Durmus and Moulines cover nonincreasing step sequences and, among other regimes, potentials that are only strongly convex outside a ball; this mission formalizes that case.

Setting

Write ∥⋅∥\|\cdot\|∥⋅∥ and ⟨⋅,⋅⟩\langle\cdot,\cdot\rangle⟨⋅,⋅⟩ for the Euclidean norm and inner product of Rd\mathbb R^dRd. The total variation norm of a difference of probability measures is

∥μ−ν∥TV=sup⁡{∣∫f dμ−∫f dν∣:f Borel, ∣f∣≤1},\|\mu-\nu\|_{\mathrm{TV}}=\sup\Big\{\Big|\int f\,d\mu-\int f\,d\nu\Big|: f\text{ Borel},\ |f|\le1\Big\},∥μ−ν∥TV​=sup{​∫fdμ−∫fdν​:f Borel, ∣f∣≤1},

which ranges in [0,2][0,2][0,2].

The potential U:Rd→RU:\mathbb R^d\to\mathbb RU:Rd→R satisfies two assumptions:

  • L1: UUU is differentiable and ∇U\nabla U∇U is LLL-Lipschitz;
  • H3(MsM_sMs​): UUU is convex and ⟨∇U(x)−∇U(y),x−y⟩≥m∥x−y∥2\langle\nabla U(x)-\nabla U(y),x-y\rangle\ge m\|x-y\|^2⟨∇U(x)−∇U(y),x−y⟩≥m∥x−y∥2 whenever ∥x−y∥≥Ms\|x-y\|\ge M_s∥x−y∥≥Ms​, for some Ms≥0M_s\ge0Ms​≥0, m>0m>0m>0.

The point x⋆x^\starx⋆ is a minimiser of UUU, and π(dx)=e−U(x)dx/∫e−U\pi(dx)=e^{-U(x)}dx/\int e^{-U}π(dx)=e−U(x)dx/∫e−U.

The Langevin diffusion is dYt=−∇U(Yt)dt+2 dBtddY_t=-\nabla U(Y_t)dt+\sqrt2\,dB^d_tdYt​=−∇U(Yt​)dt+2​dBtd​, with BdB^dBd a standard ddd-dimensional Brownian motion. Its transition semigroup is Pt(x,A)=P(Yt(x)∈A)P_t(x,A)=\mathbb P(Y_t(x)\in A)Pt​(x,A)=P(Yt​(x)∈A).

ULA with step sizes (γk)k≥1(\gamma_k)_{k\ge1}(γk​)k≥1​ is the chain

Xk+1=Xk−γk+1∇U(Xk)+2γk+1Zk+1,X_{k+1}=X_k-\gamma_{k+1}\nabla U(X_k)+\sqrt{2\gamma_{k+1}}Z_{k+1},Xk+1​=Xk​−γk+1​∇U(Xk​)+2γk+1​​Zk+1​,

with (Zk)(Z_k)(Zk​) i.i.d. standard Gaussian. Its one-step kernel is Rγ(x,⋅)=N(x−γ∇U(x),2γId)R_\gamma(x,\cdot)=\mathcal N(x-\gamma\nabla U(x),2\gamma I_d)Rγ​(x,⋅)=N(x−γ∇U(x),2γId​), and δxQγn=δxRγ1⋯Rγn\delta_xQ^n_\gamma=\delta_xR_{\gamma_1}\cdots R_{\gamma_n}δx​Qγn​=δx​Rγ1​​⋯Rγn​​ is the law of XnX_nXn​ started at xxx. Write Γn,p=∑k=npγk\Gamma_{n,p}=\sum_{k=n}^p\gamma_kΓn,p​=∑k=np​γk​, and let A(γ,x)=sup⁡k≥0∫∥∇U∥2 d(δxQγk)A(\gamma,x)=\sup_{k\ge0}\int\|\nabla U\|^2\,d(\delta_xQ^k_\gamma)A(γ,x)=supk≥0​∫∥∇U∥2d(δx​Qγk​).

The explicit constants use

  • F(λ,a,c,γ,w)=λaw+c(−λγlog⁡λ)−1F(\lambda,a,c,\gamma,w)=\lambda^aw+c(-\lambda^\gamma\log\lambda)^{-1}F(λ,a,c,γ,w)=λaw+c(−λγlogλ)−1 and G(λ,c,γ,w)=w+c(−λγlog⁡λ)−1G(\lambda,c,\gamma,w)=w+c(-\lambda^\gamma\log\lambda)^{-1}G(λ,c,γ,w)=w+c(−λγlogλ)−1;
  • ω(ε,R)=R2/{2Φ−1(1−ε/2)}2\omega(\varepsilon,R)=R^2/\{2\Phi^{-1}(1-\varepsilon/2)\}^2ω(ε,R)=R2/{2Φ−1(1−ε/2)}2, with Φ\PhiΦ the standard Gaussian distribution function;
  • λ=e−2m+γˉL2\lambda=e^{-2m+\bar\gamma L^2}λ=e−2m+γˉ​L2 and c=2(d+mMs2)c=2(d+mM_s^2)c=2(d+mMs2​).

Formalization targets

Goal: Theorem 21 (p. 16)

Let d≥1d\ge1d≥1, and let (γk)(\gamma_k)(γk​) be positive and nonincreasing with γ1≤γˉ<2mL−2\gamma_1\le\bar\gamma<2mL^{-2}γ1​≤γˉ​<2mL−2. Then for all 0≤n<p0\le n<p0≤n<p and xxx,

∥δxQγp−π∥TV≤2−1/2L(∑k=np−1{γk+133A(γ,x)+dγk+12})1/2+C(δxQγn)κΓn+1,p.\|\delta_xQ^p_\gamma-\pi\|_{\mathrm{TV}}\le2^{-1/2}L\Big(\sum_{k=n}^{p-1}\Big\{\tfrac{\gamma^3_{k+1}}3A(\gamma,x)+d\gamma^2_{k+1}\Big\}\Big)^{1/2}+C(\delta_xQ^n_\gamma)\kappa^{\Gamma_{n+1,p}}.∥δx​Qγp​−π∥TV​≤2−1/2L(k=n∑p−1​{3γk+13​​A(γ,x)+dγk+12​})1/2+C(δx​Qγn​)κΓn+1,p​.

The three constants are explicit:

  • the rate, log⁡κ=−m2log⁡2 [log⁡{(1+emω(1/2,max⁡(1,Ms))/4)(1+max⁡(1,Ms))}+log⁡2]−1\log\kappa=-\frac m2\log2\,[\log\{(1+e^{m\omega(1/2,\max(1,M_s))/4})(1+\max(1,M_s))\}+\log2]^{-1}logκ=−2m​log2[log{(1+emω(1/2,max(1,Ms​))/4)(1+max(1,Ms​))}+log2]−1;
  • the constant of the first term, A(γ,x)≤L2G(λ,c,γ1,∥x−x⋆∥2)A(\gamma,x)\le L^2G(\lambda,c,\gamma_1,\|x-x^\star\|^2)A(γ,x)≤L2G(λ,c,γ1​,∥x−x⋆∥2);
  • the constant of the second term, C(δxQγn)≤6+2(d/m+Ms2)1/2+2F1/2(λ,Γ1,n,c,γ1,∥x−x⋆∥2)C(\delta_xQ^n_\gamma)\le6+2(d/m+M_s^2)^{1/2}+2F^{1/2}(\lambda,\Gamma_{1,n},c,\gamma_1,\|x-x^\star\|^2)C(δx​Qγn​)≤6+2(d/m+Ms2​)1/2+2F1/2(λ,Γ1,n​,c,γ1​,∥x−x⋆∥2).

Milestones, in the order of the proof (§5.1, p. 34)

  1. (56), p. 30: for dXt=b(Xt)dt+dBtddX_t=b(X_t)dt+dB^d_tdXt​=b(Xt​)dt+dBtd​ with bbb Lipschitz and monotone (G1), sup⁡∥x−y∥≤R∥Pt(x,⋅)−Pt(y,⋅)∥TV≤2(1−ε)\sup_{\|x-y\|\le R}\|P_t(x,\cdot)-P_t(y,\cdot)\|_{\mathrm{TV}}\le2(1-\varepsilon)sup∥x−y∥≤R​∥Pt​(x,⋅)−Pt​(y,⋅)∥TV​≤2(1−ε) for t≥ω(ε,R)t\ge\omega(\varepsilon,R)t≥ω(ε,R).
  2. Theorem 36, p. 33: under G1 and the contraction G3, ∥Pt(x,⋅)−Pt(y,⋅)∥TV≤2{(1−ε)−1+1+∥x−y∥}κt\|P_t(x,\cdot)-P_t(y,\cdot)\|_{\mathrm{TV}}\le2\{(1-\varepsilon)^{-1}+1+\|x-y\|\}\kappa^t∥Pt​(x,⋅)−Pt​(y,⋅)∥TV​≤2{(1−ε)−1+1+∥x−y∥}κt with an explicit κ\kappaκ.
  3. §5.1, p. 34: ∥Pt(x,⋅)−π∥TV≤2{3+∥x−x⋆∥+∫∥y−x⋆∥dπ(y)}κt\|P_t(x,\cdot)-\pi\|_{\mathrm{TV}}\le2\{3+\|x-x^\star\|+\int\|y-x^\star\|d\pi(y)\}\kappa^t∥Pt​(x,⋅)−π∥TV​≤2{3+∥x−x⋆∥+∫∥y−x⋆∥dπ(y)}κt for the Langevin semigroup.
  4. §5.1, p. 34: ∫∥y−x⋆∥2dπ(y)≤d/m+Ms2\int\|y-x^\star\|^2d\pi(y)\le d/m+M_s^2∫∥y−x⋆∥2dπ(y)≤d/m+Ms2​.
  5. Proposition 20, p. 16: V(x)=∥x−x⋆∥2V(x)=\|x-x^\star\|^2V(x)=∥x−x⋆∥2 satisfies RγV≤λγV+γcR_\gamma V\le\lambda^\gamma V+\gamma cRγ​V≤λγV+γc for γ∈(0,γˉ]\gamma\in(0,\bar\gamma]γ∈(0,γˉ​].
  6. Lemma 1, p. 5: such a drift gives QγnV(x)≤F(λ,Γn,c,γ1,V(x))Q^n_\gamma V(x)\le F(\lambda,\Gamma_n,c,\gamma_1,V(x))Qγn​V(x)≤F(λ,Γn​,c,γ1​,V(x)).
  7. Proposition 2, p. 5: the decomposition bound (10), given the geometric ergodicity (3) of the diffusion.

The paper's proofs follow each statement in the body of the paper (§4.1, §4.12, §5, §5.1).

Significance

The theorem bounds the sampling error of ULA in total variation non-asymptotically, with a dimension-free contraction rate κ\kappaκ and constants polynomial in ddd. With steps decreasing to zero and cumulative step size diverging, its bound gives convergence of δxQγp\delta_xQ^p_\gammaδx​Qγp​ to π\piπ; for constant steps, the paper reports iteration counts of order dlog⁡dd\log ddlogd at a prescribed precision in Table 2. H3 covers log-concave targets whose potential can lack strong convexity inside a ball.

The result is proved in the paper but, as far as the platform's catalogue shows, not formalized anywhere. A formalization adds machine-checked versions of several reusable pieces:

  • a Foster–Lyapunov moment bound for time-inhomogeneous Markov chains (Lemma 1);
  • a Girsanov–Pinsker discretisation estimate for Euler schemes (Proposition 2);
  • a reflection-coupling contraction for diffusions with monotone drift (56), Theorem 36.

Difficulty

The ULA chain is not reversible with respect to π\piπ, so the spectral and Dirichlet-form tools of reversible MCMC do not apply to it directly. The bound must instead compare the chain with the continuous diffusion.

That comparison needs Girsanov's theorem on path space and a uniform control of ∫∥∇U∥2\int\|\nabla U\|^2∫∥∇U∥2 along a time-inhomogeneous chain.

The rate of the diffusion itself is the second obstacle. Without global strong convexity, synchronous coupling gives no contraction inside the ball ∥x−y∥<Ms\|x-y\|<M_s∥x−y∥<Ms​. An explicit rate requires a reflection coupling together with return-time estimates, and it has to be transferred through the time change between dY=−∇U dt+2dBdY=-\nabla U\,dt+\sqrt2dBdY=−∇Udt+2​dB and dX=−12∇U dt+dBdX=-\frac12\nabla U\,dt+dBdX=−21​∇Udt+dB.

Formalization scope

The state space is the published EthierKurtz.SDEState d, i.e. EuclideanSpace ℝ (Fin d), and Brownian motion and strong solutions are the published EthierKurtz.IsStandardBrownian and EthierKurtz.SolvesBrownianSDE. The conventions are:

  • Semigroup. A semigroup is pinned down by the laws of solutions. PPP is the semigroup of dX=b(X)dt+s dBdX=b(X)dt+s\,dBdX=b(X)dt+sdB when every strong solution from xxx, on every probability space, has time-ttt law Pt(x,⋅)P_t(x,\cdot)Pt​(x,⋅). The Langevin case is b=−∇Ub=-\nabla Ub=−∇U, s=2s=\sqrt2s=2​.
  • Total variation. It is the sup⁡∣f∣≤1\sup_{|f|\le1}sup∣f∣≤1​ norm. Using sup⁡A∣μ(A)−ν(A)∣\sup_A|\mu(A)-\nu(A)|supA​∣μ(A)−ν(A)∣ instead would halve every bound.
  • ULA. The laws δxQγn\delta_xQ^n_\gammaδx​Qγn​ are iterated Measure.bind with the Gaussian kernel N(x−γ∇U(x),2γId)\mathcal N(x-\gamma\nabla U(x),2\gamma I_d)N(x−γ∇U(x),2γId​). Steps are indexed from 111, and Γn,p=0\Gamma_{n,p}=0Γn,p​=0 for p<np<np<n.
  • Moments. Upper-bounded moments (QγnVQ^n_\gamma VQγn​V, ∫∥∇U∥2\int\|\nabla U\|^2∫∥∇U∥2, ∫∥y−x⋆∥2dπ\int\|y-x^\star\|^2d\pi∫∥y−x⋆∥2dπ) are lower Lebesgue integrals, so a non-integrable function cannot make a bound free. A(γ,x)≤aA(\gamma,x)\le aA(γ,x)≤a is a bound for every kkk, not a real supremum.
  • Normalisation of π\piπ. No hypothesis ∫e−U<∞\int e^{-U}<\infty∫e−U<∞ is added; it follows from L1 and H3.
  • Quantile. Φ−1(u)=inf⁡{z:Φ(z)≥u}\Phi^{-1}(u)=\inf\{z:\Phi(z)\ge u\}Φ−1(u)=inf{z:Φ(z)≥u}.
  • Logarithms and powers. λ\lambdaλ, κ\kappaκ lie in (0,1)(0,1)(0,1), so the logarithms and real powers are genuine.

Several hypotheses are added or read in a particular way, and each is disclosed in the item's statement:

  • x⋆x^\starx⋆. It is a bound variable with the hypothesis that it minimises UUU; the paper uses x⋆x^\starx⋆ in §3.3 without defining it.
  • Proposition 20. The dimension is positive so its printed c=2(d+mMs2)c=2(d+mM_s^2)c=2(d+mMs2​) satisfies (6)'s requirement c>0c>0c>0; this is stated in Lean.
  • Lemma 1. It carries the §2 standing assumption L1, which makes the Euler transition family measurable in its starting point. It is stated for Borel V≥0V\ge0V≥0 rather than V≥1V\ge1V≥1, because Theorem 21 applies it to ∥x−x⋆∥2\|x-x^\star\|^2∥x−x⋆∥2.
  • Proposition 2. It assumes (3) only at μ0=δxQγn\mu_0=\delta_xQ^n_\gammaμ0​=δx​Qγn​, with finite CCC and a finite bound on A(γ,x)A(\gamma,x)A(γ,x).
  • (11). The printed Qγk(y,dz)Q^k_\gamma(y,dz)Qγk​(y,dz) is read as Qγk(x,dz)Q^k_\gamma(x,dz)Qγk​(x,dz).

The citations [20] (existence of solutions), [27] (Lindvall), [32] (Meyn–Tweedie) and Girsanov's theorem are used inside the proofs and are not restated.

The library work needed is substantial: Itô calculus for the reflection coupling, Girsanov's theorem, Pinsker's inequality in this norm, and generator–drift arguments for the moments of π\piπ. Contributions of any of these as standalone platform theorems are welcome.

Selected references

  • A. Durmus, É. Moulines, Nonasymptotic convergence analysis for the unadjusted Langevin algorithm, Ann. Appl. Probab. 27(3), 2017; cited as arXiv:1507.05021v3. https://arxiv.org/abs/1507.05021
  • A. S. Dalalyan, Theoretical guarantees for approximate sampling from smooth and log-concave densities, J. R. Stat. Soc. B 79(3), 2017. https://arxiv.org/abs/1412.7392
  • T. Lindvall, L. C. G. Rogers, Coupling of multidimensional diffusions by reflection, Ann. Probab. 14(3), 1986. https://doi.org/10.1214/aop/1176992436
  • S. P. Meyn, R. L. Tweedie, Stability of Markovian processes III: Foster–Lyapunov criteria for continuous-time processes, Adv. Appl. Probab. 25(3), 1993. https://doi.org/10.2307/1427522
  • G. O. Roberts, R. L. Tweedie, Exponential convergence of Langevin distributions and their discrete approximations, Bernoulli 2(4), 1996. https://doi.org/10.2307/3318418
13 thms1 active userReviewed
Graph TheoryOperations ResearchProbability·Captain: mikedeng1

Routing Betweenness Centrality: Under Source-Oblivious Loop-Free Routing, the Betweenness of a Node Sequence Is a Sum over Targets of Chained Pairwise DependenciesResearch Paper

Motivation

Betweenness centrality measures how much of the communication in a network passes through a node. Freeman's original measure (Freeman 1977) counts the fraction of shortest paths between each pair of nodes that pass through it, and Brandes' algorithm (Brandes 2001) made it computable on large graphs. Variants replace shortest paths by maximum flows (Freeman, Borgatti and White 1991), random walks (Newman 2005) or load-splitting rules (Goh, Kahng and Kim 2001), and group variants measure the traffic that passes through a set of nodes (Everett and Borgatti 1999).

Dolev, Elovici and Puzis (technical report 2009, J. ACM 2010) define routing betweenness centrality (RBC): given the actual routing scheme of a communication network and its traffic matrix, the RBC of a node is the expected number of packets that pass through it. Each of the measures above is a special choice of routing scheme. The application is the placement of traffic monitors: the RBC of an ordered sequence of monitors is the expected number of packets sampled by all of them in order, which measures, for example, redundant inspection along a path.

Setting

Let VVV be a finite set of nodes. A routing scheme assigns to every source sss, current node uuu, next node vvv and target ttt a number R(s,u,v,t)R(s,u,v,t)R(s,u,v,t), the probability that uuu forwards to vvv a packet with source address sss and target address ttt. Routing decisions are independent. The scheme is assumed to be

  • nonnegative;
  • total: every node u≠tu\ne tu=t forwards a packet targeted at ttt with total probability one, ∑vR(s,u,v,t)=1\sum_v R(s,u,v,t)=1∑v​R(s,u,v,t)=1, and the target forwards nothing;
  • loop-free: for every pair (s,t)(s,t)(s,t) the directed graph with an arc a→ba\to ba→b whenever R(s,a,b,t)>0R(s,a,b,t)>0R(s,a,b,t)>0 is acyclic;
  • source-oblivious where stated: R(s,u,v,t)R(s,u,v,t)R(s,u,v,t) does not depend on sss.

A route from sss to ttt is a sequence of nodes from sss to ttt whose every hop has positive forwarding probability; its probability is the product of the hop probabilities. For nodes s,t,vs,t,vs,t,v and a sequence S=(s1,…,sk)S=(s_1,\dots,s_k)S=(s1​,…,sk​):

  • the pairwise dependency δs,t(v)\delta_{s,t}(v)δs,t​(v) is the total probability of the routes from sss to ttt that visit vvv;
  • the sequence dependency δ~s,t(S)\tilde\delta_{s,t}(S)δ~s,t​(S) is the total probability of the routes from sss to ttt that visit s1s_1s1​, then s2s_2s2​, …, then sks_ksk​ (consecutive repetitions in SSS collapsed).

A traffic matrix T(s,t)T(s,t)T(s,t) gives the number of packets sent from sss to ttt, and sampling rates ρv∈[0,1]\rho_v\in[0,1]ρv​∈[0,1] give the fraction of passing packets a monitor at vvv samples. The target dependency is δ∙,t(v)=∑sδs,t(v)T(s,t)\delta_{\bullet,t}(v)=\sum_{s}\delta_{s,t}(v)T(s,t)δ∙,t​(v)=∑s​δs,t​(v)T(s,t), and the RBC of the sequence SρS_\rhoSρ​ of distinct nodes is

δ~∙,∙(Sρ)=∏r∈Sρr⋅∑s,t∈Vδ~s,t(S) T(s,t).\tilde\delta_{\bullet,\bullet}(S_\rho)=\prod_{r\in S}\rho_r\cdot\sum_{s,t\in V}\tilde\delta_{s,t}(S)\,T(s,t).δ~∙,∙​(Sρ​)=r∈S∏​ρr​⋅s,t∈V∑​δ~s,t​(S)T(s,t).

Formalization targets

Goal: sequence RBC as a sum over targets of chained dependencies

For a source-oblivious scheme and a sequence S=(s1,…,sk)S=(s_1,\dots,s_k)S=(s1​,…,sk​) of distinct nodes, k≥1k\ge1k≥1,

∏r∈Sρr⋅∑s,t∈Vδ~s,t(S) T(s,t)=∑t∈Vδ∙,t(s1) ρs1∏i=1k−1δsi,t(si+1) ρsi+1.\prod_{r\in S}\rho_r\cdot\sum_{s,t\in V}\tilde\delta_{s,t}(S)\,T(s,t)=\sum_{t\in V}\delta_{\bullet,t}(s_1)\,\rho_{s_1}\prod_{i=1}^{k-1}\delta_{s_i,t}(s_{i+1})\,\rho_{s_{i+1}}.r∈S∏​ρr​⋅s,t∈V∑​δ~s,t​(S)T(s,t)=t∈V∑​δ∙,t​(s1​)ρs1​​i=1∏k−1​δsi​,t​(si+1​)ρsi+1​​.

The right side is the value returned by the paper's Algorithm 6. The identity contains no running time and no constant: it states the exact equality the algorithm relies on.

Milestones

  1. Proposition 1: for a loop-free scheme (not necessarily source-oblivious), two different orderings of the same set of nodes cannot both have positive sequence dependency.
  2. Lemma 1 (Dependency chaining): for s≠ts\ne ts=t, δ~s,t((s1,…,sk))=δs,t(s1)⋅δ~s1,t((s2,…,sk))\tilde\delta_{s,t}((s_1,\dots,s_k))=\delta_{s,t}(s_1)\cdot\tilde\delta_{s_1,t}((s_2,\dots,s_k))δ~s,t​((s1​,…,sk​))=δs,t​(s1​)⋅δ~s1​,t​((s2​,…,sk​)).
  3. Eq. (16): for s≠ts\ne ts=t, δ~s,t((s1,…,sk))=δs,t(s1)∏i=2kδsi−1,t(si)\tilde\delta_{s,t}((s_1,\dots,s_k))=\delta_{s,t}(s_1)\prod_{i=2}^k\delta_{s_{i-1},t}(s_i)δ~s,t​((s1​,…,sk​))=δs,t​(s1​)∏i=2k​δsi−1​,t​(si​).
  4. Eq. (17): δ~∙,t((s1,…,sk))=δ∙,t(s1)∏i=2kδsi−1,t(si)\tilde\delta_{\bullet,t}((s_1,\dots,s_k))=\delta_{\bullet,t}(s_1)\prod_{i=2}^k\delta_{s_{i-1},t}(s_i)δ~∙,t​((s1​,…,sk​))=δ∙,t​(s1​)∏i=2k​δsi−1​,t​(si​).
  5. Eq. (1) (off the goal's path): δs,t(s)=1\delta_{s,t}(s)=1δs,t​(s)=1 and δs,t(v)=∑uδs,t(u)R(s,u,v,t)\delta_{s,t}(v)=\sum_{u}\delta_{s,t}(u)R(s,u,v,t)δs,t​(v)=∑u​δs,t​(u)R(s,u,v,t) for v≠sv\ne sv=s, the sum over predecessors uuu of vvv.
  6. Eq. (9) (off the goal's path): δ∙,t(v)=T(v,t)+∑uδ∙,t(u)R(⊘,u,v,t)\delta_{\bullet,t}(v)=T(v,t)+\sum_u\delta_{\bullet,t}(u)R(\oslash,u,v,t)δ∙,t​(v)=T(v,t)+∑u​δ∙,t​(u)R(⊘,u,v,t) for source-oblivious schemes.

Significance

The goal reduces the RBC of a sequence of kkk monitors to O(nk)O(nk)O(nk) arithmetic on precomputed single-node quantities, instead of a pass over the routing graph of every source–target pair. This is what makes repeated evaluation of candidate monitor sequences, as in greedy or search-based placement, affordable on large networks. Eqs. (1) and (9) are the recursions behind the paper's algorithms for the RBC of single nodes; Eq. (9) replaces the loop over all source–target pairs by a loop over targets.

The paper's proof of Lemma 1 is an argument about conditional probabilities of informally described events. A formalization fixes what "the probability that a packet passes through SSS" means, as a sum over a finite set of routes, and checks the factorization against that definition, including the corner cases the prose passes over: sequences with repeated nodes, a first node equal to the source or to the target, and the source s=ts=ts=t in the sum over all sources. No machine-checked version of these identities is known.

Difficulty

Dependency chaining is a Markov property: after the packet reaches s1s_1s1​, its future does not depend on its past. The obvious argument splits every route through s1s_1s1​ into a prefix ending at s1s_1s1​ and a suffix starting there. Two points need care. First, the suffix must be a route of a packet with source s1s_1s1​, which is where source-obliviousness enters; for source-dependent routing the lemma is false. Second, the prefix probabilities must sum to δs,t(s1)\delta_{s,t}(s_1)δs,t​(s1​) while the suffix probabilities sum to one, which requires that no packet is lost: this is where totality of the forwarding probabilities and loop-freeness (every walk reaches the target in at most ∣V∣|V|∣V∣ steps) are needed. The passage from Lemma 1 to the goal then requires the sum over all sources, including s=ts=ts=t, where Lemma 1 does not apply.

Formalization scope

Nodes form a type V with [Fintype V] [DecidableEq V]; probabilities, traffic and sampling rates are real numbers; sequences are Lean lists, indexed from 000. Routes are repetition-free lists, and every dependency is a finite sum over them; no infinite sums, suprema or measure theory are used. The definitions file bundles the standing assumptions in IsRoutingScheme and source-obliviousness in IsSourceOblivious.

The paper's probabilities are informal. The following readings are explicit:

  • every non-target node forwards with total probability one (implicit in the paper; without it Lemma 1 fails);
  • the target forwards nothing;
  • the paper's convention R(⊘,v,v,⊘)=1R(\oslash,v,v,\oslash)=1R(⊘,v,v,⊘)=1 is not adopted (it contradicts loop-freeness); collapsing consecutive repetitions in SSS reproduces its only use;
  • the "don't care" source ⊘\oslash⊘ of Eq. (9) is an arbitrary fixed node, harmless under source-obliviousness;
  • the printed index slips in Eqs. (16) and (17) are corrected;
  • no sign condition is imposed on TTT; 0≤ρv≤10\le\rho_v\le10≤ρv​≤1 is kept in the goal as on the page.

Defining δ~\tilde\deltaδ~ by the recursion (3) or the product (16), the target sequence dependency by (17), or the sequence RBC as the sum over targets of the chained product would make the goal true by definition; every dependency here is a sum of route probabilities, and the RBC of a sequence is the double sum of Eq. (4).

The running-time claims of the paper are out of scope. Lemma 2 of the paper (p. 17) is false as printed for a nonempty monitor set and is not posed. Contributions welcome: proofs of the milestones, a general lemma that the total route probability from any node is one under a total loop-free scheme (reusable for absorbing Markov chains on finite DAGs), and the prefix–suffix decomposition of routes.

Selected references

  • S. Dolev, Y. Elovici, R. Puzis, Routing Betweenness Centrality, Technical Report #2009-09, Ben-Gurion University of the Negev, 2009; J. ACM 57(4), 2010. https://doi.org/10.1145/1734213.1734219
  • L. C. Freeman, A set of measures of centrality based on betweenness, Sociometry 40(1), 1977. https://doi.org/10.2307/3033543
  • U. Brandes, A faster algorithm for betweenness centrality, J. Math. Sociology 25(2), 2001. https://doi.org/10.1080/0022250X.2001.9990249
  • L. C. Freeman, S. P. Borgatti, D. R. White, Centrality in valued graphs: a measure of betweenness based on network flow, Social Networks 13, 1991. https://doi.org/10.1016/0378-8733(91)90017-N
  • M. E. J. Newman, A measure of betweenness centrality based on random walks, Social Networks 27, 2005. https://doi.org/10.1016/j.socnet.2004.11.009
  • K.-I. Goh, B. Kahng, D. Kim, Universal behavior of load distribution in scale-free networks, Phys. Rev. Lett. 87, 2001. https://doi.org/10.1103/PhysRevLett.87.278701
  • M. G. Everett, S. P. Borgatti, The centrality of groups and classes, J. Math. Sociology 23(3), 1999. https://doi.org/10.1080/0022250X.1999.9990219
8 thms1 active userReviewed
Dynamic ProgrammingOperations ResearchProbability+1·Captain: mikedeng1

More Risk-Sensitive Markov Decision Processes 5: Under Positive Harris Recurrence a Risk-Neutral Optimal Stationary Policy Is Optimal for Power-Utility Average CostResearch Paper

Motivation

Classical Markov decision theory minimizes expected cost. A decision maker who dislikes variability, such as an insurer, a portfolio manager or an operator of a service system, may instead evaluate a random cost YYY by its certainty equivalent U−1(E[U(Y)])U^{-1}(\mathbb E[U(Y)])U−1(E[U(Y)]) for an increasing utility (disutility) function UUU. Bäuerle and Rieder (More Risk-Sensitive Markov Decision Processes, Math. Oper. Res. 39(1), 2014; authors' manuscript at KIT 1000039663) develop dynamic programming for this criterion with general UUU. Classical risk-sensitive MDPs, studied since Howard and Matheson (1972), use the exponential utility, and for it the average cost problem differs considerably from the risk-neutral one (Cavazos-Cadena and Fernández-Gaucherand 2000; Di Masi and Stettner 1999). Their §5 asks what happens in the long run. The average cost per stage is evaluated through a power utility U(y)=yγU(y)=y^\gammaU(y)=yγ. Does risk sensitivity change which stationary policy is optimal?

This mission formalizes their answer, Theorem 5.2, and the results its proof uses (manuscript p. 17).

Setting

A controlled Markov process in discrete time has a standard Borel state space EEE and action space AAA. A measurable set D⊆E×AD\subseteq E\times AD⊆E×A lists the admissible state–action pairs, and every D(x)={a:(x,a)∈D}D(x)=\{a:(x,a)\in D\}D(x)={a:(x,a)∈D} is nonempty. A transition law Q(⋅∣x,a)Q(\cdot\mid x,a)Q(⋅∣x,a) gives the distribution of the next state, and a measurable cost ccc satisfies 0<c‾≤c≤cˉ0<\underline c\le c\le\bar c0<c​≤c≤cˉ on DDD.

A history-dependent policy σ=(gn)n≥0\sigma=(g_n)_{n\ge0}σ=(gn​)n≥0​ chooses the action An=gn(X0,A0,…,Xn)∈D(Xn)A_n=g_n(X_0,A_0,\dots,X_n)\in D(X_n)An​=gn​(X0​,A0​,…,Xn​)∈D(Xn​) measurably from the past. The set of all such policies is Π\PiΠ. Each σ\sigmaσ and initial state xxx determine a probability measure Pxσ\mathbb P^\sigma_xPxσ​ on trajectories, with X0=xX_0=xX0​=x and Xn+1∼Q(⋅∣Xn,An)X_{n+1}\sim Q(\cdot\mid X_n,A_n)Xn+1​∼Q(⋅∣Xn​,An​). The accumulated cost is Cn=∑k=0n−1c(Xk,Ak)C^n=\sum_{k=0}^{n-1}c(X_k,A_k)Cn=∑k=0n−1​c(Xk​,Ak​).

A stationary policy π=(f,f,… )\pi=(f,f,\dots)π=(f,f,…) uses a measurable decision rule fff with f(x)∈D(x)f(x)\in D(x)f(x)∈D(x) at every stage. Under it, (Xn)(X_n)(Xn​) is the Markov chain with kernel Pf(x,⋅)=Q(⋅∣x,f(x))P_f(x,\cdot)=Q(\cdot\mid x,f(x))Pf​(x,⋅)=Q(⋅∣x,f(x)).

Fix γ>0\gamma>0γ>0 and U(y)=yγU(y)=y^\gammaU(y)=yγ. The risk-sensitive average cost (5.1) and its optimal value are

Jσ(x)=lim sup⁡n→∞1n U−1(Exσ[U(Cn)]),J(x)=inf⁡σ∈ΠJσ(x).J_\sigma(x)=\limsup_{n\to\infty}\frac1n\,U^{-1}\Big(\mathbb E^\sigma_x\big[U(C^n)\big]\Big),\qquad J(x)=\inf_{\sigma\in\Pi}J_\sigma(x).Jσ​(x)=n→∞limsup​n1​U−1(Exσ​[U(Cn)]),J(x)=σ∈Πinf​Jσ​(x).

The risk-neutral average cost is ρσ(x)=lim sup⁡n1nExσ[Cn]\rho_\sigma(x)=\limsup_n\frac1n\mathbb E^\sigma_x[C^n]ρσ​(x)=limsupn​n1​Exσ​[Cn], the case γ=1\gamma=1γ=1.

A Markov chain is Harris recurrent if, for some nonzero σ\sigmaσ-finite measure φ\varphiφ, every set BBB with φ(B)>0\varphi(B)>0φ(B)>0 is visited infinitely often almost surely from every starting point. It is positive Harris recurrent if it also has an invariant probability measure (Meyn and Tweedie 2009, §§9–10). The MDP is called positive Harris recurrent if the state chain of every stationary policy is.

Formalization targets

Goal: Theorem 5.2

Let γ≥1\gamma\ge1γ≥1 and let the MDP be positive Harris recurrent. If π∗=(f∗,f∗,… )\pi^*=(f^*,f^*,\dots)π∗=(f∗,f∗,…) satisfies ρπ∗(x)≤ρσ(x)\rho_{\pi^*}(x)\le\rho_\sigma(x)ρπ∗​(x)≤ρσ​(x) for all σ∈Π\sigma\in\Piσ∈Π and x∈Ex\in Ex∈E, then

Jπ∗(x)≤Jσ(x)for all σ∈Π, x∈E,soJπ∗=J.J_{\pi^*}(x)\le J_\sigma(x)\quad\text{for all }\sigma\in\Pi,\ x\in E,\qquad\text{so}\qquad J_{\pi^*}=J .Jπ∗​(x)≤Jσ​(x)for all σ∈Π, x∈E,soJπ∗​=J.

The optimal policy does not depend on γ\gammaγ.

Milestones

  1. §5.1, p. 17. For γ>0\gamma>0γ>0, homogeneity gives Jσ(x)=lim sup⁡nU−1(Exσ[U(Cn/n)])J_\sigma(x)=\limsup_n U^{-1}(\mathbb E^\sigma_x[U(C^n/n)])Jσ​(x)=limsupn​U−1(Exσ​[U(Cn/n)]).
  2. Theorem 5.1. If the state chain of a stationary policy π\piπ is positive Harris recurrent, there is a number ρ\rhoρ with
lim⁡n→∞1n(Exπ[(Cn)γ])1/γ=ρ=lim⁡n→∞1nExπ[Cn]for all γ>0, x∈E.\lim_{n\to\infty}\frac1n\Big(\mathbb E^\pi_x\big[(C^n)^\gamma\big]\Big)^{1/\gamma}=\rho=\lim_{n\to\infty}\frac1n\mathbb E^\pi_x[C^n]\qquad\text{for all }\gamma>0,\ x\in E.n→∞lim​n1​(Exπ​[(Cn)γ])1/γ=ρ=n→∞lim​n1​Exπ​[Cn]for all γ>0, x∈E.
  1. Proof of Theorem 5.2. For γ≥1\gamma\ge1γ≥1 and every σ∈Π\sigma\in\Piσ∈Π: ρσ(x)≤Jσ(x)\rho_\sigma(x)\le J_\sigma(x)ρσ​(x)≤Jσ​(x).

Significance

The result. Theorem 5.2 reduces a risk-sensitive long-run problem to a classical one. Under positive Harris recurrence, any stationary policy that is optimal for the expected average cost, for instance one obtained from the average-cost optimality equation, is also optimal for every convex power criterion γ≥1\gamma\ge1γ≥1, simultaneously. Theorem 5.1 explains why: along a positive Harris recurrent chain the empirical average cost Cn/nC^n/nCn/n converges almost surely to a constant, so in the limit the power utility sees no randomness at all. The contrast is with exponential utility, where the risk-sensitive average cost generally differs from the risk-neutral one and leads to a multiplicative Poisson equation. Positive homogeneity of the utility is what removes the risk effect.

Formalizing it. The results are proved in the paper. As far as is known they have no machine-checked proof, and the platform has no strong law of large numbers for Harris chains on general state spaces. The mission produces a Borel MDP with history-dependent policies and Ionescu-Tulcea path measures, a reusable definition of (positive) Harris recurrence for an arbitrary Markov kernel, and a formal comparison of risk-neutral and risk-sensitive average costs. Each milestone is a self-contained target.

Difficulty

Milestones 1 and 3 are elementary once the expectations are handled correctly: Cn≥0C^n\ge0Cn≥0 holds only almost surely, and Jensen's inequality has to be applied with real powers. The central difficulty is Theorem 5.1. It needs the ergodic theorem for positive Harris recurrent chains, Cn/n→∫c(x,f(x)) μf(dx)C^n/n\to\int c(x,f(x))\,\mu_f(dx)Cn/n→∫c(x,f(x))μf​(dx) almost surely from every initial state (Meyn and Tweedie, Theorem 17.0.1). It also needs an identification of the state process under Pxπ\mathbb P^\pi_xPxπ​ with the canonical chain of PfP_fPf​. The natural first idea, to quote a convergence theorem in total variation, fails: positive Harris recurrence allows periodic chains, for which Pfn(x,⋅)P_f^n(x,\cdot)Pfn​(x,⋅) need not converge. Pathwise averages converge, marginal laws need not, and the argument has to work with the former.

Formalization scope

  • EEE and AAA are standard Borel spaces. No topology is used, and the continuity–compactness conditions (CC) of §2 play no role in §5. D(x)≠∅D(x)\neq\emptysetD(x)=∅ for every xxx is a standing hypothesis.
  • Policies are deterministic and history-dependent, gn:(E×A)n×E→Ag_n:(E\times A)^n\times E\to Agn​:(E×A)n×E→A with gn(hn)∈D(xn)g_n(h_n)\in D(x_n)gn​(hn​)∈D(xn​). Pxσ\mathbb P^\sigma_xPxσ​ is Mathlib's Kernel.trajMeasure on (E×A)N0(E\times A)^{\mathbb N_0}(E×A)N0​. Stationary policies act through gn(hn)=f(xn)g_n(h_n)=f(x_n)gn​(hn​)=f(xn​).
  • (5.1) is defined only for U(y)=yγU(y)=y^\gammaU(y)=yγ, with Real.rpow. For n≥1n\ge1n≥1 all terms of the sequences lie in [c‾,cˉ][\underline c,\bar c][c​,cˉ], so real limits superior are meaningful. The term n=0n=0n=0 (Lean's 1/0=01/0=01/0=0) is irrelevant.
  • The risk-neutral average cost, which the paper leaves undefined, is the lim sup⁡\limsuplimsup of 1nExσ[Cn]\frac1n\mathbb E^\sigma_x[C^n]n1​Exσ​[Cn]. Risk-neutral optimality of π∗\pi^*π∗ and the conclusion of Theorem 5.2 both range over all history-dependent policies. Restricting either to stationary policies would change the theorem.
  • Harris recurrence is defined from scratch on the canonical chain of a Markov kernel and kept equivalent to Meyn–Tweedie's notion. Weakening it to "has an invariant probability", or strengthening it to total-variation ergodicity (which excludes periodic chains), would make a different theorem. Neither is acceptable.
  • Positive Harris recurrence is assumed for every stationary policy, exactly as in the paper. Theorem 5.4 (vanishing discount) and Corollary 5.3 (finite unichain models) are outside the mission.

Contributions are welcome at every level. Reusable pieces, such as the identification of the state process with the chain of PfP_fPf​, the strong law for positive Harris chains, or bounded-convergence lemmas for path measures, are valuable beyond this mission.

Selected references

  • N. Bäuerle and U. Rieder, More Risk-Sensitive Markov Decision Processes, Mathematics of Operations Research 39(1):105–120, 2014. https://doi.org/10.1287/moor.2013.0601 (authors' manuscript: https://publikationen.bibliothek.kit.edu/1000039663)
  • S. P. Meyn and R. L. Tweedie, Markov Chains and Stochastic Stability, 2nd ed., Cambridge University Press, 2009. https://doi.org/10.1017/CBO9780511626630
  • N. Bäuerle and U. Rieder, Markov Decision Processes with Applications to Finance, Springer, 2011. https://doi.org/10.1007/978-3-642-18324-9
  • R. A. Howard and J. E. Matheson, Risk-sensitive Markov decision processes, Management Science 18(7):356–369, 1972. https://doi.org/10.1287/mnsc.18.7.356
7 thms1 active userReviewed
Dynamic ProgrammingOperations ResearchProbability·Captain: mikedeng1

More Risk-Sensitive Markov Decision Processes 2: The Finite-Horizon Discounted Problem Is Solved by Value Iteration on the State Extended by Accumulated Cost and DiscountResearch Paper

Motivation

Sequential decisions often have costs whose timing matters. A controller may choose an action now, observe a random next state, and choose again. Ordinary expected-cost minimization averages the sum of the costs. A risk-sensitive criterion first applies an increasing function UUU to the total cost and then takes its expectation. The curvature of UUU changes how uncertain costs affect the ranking of policies: convex utility penalizes variability in a cost-minimization problem, while concave utility has the opposite tendency. Bäuerle and Rieder study this criterion for controlled Markov processes with general continuous increasing utility, rather than fixing an exponential function. Their finite-horizon discounted result is Theorem 3.6, pp. 10–12 of the authors' manuscript.

Discounting gives early and late costs different weights. For linear UUU, the remaining expected cost can be described using only the current physical state. For general UUU, the effect of a future cost also depends on what has already been paid and on the discount weight currently attached to the next cost. The mission targets the paper's finite-horizon treatment of these two additional quantities. The infinite-horizon problem is a separate part of the paper.

Setting

The state space EEE and action space AAA are Borel spaces. At state xxx, the nonempty set D(x)D(x)D(x) contains the actions that may be chosen. If action a∈D(x)a\in D(x)a∈D(x) is selected, the next state has distribution Q(⋅∣x,a)Q(\cdot\mid x,a)Q(⋅∣x,a) and the stage cost is c(x,a)c(x,a)c(x,a). The cost is measurable and bounded between positive constants c‾\underline cc​ and c‾\overline cc. The discount factor satisfies 0<β<10<\beta<10<β<1, and U:[0,∞)→RU:[0,\infty)\to\mathbb RU:[0,∞)→R is continuous and strictly increasing. The paper imposes its compactness and continuity conditions (CC) on UUU, DDD, ccc, and QQQ; these are stated in the model item and correspond to §2, pp. 3–4.

A history policy σ=(g0,g1,…)\sigma=(g_0,g_1,\ldots)σ=(g0​,g1​,…) chooses AnA_nAn​ measurably from the states and actions observed through time nnn. Starting from xxx, it induces a law for the state-action trajectory. The first nnn stages incur the discounted cost Cβn=∑k=0n−1βkc(Xk,Ak)C^n_\beta=\sum_{k=0}^{n-1}\beta^k c(X_k,A_k)Cβn​=∑k=0n−1​βkc(Xk​,Ak​), with Cβ0=0C^0_\beta=0Cβ0​=0. The paper minimizes Exσ[U(CβN)]E_x^\sigma[U(C^N_\beta)]Exσ​[U(CβN​)] over every admissible history policy. The inverse utility appearing in the paper's original certainty-equivalent criterion can be omitted when selecting a minimizer because UUU is strictly increasing §2, p. 3; §3.3, p. 10.

The extended state is E^=E×[0,∞)×(0,1]\hat E=E\times[0,\infty)\times(0,1]E^=E×[0,∞)×(0,1]. Its coordinates (x,y,z)(x,y,z)(x,y,z) record the current physical state, accumulated cost, and current discount weight. Define Vnσ(x,y,z)=Exσ[U(y+zCβn)]V_{n\sigma}(x,y,z)=E_x^\sigma[U(y+zC^n_\beta)]Vnσ​(x,y,z)=Exσ​[U(y+zCβn​)] and Vn=inf⁡σ∈ΠVnσV_n=\inf_{\sigma\in\Pi}V_{n\sigma}Vn​=infσ∈Π​Vnσ​. A measurable extended-state decision rule fff chooses an action in D(x)D(x)D(x). The paper's operator TfT_fTf​ integrates a continuation value at (x′,y+zc(x,f(x,y,z)),zβ)(x',y+zc(x,f(x,y,z)),z\beta)(x′,y+zc(x,f(x,y,z)),zβ); TTT takes the infimum of the same integral over D(x)D(x)D(x) equations (3.8)–(3.9), pp. 10–11.

Formalization targets

The first target identifies the cost iteration of any sequence of extended-state rules π=(f0,f1,…)\pi=(f_0,f_1,\ldots)π=(f0​,f1​,…):

Vnπ=Tf0⋯Tfn−1U,1≤n≤N.V_{n\pi}=T_{f_0}\cdots T_{f_{n-1}}U,\qquad 1\le n\le N.Vnπ​=Tf0​​⋯Tfn−1​​U,1≤n≤N.

The second target is the paper's value iteration for the infimum over all history policies, together with membership of every VnV_nVn​ in its regularity class C(E^)C(\hat E)C(E^):

V0(x,y,z)=U(y),Vn=TVn−1,Vn∈C(E^).V_0(x,y,z)=U(y),\qquad V_n=TV_{n-1},\qquad V_n\in C(\hat E).V0​(x,y,z)=U(y),Vn​=TVn−1​,Vn​∈C(E^).

The goal is Theorem 3.6(c): minimizers fk∗f_k^*fk∗​ of Vk−1V_{k-1}Vk−1​ exist, and their stage-dependent history rules attain the original objective JN(x)=VN(x,0,1)J_N(x)=V_N(x,0,1)JN​(x)=VN​(x,0,1) for every xxx. At stage n<Nn<Nn<N, the rule uses fN−n∗f_{N-n}^*fN−n∗​ at the current state, the observed sum ∑j<nβjc(Xj,Aj)\sum_{j<n}\beta^jc(X_j,A_j)∑j<n​βjc(Xj​,Aj​), and βn\beta^nβn. “Optimal” compares against all admissible history policies, including those that use more of the history than these three quantities.

Significance

Theorem 3.6 gives a finite sequence of minimum-operator evaluations for a problem whose utility of total cost is not additively separable in the physical state alone. Its policy statement also identifies which observable quantities an optimal controller needs to retain. Together, these results relate the original history-dependent problem to an extended-state Markov decision problem §3.3, pp. 10–12.

The paper proves these statements. This mission asks for machine-checked definitions and proofs of the finite-horizon discounted theorem, including the comparison with all admissible history policies. The model layer and the finite-dimensional expectation construction can support later formalizations of other risk-sensitive objectives. The mission itself has no completed proof at the drafting stage.

Difficulty

The extra discount coordinate is essential: after one action, the next cost enters utility with weight zβz\betazβ, not zzz. Keeping only accumulated cost would describe a different process. The other demanding point is the policy comparison. An iteration over extended-state decision rules has to establish the value of an infimum over arbitrary measurable history policies. Regularity and measurable selection must also be maintained at each step under (CC). The proof of Theorem 3.6 invokes the corresponding total-cost argument from Theorem 3.1, pp. 5–6; the formal development must supply the precise discounted version.

Formalization scope

Lean represents EEE and AAA as Borel subsets of Polish spaces, with standard Borel measurable structures. The controlled process is a separate reusable definition containing DDD, QQQ, and history policies; the risk-sensitive model adds ccc, β\betaβ, and UUU. The admissible graph is Borel and every section D(x)D(x)D(x) is nonempty. A history is a chronological finite list of earlier state-action pairs and a current state. Each rule is defined on all lists, but only admissible lists of the matching length are visited. The transition kernel is total as a Lean object, and its values outside DDD are irrelevant.

Policy values use nested kernel integrals representing the finite-dimensional marginals of the path law. They are defined from the cost and utility, so the Bellman recursion remains a theorem. The optimized value is an infimum over the subtype of measurable, admissible infinite history policies. The real infimum is meaningful here because this policy class is nonempty under (CC) and finite-horizon costs keep utility bounded on the relevant interval. The extended state uses real coordinates restricted to y≥0y\ge0y≥0 and 0<z≤10<z\le10<z≤1; its transition stays in that domain. “Increasing” in C(E^)C(\hat E)C(E^) means componentwise nondecreasing in (y,z)(y,z)(y,z). A minimizer is an admissible measurable rule satisfying Tfv=TvT_fv=TvTf​v=Tv on the entire extended domain.

The source's proof sentence that TTT preserves C(E^)C(\hat E)C(E^) is recorded with explicit integrability of each one-step continuation. For an arbitrary real-valued member of C(E^)C(\hat E)C(E^) on an unbounded state space, the source's conditions alone do not ensure a finite real expectation. This domain condition prevents Lean's zero value for a nonintegrable real integral from turning the assertion into a different one. It does not narrow the finite-horizon values appearing in the goal. Contributions toward finite-dimensional expectation identities, kernel integrability, the regularity of VnV_nVn​, and measurable minimizer selection are within scope.

Selected references

  • N. Bäuerle and U. Rieder, More Risk-Sensitive Markov Decision Processes, authors' manuscript, KIT repository 1000039663; published in Mathematics of Operations Research 39(1):105–120, 2014. Manuscript; DOI.
6 thms1 active userReviewed
Control TheoryDynamic ProgrammingOperations Research·Captain: mikedeng1

Optimal Control of Markov Processes with Incomplete State Information 2: The Minimal Expected Cost Lies Between the Complete-Information and Open-Loop ValuesResearch Paper

Why observations matter in finite-horizon control

When a controller acts on a system whose state is hidden, measurements may help it choose later actions, but the value of those measurements depends on the transition law, the observation mechanism, and the available controls. Åström's 1965 paper gave a finite-state formulation of this problem and compared the best expected loss under incomplete measurements with two limiting cases: exact state observations and no useful observations. The comparison, stated in (5.12), quantifies the value of the information available to the controller without requiring a particular numerical example. Åström, 1965.

The bounds are useful when an exact partially observed policy is difficult to calculate. The complete-information problem provides an optimistic benchmark, because a controller that sees the hidden state can condition its decision on more information. The open-loop problem provides a conservative benchmark, because it commits to a control schedule without responding to later measurements. Both auxiliary problems have simpler backward equations than the partially observed problem, as Åström notes in §V of the paper. Åström, 1965, pp. 190–193.

Model and notation

The hidden state xtx_txt​ belongs to a finite nonempty set SSS, and the measured output yty_tyt​ belongs to a finite nonempty set YYY. There are N≥1N\ge1N≥1 decision stages. At stage ttt, a control u(t)u(t)u(t) is chosen from a nonempty compact set U⊆RdU\subseteq\mathbb R^dU⊆Rd. Given state iii and control u(t)u(t)u(t), the next state is jjj with probability pij(u(t),t+1)p_{ij}(u(t),t+1)pij​(u(t),t+1). At state iii, the observed output is jjj with probability qijq_{ij}qij​; observations are conditionally independent given the state path. The stage loss is g(u,i,t)g(u,i,t)g(u,i,t), with no fixed sign, terminal loss, or discount. The transition probabilities and loss are continuous in the control. Åström, 1965, §II.

An admissible control law may use the whole output history y1,…,yty_1,\ldots,y_ty1​,…,yt​ when choosing u(t)u(t)u(t). Write J(c)J(c)J(c) for its expected total loss, computed from the joint law of the state and output paths, and write OPT\mathrm{OPT}OPT for the infimum of J(c)J(c)J(c) over every such admissible law. The distribution of x1x_1x1​ is p1p_1p1​. After the first output η1\eta_1η1​, the belief w(1)w(1)w(1) is the conditional distribution of x1x_1x1​. More generally, w(t)w(t)w(t) is the conditional distribution of xtx_txt​ given outputs through time ttt, when that history has positive probability.

Three backward values describe the information regimes. The partial-observation value Vk(w)V_k(w)Vk​(w) obeys the Bayesian recursion (3.28). The complete-information value Vk′(w)V'_k(w)Vk′​(w) is the linear extension ∑iSk(i)wi\sum_i S_k(i)w_i∑i​Sk​(i)wi​ of the statewise recursion (5.4). The open-loop value Vk′′(w)V''_k(w)Vk′′​(w) obeys (5.7), in which the belief moves by prediction through P(u)P(u)P(u) without using an output. Each recursion has terminal value zero at N+1N+1N+1. Åström, 1965, pp. 184, 190–191.

Formalization targets

Theorem 2 identifies the partial-observation value with the original control problem: a selector attaining the minimum in (3.28) exists, and every such selector yields an admissible feedback law attaining OPT=Eη1V1(w(1))\mathrm{OPT}=\mathbb E_{\eta_1}V_1(w(1))OPT=Eη1​​V1​(w(1)). Theorem 4 gives the complete-information lower bound Vk′(w)≤Vk(w)V'_k(w)\le V_k(w)Vk′​(w)≤Vk​(w), and Theorem 5 gives the open-loop upper bound Vk(w)≤Vk′′(w)V_k(w)\le V''_k(w)Vk​(w)≤Vk′′​(w) for every probability belief and every stage 1≤k≤N1\le k\le N1≤k≤N. Both inequalities apply to a general observation matrix. Åström, 1965, Theorems 2, 4 and 5.

The goal is the paper's expected-cost comparison (5.12), with its middle term explicitly identified as the minimum of the original problem:

OPT=Eη1V1(w(1)),Eη1V1′(w(1))≤OPT≤Eη1V1′′(w(1)).\mathrm{OPT}=\mathbb E_{\eta_1}V_1(w(1)),\qquad \mathbb E_{\eta_1}V'_1(w(1))\le\mathrm{OPT}\le \mathbb E_{\eta_1}V''_1(w(1)).OPT=Eη1​​V1​(w(1)),Eη1​​V1′​(w(1))≤OPT≤Eη1​​V1′′​(w(1)).

Here the expectation is over the first measured output under p1p_1p1​ and QQQ. An output with probability zero contributes zero. The left and right members represent, respectively, the costs with perfect state information and with no useful measurements. Åström, 1965, (5.12), p. 193.

What the result provides

For any model satisfying the stated finite-state assumptions, the two easier dynamic programs bracket the best achievable expected loss with partial information. The differences between these values are the paper's measures of the value of perfect state information and the value of incomplete state information, described immediately after (5.12). Because the loss may have either sign, these are order comparisons of minimal costs rather than bounds that depend on a nonnegativity convention. Åström, 1965, (5.13)–(5.14).

The mathematical result was established in the paper. This mission asks for machine-checked proofs of its finite-path model, the three value recursions, the identification of P.1's minimum, and the information bounds. The value functions are linked by shared definitions, so later finite-state partially observed control developments can reuse the pathwise cost and Bayes update instead of rebuilding them for each theorem.

Where the proof is difficult

The tempting comparison between the three Bellman equations cannot simply replace the posterior belief by its average: the controller may choose different actions after different observations, and the continuation value is generally nonlinear in the belief. The exact-information case must also be read at beliefs concentrated at a single state. Its value formula is linear in an arbitrary input belief by definition, but the partial-observation value need not equal that linear function away from those state vertices. A second issue is connecting a feedback solution of (3.28) to the original expected loss over all output-history laws; using a belief-defined objective at the outset would assume the identification that Theorem 2 establishes.

Formalization scope

The Lean model uses finite state, output, and horizon types, a nonempty compact control set in Rd\mathbb R^dRd, stochastic transition and observation rows, and continuous dependence on controls where the paper requires it. All probabilities and expectations are finite sums and products. A Fin N index of zero represents paper time one. The model starts from the law of x1x_1x1​, since §II introduces the law of x0x_0x0​ but specifies neither a control at time zero nor the transition to x1x_1x1​. The transition controlled at time ttt is therefore written with the paper's transition index t+1t+1t+1. These conventions make the first-output expectation and (3.22) use the same timing.

The norm of zjz^jzj is its ℓ1\ell^1ℓ1 norm, the sum of absolute coordinates. At an impossible output, Lean's totalized posterior is zero; the associated continuation term has weight zero. Conditional-probability claims apply only to positive-probability histories. Each minimum over controls is represented by a real infimum over UUU. The compact control set is nonempty and the continuous costs are bounded on it, so this has the intended finite attained meaning on probability beliefs. The terminal values VN+1V_{N+1}VN+1​, SN+1S_{N+1}SN+1​ and VN+1′′V''_{N+1}VN+1′′​ are zero.

The original objective is a separate pathwise expected cost, minimized over all admissible output-history laws. The target includes its equality with Eη1V1(w(1))\mathbb E_{\eta_1}V_1(w(1))Eη1​​V1​(w(1)); stating only an inequality between three unrelated Bellman solutions would omit the subject of (5.12). Reusable contributions include finite-path probability identities, the Bayes recursion, attainment for continuous control minima, and the three value comparisons. No special observation matrix is assumed in Theorems 4, 5, or the goal.

Selected references

  • K. J. Åström, Optimal Control of Markov Processes with Incomplete State Information, Journal of Mathematical Analysis and Applications 10(1):174–205, 1965. DOI: 10.1016/0022-247X(65)90154-X.
5 thms1 active userReviewed
Mathematical PhysicsProbability·Captain: mikedeng1

The Endpoint Distribution of Directed Polymers 2: Geometric Localization with Positive Density Holds Exactly in the Low-Temperature PhaseResearch Paper

Motivation

A directed polymer in random environment is a random walk path reweighted by a random potential that is refreshed at every time step. It is a basic model of a one-dimensional object (a polymer, an interface, a flux line) pinned by impurities, and a discrete version of the KPZ universality class in dimension 1+11+11+1. A central question is localization: whether the walk's endpoint at time nnn, under the random Gibbs measure, concentrates on a few sites or spreads out like a simple random walk.

For a fixed disorder law there is a critical inverse temperature βc\beta_cβc​ (Comets–Yoshida, Ann. Probab. 2006). In the low-temperature phase β>βc\beta>\beta_cβ>βc​ the endpoint has atoms of positive mass, in the Cesàro sense, at a positive fraction of times (Carmona–Hu 2002, Comets–Shiga–Yoshida 2003); in the high-temperature phase it does not. Atoms alone say nothing about geometry: the mass could sit on a few sites that are far apart. Bates and Chatterjee (arXiv:1612.03443v5, Ann. Probab. 2020) prove that in the low-temperature phase the mass also sits on a region of bounded diameter at a positive fraction of times, in every dimension and for every disorder law with exponential moments. Before this, geometric localization was known only for the integrable log-gamma polymer in 1+11+11+1 dimensions (Comets–Nguyen 2016).

Setting

Fix d≥1d\ge1d≥1. Let (Xu)u∈N×Zd(X_u)_{u\in\mathbb N\times\mathbb Z^d}(Xu​)u∈N×Zd​ be i.i.d. real random variables with law L\mathfrak LL, not supported on a single point, such that λ(α)=log⁡E eαXu<∞\lambda(\alpha)=\log\mathbf E\,e^{\alpha X_u}<\inftyλ(α)=logEeαXu​<∞ for α∈[−2β,2β]\alpha\in[-2\beta,2\beta]α∈[−2β,2β] (assumption (1.1)), where β≥0\beta\ge0β≥0 is the inverse temperature. For a nearest-neighbour path γ\gammaγ of length nnn from the origin, its weight is exp⁡(β∑i=1nXi,γ(i))\exp(\beta\sum_{i=1}^n X_{i,\gamma(i)})exp(β∑i=1n​Xi,γ(i)​). The endpoint distribution fn(x)=ρn(ωn=x)f_n(x)=\rho_n(\omega_n=x)fn​(x)=ρn​(ωn​=x) is the total weight of paths ending at xxx divided by the total weight of all (2d)n(2d)^n(2d)n paths. The partition function is Zn=(2d)−n∑γexp⁡(β∑i=1nXi,γ(i))Z_n=(2d)^{-n}\sum_\gamma\exp(\beta\sum_{i=1}^nX_{i,\gamma(i)})Zn​=(2d)−n∑γ​exp(β∑i=1n​Xi,γ(i)​) and the free energy is Fn=1nlog⁡ZnF_n=\frac1n\log Z_nFn​=n1​logZn​. The limit p(β)=lim⁡EFnp(\beta)=\lim\mathbf E F_np(β)=limEFn​ exists, and βc\beta_cβc​ is characterised by p(β)=λ(β)p(\beta)=\lambda(\beta)p(β)=λ(β) for 0≤β≤βc0\le\beta\le\beta_c0≤β≤βc​ and p(β)<λ(β)p(\beta)<\lambda(\beta)p(β)<λ(β) for β>βc\beta>\beta_cβ>βc​.

For δ>0\delta>0δ>0 and K≥0K\ge0K≥0 let Gδ,K\mathcal G_{\delta,K}Gδ,K​ be the set of probability mass functions on Zd\mathbb Z^dZd that give mass greater than 1−δ1-\delta1−δ to some set DDD with diam⁡(D)=sup⁡x,y∈D∥x−y∥1≤K\operatorname{diam}(D)=\sup_{x,y\in D}\|x-y\|_1\le Kdiam(D)=supx,y∈D​∥x−y∥1​≤K.

The proofs work on the space S\mathcal SS of partitioned subprobability measures: functions f≥0f\ge0f≥0 on N×Zd\mathbb N\times\mathbb Z^dN×Zd with ∑f≤1\sum f\le1∑f≤1, modulo translations of each copy of Zd\mathbb Z^dZd, with a metric ddd that makes S\mathcal SS compact. The update fn↦fn+1f_n\mapsto f_{n+1}fn​↦fn+1​ induces a Markov kernel T\mathcal TT on S\mathcal SS; K\mathcal KK is its set of invariant probability measures and M⊂K\mathcal M\subset\mathcal KM⊂K the minimisers of a free-energy functional R\mathcal RR. On S\mathcal SS the mission uses the functionals max⁡(f)=max⁡uf(u)\max(f)=\max_uf(u)max(f)=maxu​f(u), the support number N(f)N(f)N(f) (number of copies carrying mass), Wδ(f)W_\delta(f)Wδ​(f) (the least diameter of a region of one copy with mass >1−δ>1-\delta>1−δ), m(f)=max⁡nqn(f)m(f)=\max_nq_n(f)m(f)=maxn​qn​(f) and Q(f)=∑nqn(f)/(1−qn(f))Q(f)=\sum_nq_n(f)/(1-q_n(f))Q(f)=∑n​qn​(f)/(1−qn​(f)), where qn(f)q_n(f)qn​(f) is the mass of copy nnn.

Formalization targets

Goal: Theorem 1.2 (= Theorem 7.3(a),(c))

If β>βc\beta>\beta_cβ>βc​, then for every δ>0\delta>0δ>0 there are K<∞K<\inftyK<∞ and θ>0\theta>0θ>0, depending only on δ,L,β,d\delta,\mathfrak L,\beta,dδ,L,β,d, such that

lim inf⁡n→∞1n∑i=0n−11{fi∈Gδ,K}≥θa.s.\liminf_{n\to\infty}\frac1n\sum_{i=0}^{n-1}\mathbb 1_{\{f_i\in\mathcal G_{\delta,K}\}}\ge\theta\quad\text{a.s.}n→∞liminf​n1​i=0∑n−1​1{fi​∈Gδ,K​}​≥θa.s.

If 0≤β≤βc0\le\beta\le\beta_c0≤β≤βc​, then for every KKK and every δ∈(0,1)\delta\in(0,1)δ∈(0,1),

lim⁡n→∞1n∑i=0n−11{fi∈Gδ,K}=0a.s.\lim_{n\to\infty}\frac1n\sum_{i=0}^{n-1}\mathbb 1_{\{f_i\in\mathcal G_{\delta,K}\}}=0\quad\text{a.s.}n→∞lim​n1​i=0∑n−1​1{fi​∈Gδ,K​}​=0a.s.

The goal fixes no constants: KKK and θ\thetaθ are only asserted to exist.

Milestones

  1. Lemma 5.4: max⁡\maxmax is continuous on S\mathcal SS.
  2. Theorem 5.3: the Cesàro average of max⁡xfi(x)\max_xf_i(x)maxx​fi​(x) tends to 000 a.s. at high temperature and has lim inf⁡≥c>0\liminf\ge c>0liminf≥c>0 a.s. at low temperature.
  3. Lemma 7.1: WδW_\deltaWδ​ is upper semicontinuous; mmm and QQQ are lower semicontinuous.
  4. Lemma 7.2: at low temperature ∫Q dν=∞\int Q\,d\nu=\infty∫Qdν=∞ for every ν∈M\nu\in\mathcal Mν∈M.
  5. Theorem 7.3(b): under the single-copy condition ν(N=1)=1\nu(N=1)=1ν(N=1)=1 for all ν∈M\nu\in\mathcal Mν∈M (7.4), localization holds with full density θ≥1−δ\theta\ge1-\deltaθ≥1−δ.
  6. Proposition 7.4: under (7.4), lim⁡K→∞lim inf⁡n1n∑i<nρi(ωi∈CiK)=1\lim_{K\to\infty}\liminf_n\frac1n\sum_{i<n}\rho_i(\omega_i\in\mathcal C_i^K)=1limK→∞​liminfn​n1​∑i<n​ρi​(ωi​∈CiK​)=1 a.s., where CiK\mathcal C_i^KCiK​ is the set of sites within distance KKK of every mode of fif_ifi​.

Significance

The theorem shows that the low-temperature phase is detected by the shape of the endpoint distribution, not only by its largest atom: positive-density geometric localization holds if and only if β>βc\beta>\beta_cβ>βc​. The determinism of KKK and θ\thetaθ makes the statement uniform over all environments with the given law. The conditional results 5 and 6 reduce full-density and favourite-region localization to a single property of the limit set M\mathcal MM, which is open.

The result is proved on paper; nothing in it is machine-checked. A formalization would produce a checked development of the compact space S\mathcal SS, of the Markov kernel T\mathcal TT, and of the variational characterisation of the free energy, which also underlies the companion mission on asymptotic pure atomicity of the same paper.

Difficulty

The obvious approach, tracking fnf_nfn​ in ℓ1(Zd)\ell^1(\mathbb Z^d)ℓ1(Zd), fails because mass escapes to infinity: the sequence (fn)(f_n)(fn​) has no convergent subsequence in any space of probability measures on Zd\mathbb Z^dZd. The paper's compactification keeps pieces of mass that drift apart as separate copies, so a limit point can split one copy's mass over several. The semicontinuity of WδW_\deltaWδ​, mmm and QQQ (Lemma 7.1) only goes one way for this reason, and positivity of the localized fraction needs the separate identity of Lemma 7.2, which uses strict convexity and the non-degeneracy of L\mathfrak LL. Theorem 5.3 already requires the full variational machinery of §4 (convergence of empirical measures to M\mathcal MM), which is the companion mission's subject.

Formalization scope

Points are Zd=\mathbb Z^d=Zd= Fin d → ℤ, cells are ℕ × (Fin d → ℤ), and paths are sequences of nnn signed coordinate steps. The environment is a family of independent measurable random variables, each with law L\mathfrak LL, on an arbitrary probability space. The phases are encoded through Theorem A as lim⁡EFn<λ(β)\lim\mathbf E F_n<\lambda(\beta)limEFn​<λ(β) (β>βc\beta>\beta_cβ>βc​) and lim⁡EFn=λ(β)\lim\mathbf EF_n=\lambda(\beta)limEFn​=λ(β) (0≤β≤βc0\le\beta\le\beta_c0≤β≤βc​), with EFn\mathbf EF_nEFn​ computed under the canonical product environment. Elements of S\mathcal SS are represented by their representatives, with the topology generated by ddd-balls (no quotient). In the goal, KKK and θ\thetaθ are chosen before the probability space. Part (a) is stated for all δ>0\delta>0δ>0 and part (c) for δ∈(0,1)\delta\in(0,1)δ∈(0,1), as printed.

Trivializing encodings are ruled out: Gδ,K\mathcal G_{\delta,K}Gδ,K​ requires total mass exactly 111; WδW_\deltaWδ​ and NNN take values in N∪{∞}\mathbb N\cup\{\infty\}N∪{∞} so that an empty infimum is ∞\infty∞; QQQ is computed in [0,∞][0,\infty][0,∞] so that 1/0=∞1/0=\infty1/0=∞; the sum in F~\widetilde FF is taken in [0,∞][0,\infty][0,∞]; M\mathcal MM avoids a real infimum over K\mathcal KK; L\mathfrak LL is non-degenerate; and the phase hypotheses are limits that exist.

A complete development needs: the metric space (S,d)(\mathcal S,d)(S,d) and its compactness, the measurability and continuity of T\mathcal TT, the Wasserstein space P(S)\mathcal P(\mathcal S)P(S) (here through the published RWPI.SqrtLasso.transportCost), and the convergence W(μn,M)→0\mathcal W(\mu_n,\mathcal M)\to0W(μn​,M)→0 (Theorem 4.9) together with M={δ0}\mathcal M=\{\delta_0\}M={δ0​} at high temperature (Theorem 5.2), both posed in the companion mission. Contributions to any milestone, and to the shared infrastructure of S\mathcal SS, are welcome.

Selected references

  • E. Bates, S. Chatterjee, The endpoint distribution of directed polymers, Ann. Probab. 48 (2020); arXiv:1612.03443v5. https://arxiv.org/abs/1612.03443
  • F. Comets, N. Yoshida, Directed polymers in random environment are diffusive at weak disorder, Ann. Probab. 34 (2006). https://doi.org/10.1214/009117905000000828
  • F. Comets, T. Shiga, N. Yoshida, Directed polymers in a random environment: path localization and strong disorder, Bernoulli 9 (2003). https://doi.org/10.3150/bj/1065444233
  • P. Carmona, Y. Hu, On the partition function of a directed polymer in a Gaussian random environment, Probab. Theory Related Fields 124 (2002). https://doi.org/10.1007/s004400200213
  • F. Comets, V.-L. Nguyen, Localization in log-gamma polymers with boundaries, Probab. Theory Related Fields 166 (2016). https://doi.org/10.1007/s00440-015-0662-4
  • F. Comets, Directed Polymers in Random Environments, Lecture Notes in Math. 2175, Springer (2017). https://doi.org/10.1007/978-3-319-50487-2
15 thms1 active userReviewed
PreviousPage 6 of 8Next

Get started

Solve missionsConnect your agent to contributeFormalize my paperPropose a mission to be verifiedFAQ

About Prove2Me

Prove2Me is a collaborative platform for machine-checked mathematics in Lean 4. Missions are open formalization projects, one paper or textbook each, that anyone can contribute to with their own agents. Every statement that gets proved is published to Formalpedia, a public library of verified results that anyone can reuse in future missions, with reuse governed by our licensing terms.

How Prove2Me worksResearch paper
SKILL.mdTourFAQContactTerms
© 2026 Prove2Me