Prove2Me
Navigate
DiscoverCollectionsFormalpediaBlogsUsersMomentumMy Missions+
Prove2Me
⌕
Log in
← Collections

The OR Formalization Drive

Help us formalize the operations research literature in Lean.

441 open missions

Missions

201–220 of 441
OpenCompletedAll
AnalysisDynamic ProgrammingOperations Research+3·Captain: mikedeng1

Stochastic Optimal Control: The Discrete-Time Case VI: Lower Semianalytic Functions — Analytically Measurable ε-Optimal Selectors (Jankov–von Neumann)Textbook

Motivation

Dynamic programming over uncountable state and control spaces needs two things at every stage: the optimal cost-to-go, obtained by minimizing over the control, must be a function that can be integrated against the next stage's transition probabilities, and a policy that nearly attains the minimum must be measurable, so that it defines a stochastic process. With Borel-measurable costs and Borel-measurable policies both requirements fail. Minimizing a Borel function of (x,y)(x,y)(x,y) over yyy produces a function whose level sets are projections of Borel sets, and such projections need not be Borel (Suslin, 1917). The repair, developed by Blackwell, Freedman and Orkin (1974), Shreve and Bertsekas, and set out in Chapter 7 of Bertsekas and Shreve's Stochastic Optimal Control: The Discrete-Time Case (1978), is to enlarge the class of costs to the lower semianalytic functions and the class of policies to the analytically or universally measurable ones. Sections 7.6–7.7 of the book establish that this class is closed under partial minimization and admits measurable ε-optimal selectors. Chapters 8–10 of the book, and much of the later literature on Borel-space Markov decision processes (Hernández-Lerma and Lasserre; Feinberg and coauthors), build on these results.

Timeline:

  • 1917: Suslin shows that projections of Borel sets need not be Borel and introduces analytic sets; Lusin proves that analytic sets are universally measurable.
  • 1941–1949: Jankov and von Neumann independently prove that an analytic subset of a product admits a selector measurable with respect to the σ-algebra generated by analytic sets.
  • 1974: Blackwell, Freedman and Orkin use analytic sets to construct ε-optimal policies in Borel dynamic programming.
  • 1978: Bertsekas and Shreve give the treatment used here (§7.6–7.7), including the selection theorem for lower semianalytic functions, Proposition 7.50.

Setting

A Borel space is a topological space homeomorphic to a Borel subset of a complete separable metric space (Definition 7.7); its Borel σ-algebra is BX\mathscr B_XBX​. The Baire space is N=NN\mathscr N=\mathbb N^{\mathbb N}N=NN with the product topology. A set A⊆XA\subseteq XA⊆X is analytic if it is empty or the image of N\mathscr NN under a continuous map; by Proposition 7.41 this is the book's Definition 7.16 (the Suslin operation applied to closed sets). Every Borel set is analytic, and the converse fails when XXX is uncountable.

Three σ-algebras on XXX are in play. The analytic σ-algebra AX\mathscr A_XAX​ is generated by the analytic sets (Definition 7.19). The universal σ-algebra is UX=⋂pBX(p)\mathscr U_X=\bigcap_{p}\mathscr B_X(p)UX​=⋂p​BX​(p), the intersection over all probability measures ppp on (X,BX)(X,\mathscr B_X)(X,BX​) of the ppp-completions of BX\mathscr B_XBX​ (Definition 7.18). For a function fff from D⊆XD\subseteq XD⊆X into a Borel space YYY, fff is analytically measurable if D∈AXD\in\mathscr A_XD∈AX​ and f−1(B)∈AXf^{-1}(B)\in\mathscr A_Xf−1(B)∈AX​ for every B∈BYB\in\mathscr B_YB∈BY​, and universally measurable if the same holds with UX\mathscr U_XUX​ (Definition 7.20).

Let R∗=[−∞,∞]R^*=[-\infty,\infty]R∗=[−∞,∞]. A function f:D→R∗f:D\to R^*f:D→R∗ is lower semianalytic if DDD is analytic and {x∈D∣f(x)<c}\{x\in D\mid f(x)<c\}{x∈D∣f(x)<c} is analytic for every real ccc (Definition 7.21). For D⊆X×YD\subseteq X\times YD⊆X×Y write Dx={y∣(x,y)∈D}D_x=\{y\mid (x,y)\in D\}Dx​={y∣(x,y)∈D}, projX(D)={x∣Dx≠∅}\mathrm{proj}_X(D)=\{x\mid D_x\neq\emptyset\}projX​(D)={x∣Dx​=∅}, and define the partial infimum

f∗(x)=inf⁡y∈Dxf(x,y),x∈projX(D).f^*(x)=\inf_{y\in D_x}f(x,y),\qquad x\in\mathrm{proj}_X(D).f∗(x)=y∈Dx​inf​f(x,y),x∈projX​(D).

A selector is a function φ:projX(D)→Y\varphi:\mathrm{proj}_X(D)\to Yφ:projX​(D)→Y whose graph Gr(φ)\mathrm{Gr}(\varphi)Gr(φ) lies in DDD.

Formalization targets

Goal: Proposition 7.50

Let X,YX,YX,Y be Borel spaces, D⊆X×YD\subseteq X\times YD⊆X×Y analytic, and f:D→R∗f:D\to R^*f:D→R∗ lower semianalytic.

(a) For every ε>0\varepsilon>0ε>0 there is an analytically measurable selector φ\varphiφ with

f[x,φ(x)]≤{f∗(x)+εif f∗(x)>−∞,−1/εif f∗(x)=−∞.f[x,\varphi(x)]\le\begin{cases}f^*(x)+\varepsilon&\text{if }f^*(x)>-\infty,\\-1/\varepsilon&\text{if }f^*(x)=-\infty.\end{cases}f[x,φ(x)]≤{f∗(x)+ε−1/ε​if f∗(x)>−∞,if f∗(x)=−∞.​

(b) The set III of points where the infimum is attained is universally measurable, and for every ε>0\varepsilon>0ε>0 there is a universally measurable selector φ\varphiφ with f[x,φ(x)]=f∗(x)f[x,\varphi(x)]=f^*(x)f[x,φ(x)]=f∗(x) on III and the bounds of (a) off III.

The goal fixes no constant beyond the book's ε\varepsilonε and −1/ε-1/\varepsilon−1/ε.

Milestones

In attack order: Proposition 7.40 (Borel images and preimages of analytic sets are analytic), Corollary 7.42.1 (AX⊆UX\mathscr A_X\subseteq\mathscr U_XAX​⊆UX​), Corollary 7.44.2 (composites of analytically measurable maps are universally measurable), and Proposition 7.49, the Jankov–von Neumann theorem:

A⊆X×Y analytic ⟹ ∃ φ:projX(A)→Y analytically measurable, Gr(φ)⊆A.A\subseteq X\times Y\text{ analytic}\ \Longrightarrow\ \exists\,\varphi:\mathrm{proj}_X(A)\to Y\ \text{analytically measurable},\ \mathrm{Gr}(\varphi)\subseteq A.A⊆X×Y analytic ⟹ ∃φ:projX​(A)→Y analytically measurable, Gr(φ)⊆A.

Further items of the mission, on the same definitions: Proposition 7.39 (projections of analytic sets are analytic, and every analytic set is a projection of a Borel set), Lemma 7.30(1) (strict and non-strict, real and extended level sets give the same class) and Proposition 7.47 (lower semianalytic functions are exactly partial infima of Borel functions).

Significance

Proposition 7.50 is the selection theorem behind the existence of ε-optimal policies in Borel-space dynamic programming. In the finite-horizon model of Chapter 8 the optimal cost-to-go at each stage is lower semianalytic, by Propositions 7.47 and 7.48. Proposition 7.50 then turns the one-stage minimization into a measurable policy, analytically measurable when only ε-optimality is required and universally measurable when the minimum is attained. Chapters 8–9 of the book (the finite-horizon recursion JK∗=TK(J0)J^*_K=T^K(J_0)JK∗​=TK(J0​) and the optimality equation under (P), (N), (D)) use it at every step. Downstream catalog papers on average-cost and stochastic shortest-path problems over Borel spaces cite these results.

All results here are proved in the book and in the descriptive set theory literature (Kechris, Classical Descriptive Set Theory, §18 and §29). None is formalized on Prove2Me. Mathlib has analytic sets in Polish-type settings, the Lusin separation theorem and Suslin's theorem, but it has no universal σ-algebra, no analytic σ-algebra, no lower semianalytic functions and no Jankov–von Neumann uniformization. The definitions in this mission are reusable by the later missions of the series (Chapters 8–10), which restate them locally until these are published.

Difficulty

The obvious route to a selector is to choose, for each xxx, a minimizing or near-minimizing yyy. The axiom of choice provides such a function, but nothing makes it measurable, and the conclusion of the theorem is exactly that measurability. The Borel route fails too: the set {x∣f∗(x)<c}\{x\mid f^*(x)<c\}{x∣f∗(x)<c} is a projection of a Borel set, which is analytic but in general not Borel, so no Borel-measurable selector exists in general. The Jankov–von Neumann theorem needs a lexicographically least branch of a continuous parametrization of AAA by N\mathscr NN, and an argument that the resulting map is measurable with respect to AX\mathscr A_XAX​, which is generated by sets that are not closed under complementation. Part (b) adds a further obstacle: the composite of two analytically measurable maps need not be analytically measurable, so the exact selector is only universally measurable. Proving that requires Lusin's theorem that analytic sets are measurable for every completed probability measure.

Formalization scope

  • A Borel space is a type with a topology satisfying the class IsBorelSpace (Definition 7.7, the ambient complete separable metric space taken in the same universe), together with Mathlib's [MeasurableSpace X] [BorelSpace X], so measurable sets are exactly the Borel sets. On X×YX\times YX×Y the product σ-algebra is used; it coincides with BX×Y\mathscr B_{X\times Y}BX×Y​ for separable metrizable spaces (Proposition 7.13).
  • Analytic sets are Mathlib's MeasureTheory.AnalyticSet (empty or a continuous image of ℕ → ℕ).
  • R∗R^*R∗ is EReal. The book uses ∞−∞=∞\infty-\infty=\infty∞−∞=∞, and Mathlib's EReal uses ⊥+⊤=⊥\bot+\top=\bot⊥+⊤=⊥. No statement of this mission adds infinities of opposite sign; f∗(x)+εf^*(x)+\varepsilonf∗(x)+ε adds a real number.
  • Functions on DDD and on projX(D)\mathrm{proj}_X(D)projX​(D) are functions on subtypes. The graph condition Gr(φ)⊆D\mathrm{Gr}(\varphi)\subseteq DGr(φ)⊆D is part of every selector statement.
  • Universally measurable means NullMeasurableSet E p for every probability measure p.
  • "Analytically measurable" refers to the σ-algebra generated by analytic sets. Replacing it by the power set, dropping the graph condition, or dropping the −1/ε-1/\varepsilon−1/ε case would make the selection theorems a consequence of the axiom of choice. The statements rule all three out.

Not included: Lusin's theorem in Suslin-scheme form (Proposition 7.42, which needs the Suslin operation as a definition), Proposition 7.43 on P(X)P(X)P(X), the integration results of Propositions 7.46 and 7.48, and Lemma 7.30(2)–(4). None is used in the proof of the goal. Contributions welcome: the bridge between IsBorelSpace and Mathlib's StandardBorelSpace, the universal σ-algebra API, and the Jankov–von Neumann theorem itself.

Selected references

  • D. P. Bertsekas and S. E. Shreve, Stochastic Optimal Control: The Discrete-Time Case, Academic Press 1978; Athena Scientific 1996, §7.6–7.7. https://web.mit.edu/dimitrib/www/soc.html
  • D. Blackwell, D. Freedman and M. Orkin, The optimal reward operator in dynamic programming, Annals of Probability 2 (1974) 926–941. https://doi.org/10.1214/aop/1176996558
  • A. S. Kechris, Classical Descriptive Set Theory, Graduate Texts in Mathematics 156, Springer 1995, §18 (Jankov–von Neumann uniformization), §29 (measurability of analytic sets). https://doi.org/10.1007/978-1-4612-4190-4
  • S. E. Shreve and D. P. Bertsekas, Universally measurable policies in dynamic programming, Mathematics of Operations Research 4 (1979) 15–30. https://doi.org/10.1287/moor.4.1.15
8 thms2 active usersReviewed
Dynamical SystemsOperations ResearchProbability+1·Captain: mikedeng1

Dynamics of Stochastic Approximation Algorithms 4: Subgaussian Martingale Noise with Σ exp(−c/γ_n) < ∞ for Every c > 0 Satisfies Assumption A1 Almost SurelyResearch Paper

Motivation

A stochastic approximation algorithm is a recursion

xn+1−xn=γn+1(F(xn)+Un+1)x_{n+1}-x_n=\gamma_{n+1}\big(F(x_n)+U_{n+1}\big)xn+1​−xn​=γn+1​(F(xn​)+Un+1​)

in Rd\mathbb R^dRd, where FFF is a vector field, γn\gamma_nγn​ are small step sizes and Un+1U_{n+1}Un+1​ is noise. Such recursions go back to Robbins and Monro's root-finding scheme (Robbins–Monro 1951) and underlie stochastic gradient descent, temporal-difference learning, adaptive control and learning in games. The ODE method studies them by comparing the iterates with the trajectories of x˙=F(x)\dot x=F(x)x˙=F(x).

Benaïm's lecture notes (Benaïm 1999) organize the ODE method in two steps. A deterministic step, Proposition 4.1, shows that whenever the noise satisfies a condition called A1 (together with a boundedness condition on the iterates), the interpolated process is an asymptotic pseudotrajectory of the flow of FFF. A probabilistic step then verifies A1 for concrete noise models. Proposition 4.2 does this for martingale difference noise with bounded qqq-th moments, at the price of step sizes with ∑nγn1+q/2<∞\sum_n\gamma_n^{1+q/2}<\infty∑n​γn1+q/2​<∞. This mission formalizes the second verification, Proposition 4.4: when the noise is subgaussian, A1 holds almost surely under the much weaker requirement that ∑ne−c/γn<∞\sum_ne^{-c/\gamma_n}<\infty∑n​e−c/γn​<∞ for every c>0c>0c>0, which allows step sizes decaying only slightly faster than 1/log⁡n1/\log n1/logn. The notes attribute the result to Duflo (1997), see also Kushner and Yin (1997) and Benaïm and Hirsch (1996).

Setting

Let {γn}n≥1\{\gamma_n\}_{n\ge1}{γn​}n≥1​ be a deterministic sequence with γn≥0\gamma_n\ge0γn​≥0, ∑nγn=∞\sum_n\gamma_n=\infty∑n​γn​=∞ and γn→0\gamma_n\to0γn​→0 (a step sequence). Put τ0=0\tau_0=0τ0​=0, τn=∑i=1nγi\tau_n=\sum_{i=1}^n\gamma_iτn​=∑i=1n​γi​, and let

m(t)=sup⁡{k≥0: t≥τk}m(t)=\sup\{k\ge0:\ t\ge\tau_k\}m(t)=sup{k≥0: t≥τk​}

be the index of the step that contains time t≥0t\ge0t≥0. For a sequence {Un}n≥1\{U_n\}_{n\ge1}{Un​}n≥1​ define the piecewise constant processes Uˉ(t)=Um(t)+1\bar U(t)=U_{m(t)+1}Uˉ(t)=Um(t)+1​ and γˉ(t)=γm(t)+1\bar\gamma(t)=\gamma_{m(t)+1}γˉ​(t)=γm(t)+1​, so that step n+1n+1n+1 occupies the time interval [τn,τn+1)[\tau_n,\tau_{n+1})[τn​,τn+1​) of length γn+1\gamma_{n+1}γn+1​.

Assumption A1 asks that for every T>0T>0T>0

lim⁡n→∞sup⁡{∥∑i=nk−1γi+1Ui+1∥: k=n+1,…,m(τn+T)}=0,\lim_{n\to\infty}\sup\Big\{\Big\|\sum_{i=n}^{k-1}\gamma_{i+1}U_{i+1}\Big\|:\ k=n+1,\dots,m(\tau_n+T)\Big\}=0,n→∞lim​sup{​i=n∑k−1​γi+1​Ui+1​​: k=n+1,…,m(τn​+T)}=0,

or, in the form the notes call equivalent, lim⁡t→∞Δ(t,T)=0\lim_{t\to\infty}\Delta(t,T)=0limt→∞​Δ(t,T)=0 for every T>0T>0T>0, where

Δ(t,T)=sup⁡0≤h≤T∥∫tt+hUˉ(s) ds∥.\Delta(t,T)=\sup_{0\le h\le T}\Big\|\int_t^{t+h}\bar U(s)\,ds\Big\|.Δ(t,T)=0≤h≤Tsup​​∫tt+h​Uˉ(s)ds​.

Let (Ω,F,P)(\Omega,\mathcal F,P)(Ω,F,P) be a probability space with a nondecreasing sequence {Fn}\{\mathcal F_n\}{Fn​} of sub-σ\sigmaσ-algebras, and F:Rd→RdF:\mathbb R^d\to\mathbb R^dF:Rd→Rd continuous. A sequence {xn}\{x_n\}{xn​} given by the recursion above is a Robbins–Monro algorithm if γ\gammaγ is deterministic, UnU_nUn​ is Fn\mathcal F_nFn​-measurable, and E(Un+1∣Fn)=0E(U_{n+1}\mid\mathcal F_n)=0E(Un+1​∣Fn​)=0. The noise is subgaussian if there is a number Γ>0\Gamma>0Γ>0 such that for all nnn and all θ∈Rd\theta\in\mathbb R^dθ∈Rd

E(exp⁡⟨θ,Un+1⟩ ∣ Fn)≤exp⁡(Γ2∥θ∥2).E\big(\exp\langle\theta,U_{n+1}\rangle\,\big|\,\mathcal F_n\big)\le\exp\Big(\frac\Gamma2\|\theta\|^2\Big).E(exp⟨θ,Un+1​⟩​Fn​)≤exp(2Γ​∥θ∥2).

Bounded noise, ∥Un∥≤Γ\|U_n\|\le\sqrt\Gamma∥Un​∥≤Γ​, is an example.

Formalization targets

Goal: Proposition 4.4

For a Robbins–Monro algorithm with subgaussian noise and a deterministic step sequence such that

∑ne−c/γn<∞for each c>0,\sum_ne^{-c/\gamma_n}<\infty\qquad\text{for each }c>0,n∑​e−c/γn​<∞for each c>0,

with probability one the realised noise sequence satisfies A1, in both of its forms, simultaneously for all T>0T>0T>0.

Milestones

  1. The exponential supermartingale. For every θ∈Rd\theta\in\mathbb R^dθ∈Rd,
Zn(θ)=exp⁡[∑i=1n⟨θ,γiUi⟩−Γ2∑i=1nγi2∥θ∥2]Z_n(\theta)=\exp\Big[\sum_{i=1}^n\langle\theta,\gamma_iU_i\rangle-\frac\Gamma2\sum_{i=1}^n\gamma_i^2\|\theta\|^2\Big]Zn​(θ)=exp[i=1∑n​⟨θ,γi​Ui​⟩−2Γ​i=1∑n​γi2​∥θ∥2]

is a supermartingale. 2. Directional maximal tail bound. For every unit vector eee, α>0\alpha>0α>0, nnn and T>0T>0T>0,

P(sup⁡n<k≤m(τn+T)⟨e,∑i=nk−1γi+1Ui+1⟩≥α)≤exp⁡(−α22Γ∑i=nm(τn+T)−1γi+12).P\Big(\sup_{n<k\le m(\tau_n+T)}\Big\langle e,\sum_{i=n}^{k-1}\gamma_{i+1}U_{i+1}\Big\rangle\ge\alpha\Big)\le\exp\Big(\frac{-\alpha^2}{2\Gamma\sum_{i=n}^{m(\tau_n+T)-1}\gamma_{i+1}^2}\Big).P(n<k≤m(τn​+T)sup​⟨e,i=n∑k−1​γi+1​Ui+1​⟩≥α)≤exp(2Γ∑i=nm(τn​+T)−1​γi+12​−α2​).
  1. Eq. (18). There are C,C′>0C,C'>0C,C′>0 depending only on ddd and Γ\GammaΓ with
P(Δ(t,T)≥α)≤Cexp⁡(−α2C′∫tt+Tγˉ(s) ds)(t≥0, T>0, α>0).P(\Delta(t,T)\ge\alpha)\le C\exp\Big(\frac{-\alpha^2}{C'\int_t^{t+T}\bar\gamma(s)\,ds}\Big)\qquad(t\ge0,\ T>0,\ \alpha>0).P(Δ(t,T)≥α)≤Cexp(C′∫tt+T​γˉ​(s)ds−α2​)(t≥0, T>0, α>0).
  1. Block comparison. Δ(t,T)≤2Δ(kT,T)+Δ((k+1)T,T)\Delta(t,T)\le2\Delta(kT,T)+\Delta((k+1)T,T)Δ(t,T)≤2Δ(kT,T)+Δ((k+1)T,T) for kT≤t<(k+1)TkT\le t<(k+1)TkT≤t<(k+1)T.

Significance

Proposition 4.4 is the sufficient condition for the ODE method when the noise has Gaussian-type tails. Its step-size condition holds whenever γnlog⁡n→0\gamma_n\log n\to0γn​logn→0, so it admits steps that decrease far more slowly than the ∑γn2<∞\sum\gamma_n^2<\infty∑γn2​<∞ of the classical L2L^2L2 theory; slowly decreasing steps are what practitioners use to keep algorithms responsive. Combined with Proposition 4.1 it shows that the interpolated process of such an algorithm, with bounded iterates, is almost surely an asymptotic pseudotrajectory of the flow of FFF, and the limit set theorems of the notes then locate the limit points of the algorithm.

The result is proved in the notes and in the cited literature; it has not, to our knowledge, been machine-checked. A formal proof would add reusable pieces: an exponential supermartingale and maximal inequality for vector-valued martingale differences with a conditional subgaussian bound (Mathlib's conditional subgaussian notion is scalar), a Borel–Cantelli argument along the grid kTkTkT, and the continuous-time bookkeeping of Uˉ\bar UUˉ, γˉ\bar\gammaγˉ​ and Δ\DeltaΔ shared with the other missions of this series.

Difficulty

The moment method of Proposition 4.2 does not reach this regime: any fixed polynomial moment of the window sums decays only polynomially in the window's step sizes, and under ∑e−c/γn<∞\sum e^{-c/\gamma_n}<\infty∑e−c/γn​<∞ alone polynomial bounds are not summable over windows. Exponential tail bounds are needed, and they must be maximal (uniform over the window) and must hold for the norm of a vector, not only for a scalar. The continuous-time deviation Δ(t,T)\Delta(t,T)Δ(t,T) involves partial steps at both ends of [t,t+h][t,t+h][t,t+h], so the bound must be stated in terms of ∫tt+Tγˉ\int_t^{t+T}\bar\gamma∫tt+T​γˉ​ rather than a sum over whole steps, with constants that do not depend on ttt, TTT or α\alphaα. Finally, A1 quantifies over all T>0T>0T>0: the almost-sure statement must hold on a single event of full probability for every TTT.

Formalization scope

The space is Rd\mathbb R^dRd as EuclideanSpace ℝ (Fin d) (the paper writes Rm\mathbb R^mRm); time is real. The sequences γ\gammaγ and UUU are indexed by N\mathbb NN, and their values at 000 are unused, as the paper indexes them from 111. The filtration is a Mathlib Filtration ℕ; Un+1U_{n+1}Un+1​ is Fn+1\mathcal F_{n+1}Fn+1​-strongly measurable and integrable, and E(Un+1∣Fn)=0E(U_{n+1}\mid\mathcal F_n)=0E(Un+1​∣Fn​)=0 almost surely. The subgaussian condition requires exp⁡⟨θ,Un+1⟩\exp\langle\theta,U_{n+1}\rangleexp⟨θ,Un+1​⟩ to be integrable for every θ\thetaθ and nnn. The summand e−c/γne^{-c/\gamma_n}e−c/γn​ is taken to be 000 when γn=0\gamma_n=0γn​=0, its limiting value. The suprema in A1 and Δ\DeltaΔ are taken in [0,∞][0,\infty][0,∞]; the supremum over an empty range of kkk is 000. In Eq. (18) the constants are chosen before the probability space, the algorithm and t,T,αt,T,\alphat,T,α.

The following readings are excluded and are not acceptable formalizations: a subgaussian condition that holds vacuously because the exponential is not integrable (Lean's conditional expectation of a non-integrable function is 000); a summability condition made trivial or false by the convention c/0=0c/0=0c/0=0; and the conclusion "for each TTT, A1 holds almost surely" in place of "almost surely, A1 holds for all TTT". The second sentence of Proposition 4.4 (the asymptotic pseudotrajectory conclusion) is outside this mission.

All hypotheses are satisfiable: U=0U=0U=0, x=0x=0x=0, F=0F=0F=0, Γ=1\Gamma=1Γ=1 and γn=1/n\gamma_n=1/nγn​=1/n satisfy every one of them.

Contributions welcome: a maximal inequality for nonnegative supermartingales in the form needed here, vector subgaussian tail bounds for martingale transforms with deterministic weights (reusable well beyond this mission), lemmas on the step processes and Δ\DeltaΔ (measurability, local integrability, additivity), and the proofs of the milestones.

Selected references

  • M. Benaïm, Dynamics of Stochastic Approximation Algorithms, Séminaire de Probabilités XXXIII, Lecture Notes in Mathematics 1709, Springer, 1999, pp. 1–68. https://doi.org/10.1007/BFb0096509
  • M. Duflo, Random Iterative Models, Applications of Mathematics 34, Springer, 1997.
  • H. J. Kushner and G. G. Yin, Stochastic Approximation Algorithms and Applications, Springer, 1997.
  • M. Benaïm and M. W. Hirsch, Asymptotic pseudotrajectories and chain recurrent flows, with applications, Journal of Dynamics and Differential Equations 8 (1996), 141–176. https://doi.org/10.1007/BF02218617
  • H. Robbins and S. Monro, A stochastic approximation method, Annals of Mathematical Statistics 22 (1951), 400–407. https://doi.org/10.1214/aoms/1177729586
10 thms2 active usersReviewed
Operations ResearchOptimizationProbability+1·Captain: mikedeng1

Dimensioning Large Call Centers II: Asymptotically Optimal Staffing in the Efficiency-Driven RegimeResearch Paper

Why staffing large call centers is a mathematical question

A call center must choose enough servers to limit waiting while paying for every server it staffs. When arrivals are heavy, small changes in the number of servers can change the probability of delay substantially. Borst, Mandelbaum, and Reiman study how to make this choice when the arrival rate grows and the costs of staffing and waiting need not grow at the same rate. Their CWI report treats several regimes within one queueing model. This mission concerns the efficiency-driven regime, where the incremental staffing cost eventually dominates the conditional waiting cost at every fixed positive square-root staffing offset. The resulting rule chooses an offset by optimizing a simpler cost that treats the probability of waiting as one.

The result is useful when the staffing-cost and waiting-cost primitives change with system scale. It says that the simplified choice still attains the optimal total cost asymptotically, even though the actual staffing decision is an integer and the simplified problem uses a real variable. The report states this as Theorem 6.1 on printed page 19, with its interpretation of asymptotic optimality supplied by Corollary 3.3 on printed page 14.

The Erlang-C cost model

Customers arrive at rate λ>0\lambda>0λ>0 and receive exponential service at rate μ>0\mu>0μ>0 per server. The service rate μ\muμ is fixed as λ\lambdaλ grows. For an integer number of servers N>λ/μN>\lambda/\muN>λ/μ, the Erlang-C delay probability π(N,λ/μ)\pi(N,\lambda/\mu)π(N,λ/μ) is the explicit finite-sum expression in Section 2 of the report. A customer who waits has an exponential waiting time with rate Nμ−λN\mu-\lambdaNμ−λ. Let Dλ(t)D_\lambda(t)Dλ​(t) be the cost of a wait of length ttt. It is strictly increasing on t≥0t\ge0t≥0, satisfies Dλ(0)=0D_\lambda(0)=0Dλ​(0)=0, and has finite exponential expectation at every positive rate. The resulting conditional waiting cost is

G(N,λ)=(Nμ−λ)∫0∞Dλ(t)e−(Nμ−λ)t dt.G(N,\lambda)=(N\mu-\lambda)\int_0^\infty D_\lambda(t)e^{-(N\mu-\lambda)t}\,dt.G(N,λ)=(Nμ−λ)∫0∞​Dλ​(t)e−(Nμ−λ)tdt.

The staffing cost F(N)F(N)F(N) is one fixed, convex, strictly increasing function of the server count. Its continuous extension is evaluated at real N>0N>0N>0. Total cost per unit of time at a stable integer level is

C(N,λ)=F(N)+λπ(N,λ/μ)G(N,λ).C(N,\lambda)=F(N)+\lambda\pi(N,\lambda/\mu)G(N,\lambda).C(N,λ)=F(N)+λπ(N,λ/μ)G(N,λ).

Write Nλ∗N^*_\lambdaNλ∗​ for any minimizing stable integer level. Ties are permitted. For a positive real offset xxx, define Nλ(x)=λ/μ+xλ/μN_\lambda(x)=\lambda/\mu+x\sqrt{\lambda/\mu}Nλ​(x)=λ/μ+xλ/μ​, Fλ(x)=F(Nλ(x))−F(λ/μ)F_\lambda(x)=F(N_\lambda(x))-F(\lambda/\mu)Fλ​(x)=F(Nλ​(x))−F(λ/μ), and Gλ(x)=λG(Nλ(x),λ)G_\lambda(x)=\lambda G(N_\lambda(x),\lambda)Gλ​(x)=λG(Nλ​(x),λ). The report extends Erlang-C continuously to πλ(x)\pi_\lambda(x)πλ​(x) and writes the incremental continuous objective as Cλ(x)=Fλ(x)+πλ(x)Gλ(x)C_\lambda(x)=F_\lambda(x)+\pi_\lambda(x)G_\lambda(x)Cλ​(x)=Fλ​(x)+πλ​(x)Gλ​(x). These definitions and the integer-extension identity are from Section 3, printed pages 11–12.

Formalization targets

The report defines the efficiency-driven regime by

for every κ>0,lim⁡λ→∞Fλ(κ)Gλ(κ)=+∞.\text{for every }\kappa>0,\qquad \lim_{\lambda\to\infty}\frac{F_\lambda(\kappa)}{G_\lambda(\kappa)}=+\infty.for every κ>0,λ→∞lim​Gλ​(κ)Fλ​(κ)​=+∞.

For each λ>0\lambda>0λ>0, choose yλ∗>0y^*_\lambda>0yλ∗​>0 to minimize Fλ(y)+Gλ(y)F_\lambda(y)+G_\lambda(y)Fλ​(y)+Gλ​(y) over y>0y>0y>0. Let Sλ(y)S_\lambda(y)Sλ​(y) be the smaller cost of the stable integer levels immediately below and above Nλ(y)N_\lambda(y)Nλ​(y); if the lower one is unstable, use the upper one. The goal, Theorem 6.1 together with Corollary 3.3, is

lim⁡λ→∞Sλ(yλ∗)−F(λ/μ)C(Nλ∗,λ)−F(λ/μ)=1.\lim_{\lambda\to\infty} \frac{S_\lambda(y^*_\lambda)-F(\lambda/\mu)} {C(N^*_\lambda,\lambda)-F(\lambda/\mu)}=1.λ→∞lim​C(Nλ∗​,λ)−F(λ/μ)Sλ​(yλ∗​)−F(λ/μ)​=1.

The milestone path includes the convexity of the conditional waiting cost (Lemma C.1), the agreement of the continuous Erlang-C extension with its integer formula, the two approximation lemmas and their corollary (Lemmas 3.1–3.2 and Corollary 3.3), the convex staffing-cost comparison of equation (13), and all three clauses of the Halfin–Whitt limit in Lemma 4.1. This ordering follows the objects each later statement uses.

What the result gives

The theorem certifies a staffing rule defined by a one-variable surrogate rather than the exact Erlang-C probability in the objective. Its guarantee concerns the incremental total cost above the unavoidable baseline F(λ/μ)F(\lambda/\mu)F(λ/μ), which is the economically relevant quantity when comparing two near-minimal stable staffing levels. The ratio tends to one, so the theorem is stronger than a claim that the two costs merely have the same growth order. The source also presents other regimes with different surrogates; their conclusions are separate targets in this series.

The paper proves the mathematical theorem. This mission asks for a Lean proof of its closed-form model and the surrounding lemmas. The complete development would make the report's approximation framework reusable for later results that combine a continuous queueing approximation, a surrogate minimizer, and integer rounding. It would also expose the exact assumptions needed to pass between real and integer staffing levels. No machine-checked proof of this report's Theorem 6.1 is claimed here.

Where the difficulty lies

The simple objective replaces the delay probability πλ(y)\pi_\lambda(y)πλ​(y) by one. That replacement is accurate near zero offset, but the minimizing offset itself changes with λ\lambdaλ. Pointwise asymptotics at a fixed positive offset do not directly control the value of an objective at its moving minimizer. The proof therefore has to relate the regime assumption to the location of the relevant minimizers before using the Halfin–Whitt limit. Integer rounding introduces another boundary issue: when Nλ(y)N_\lambda(y)Nλ​(y) is just above λ/μ\lambda/\muλ/μ, its floor need not be stable, so evaluating the ordinary Erlang-C formula there would compare the target against a meaningless cost. These difficulties are visible already in the statements of Theorem 6.1 and Lemma 3.2.

Formalization scope and conventions

Lean represents λ\lambdaλ, μ\muμ, offsets, and costs as real numbers; arrival-rate limits use the real filter at +∞+\infty+∞. Staffing counts are natural numbers. The service rate is positive and fixed. A WaitModel packages strict increase and normalization of DλD_\lambdaDλ​ on nonnegative waits together with integrability against every positive exponential rate. This integrability expresses the report's finiteness assumption for GGG and prevents a nonintegrable real integral from silently evaluating to zero. The hypotheses on FFF are convexity and strict increase on positive real staffing levels; FFF does not depend on λ\lambdaλ.

The report asserts that G(N,λ)G(N,\lambda)G(N,λ) diverges as NNN decreases to λ/μ\lambda/\muλ/μ, although the stated assumptions permit bounded increasing waiting penalties for which that assertion fails. The goal therefore includes this explicit divergence hypothesis, which also supports existence of the continuous minimizer used in the report's argument. The integer optimum and the surrogate optimum are functions constrained to be minimizers at every positive arrival rate. They cannot be arbitrary choices that make the conclusion vacuous. The continuous optimum appears only in the framework milestones; it is not a hypothesis of Theorem 6.1.

All formulas are total Lean functions. Their values at λ≤0\lambda\le0λ≤0, unstable integer counts, nonpositive offsets, or invalid parameters to the continuous Erlang-C integral have no queueing interpretation. Every theorem using them constrains its relevant inputs. The definition of SλS_\lambdaSλ​ ignores an unstable floor and uses the stable ceiling. At a positive offset and arrival rate this ceiling is above offered load. The Gaussian density, its cumulative integral, the hazard rate, and the delay function use the explicit formulas of Section 4; the value of the delay function at zero is the continuous extension needed by Lemma 4.1.

The queue's stochastic construction is outside this mission. The formal objects are the report's cost formulas and asymptotic comparisons, not a continuous-time Markov chain. Useful contributions include proofs of the special-function limit, convexity of conditional waiting cost, the integer-extension identity, and the reusable approximation lemmas. The regime condition is the full limit in equation (23); weakening it to an unrelated boundedness condition would change the theorem.

Selected references

  • Sem Borst, Avi Mandelbaum, and Martin I. Reiman, Dimensioning Large Call Centers, CWI Report PNA-R0015, 2000. Report PDF. Theorem 6.1, printed p. 19; Corollary 3.3, printed p. 14; Lemma 4.1, printed p. 15; Lemma C.1, printed p. 40.
22 thms2 active usersReviewed
Operations ResearchOptimizationProbability+1·Captain: mikedeng1

Dimensioning Large Call Centers III: Asymptotically Optimal Staffing in the Quality-Driven RegimeResearch Paper

Motivation

How many agents should a call center staff? Telephone call centers employ millions of people, and staffing is their largest cost, so the question is asked every half hour of every day (Gans, Koole & Mandelbaum, 2003). The classical model is the M/M/N (Erlang-C) queue: calls arrive at rate λ\lambdaλ, service times are exponential with mean 1/μ1/\mu1/μ, and NNN agents serve in parallel. Practitioners use the square-root safety staffing rule N≈λ/μ+yλ/μN \approx \lambda/\mu + y\sqrt{\lambda/\mu}N≈λ/μ+yλ/μ​, which Halfin and Whitt (1981) justified in the regime where the probability of waiting stays bounded away from 000 and 111.

Borst, Mandelbaum and Reiman (CWI Report PNA-R0015, 2000; published as Operations Research 52(1), 2004) asked when such a rule is actually optimal: given a staffing cost and a waiting cost, which staffing level minimizes total cost as the arrival rate grows? They identified three regimes according to how the two costs compare. This mission formalizes their third case, the quality-driven regime, in which waiting is so expensive relative to staffing that the optimal number of agents exceeds the offered load by more than any fixed multiple of its square root.

Setting

Fix a service rate μ>0\mu > 0μ>0. For every arrival rate λ>0\lambda > 0λ>0 a waiting-cost function DλD_\lambdaDλ​ assigns cost Dλ(t)D_\lambda(t)Dλ​(t) to a wait of ttt time units; it satisfies Dλ(0)=0D_\lambda(0) = 0Dλ​(0)=0, is strictly increasing, and t↦Dλ(t)e−θtt \mapsto D_\lambda(t)e^{-\theta t}t↦Dλ​(t)e−θt is integrable on (0,∞)(0,\infty)(0,∞) for every θ>0\theta > 0θ>0. A staffing cost FFF, defined for real N>0N > 0N>0, is convex and strictly increasing.

For an integer N>λ/μN > \lambda/\muN>λ/μ the probability of waiting is the Erlang-C formula

π(N,ν)=νNN!{(1−ν/N)∑n=0N−1νnn!+νNN!}−1,ν=λ/μ,\pi(N,\nu) = \frac{\nu^N}{N!}\Bigl\{(1-\nu/N)\sum_{n=0}^{N-1}\frac{\nu^n}{n!} + \frac{\nu^N}{N!}\Bigr\}^{-1},\qquad \nu = \lambda/\mu,π(N,ν)=N!νN​{(1−ν/N)n=0∑N−1​n!νn​+N!νN​}−1,ν=λ/μ,

the expected waiting cost of a delayed customer is G(N,λ)=(Nμ−λ)∫0∞Dλ(t)e−(Nμ−λ)t dtG(N,\lambda) = (N\mu-\lambda)\int_0^\infty D_\lambda(t)e^{-(N\mu-\lambda)t}\,dtG(N,λ)=(Nμ−λ)∫0∞​Dλ​(t)e−(Nμ−λ)tdt, and the total cost per unit time is C(N,λ)=F(N)+λ π(N,λ/μ) G(N,λ)C(N,\lambda) = F(N) + \lambda\,\pi(N,\lambda/\mu)\,G(N,\lambda)C(N,λ)=F(N)+λπ(N,λ/μ)G(N,λ). An optimal staffing level Nλ∗N^*_\lambdaNλ∗​ minimizes C(⋅,λ)C(\cdot,\lambda)C(⋅,λ) over the integers N>λ/μN > \lambda/\muN>λ/μ.

Write Nλ(x)=λ/μ+xλ/μN_\lambda(x) = \lambda/\mu + x\sqrt{\lambda/\mu}Nλ​(x)=λ/μ+xλ/μ​, and for x>0x > 0x>0 put Fλ(x)=F(Nλ(x))−F(λ/μ)F_\lambda(x) = F(N_\lambda(x)) - F(\lambda/\mu)Fλ​(x)=F(Nλ​(x))−F(λ/μ), Gλ(x)=λG(Nλ(x),λ)G_\lambda(x) = \lambda G(N_\lambda(x),\lambda)Gλ​(x)=λG(Nλ​(x),λ), and πλ(x)=H(Nλ(x),λ/μ)\pi_\lambda(x) = H(N_\lambda(x),\lambda/\mu)πλ​(x)=H(Nλ​(x),λ/μ), where H(M,α)={α∫0∞e−αtt(1+t)M−1dt}−1H(M,\alpha) = \{\alpha\int_0^\infty e^{-\alpha t}t(1+t)^{M-1}dt\}^{-1}H(M,α)={α∫0∞​e−αtt(1+t)M−1dt}−1 extends the Erlang-C formula to real MMM. The normalized cost is Cλ(x)=Fλ(x)+πλ(x)Gλ(x)C_\lambda(x) = F_\lambda(x) + \pi_\lambda(x)G_\lambda(x)Cλ​(x)=Fλ​(x)+πλ​(x)Gλ​(x), and a surrogate cost is C[z;F^,π^,G^]=F^(z)+π^(z)G^(z)C[z;\hat F,\hat\pi,\hat G] = \hat F(z) + \hat\pi(z)\hat G(z)C[z;F^,π^,G^]=F^(z)+π^(z)G^(z). Rounding is measured by Sλ(x)=min⁡{C(⌊Nλ(x)⌋,λ),C(⌈Nλ(x)⌉,λ)}S_\lambda(x) = \min\{C(\lfloor N_\lambda(x)\rfloor,\lambda), C(\lceil N_\lambda(x)\rceil,\lambda)\}Sλ​(x)=min{C(⌊Nλ​(x)⌋,λ),C(⌈Nλ​(x)⌉,λ)}.

Two special functions appear. The Halfin–Whitt delay function is P(x)=1/(1+x/h(−x))P(x) = 1/(1 + x/h(-x))P(x)=1/(1+x/h(−x)) with h=ϕ/(1−Φ)h = \phi/(1-\Phi)h=ϕ/(1−Φ) the standard normal hazard rate. The Stirling-type approximation is

Qλ(x)=exp⁡{Nλ(x)[1−rλ(x)+log⁡rλ(x)]}2πNλ(x) (1−rλ(x)),rλ(x)=λ/μNλ(x).Q_\lambda(x) = \frac{\exp\{N_\lambda(x)[1 - r_\lambda(x) + \log r_\lambda(x)]\}}{\sqrt{2\pi N_\lambda(x)}\,(1-r_\lambda(x))},\qquad r_\lambda(x) = \frac{\lambda/\mu}{N_\lambda(x)}.Qλ​(x)=2πNλ​(x)​(1−rλ​(x))exp{Nλ​(x)[1−rλ​(x)+logrλ​(x)]}​,rλ​(x)=Nλ​(x)λ/μ​.

Asymptotic relations are limits of ratios as λ→∞\lambda\to\inftyλ→∞: aλ≈∞bλa_\lambda \stackrel{\infty}{\approx} b_\lambdaaλ​≈∞bλ​ means aλ/bλ→1a_\lambda/b_\lambda \to 1aλ​/bλ​→1, and aλ≪∞bλa_\lambda \stackrel{\infty}{\ll} b_\lambdaaλ​≪∞​bλ​ means aλ/bλ→0a_\lambda/b_\lambda \to 0aλ​/bλ​→0.

Formalization targets

Goal: Theorem 7.1

Assume the regime is quality-driven, display (27): Fλ(κ)≪∞Gλ(κ)F_\lambda(\kappa) \stackrel{\infty}{\ll} G_\lambda(\kappa)Fλ​(κ)≪∞​Gλ​(κ) for every κ>0\kappa > 0κ>0. Let yλ∗y^*_\lambdayλ∗​ minimize Fλ(y)+Qλ(y)Gλ(y)F_\lambda(y) + Q_\lambda(y)G_\lambda(y)Fλ​(y)+Qλ​(y)Gλ​(y) over y>0y > 0y>0. Then

lim⁡λ→∞Sλ(yλ∗)−F(λ/μ)C(Nλ∗,λ)−F(λ/μ)=1.\lim_{\lambda\to\infty}\frac{S_\lambda(y^*_\lambda) - F(\lambda/\mu)}{C(N^*_\lambda,\lambda) - F(\lambda/\mu)} = 1.λ→∞lim​C(Nλ∗​,λ)−F(λ/μ)Sλ​(yλ∗​)−F(λ/μ)​=1.

The statement fixes no constants and no rate; it asserts only that rounding the surrogate optimum loses a vanishing fraction of the excess cost.

Milestones

In attack order: Lemma C.1 (GλG_\lambdaGλ​ strictly convex decreasing); the identity H(N,ν)=π(N,ν)H(N,\nu) = \pi(N,\nu)H(N,ν)=π(N,ν) at integer NNN (Section 3, p. 12); Lemma 3.1 and Lemma 3.2; Corollary 3.3 (the asymptotic optimality criterion); Lemma B.1 (PPP strictly convex decreasing); display (15); Lemma 4.1 (Halfin and Whitt); and the first statement of Lemma 4.2, πλ(xλ)≈∞Qλ(xλ)\pi_\lambda(x_\lambda) \stackrel{\infty}{\approx} Q_\lambda(x_\lambda)πλ​(xλ​)≈∞Qλ​(xλ​) whenever xλ→∞x_\lambda\to\inftyxλ​→∞.

Significance

Theorem 7.1 completes the paper's picture of optimal staffing. In the rationalized regime the square-root rule with the Halfin–Whitt function PPP is optimal; in the efficiency-driven regime staffing barely exceeds the load; in the quality-driven regime the staffing excess outgrows λ/μ\sqrt{\lambda/\mu}λ/μ​ and PPP must be replaced by the Stirling-type expression QλQ_\lambdaQλ​. The theorem gives a one-dimensional minimization whose solution is asymptotically optimal, which turns a discrete optimization over NNN into a smooth problem, and it marks the boundary of validity of square-root staffing.

The result is proved in the paper; it is not formalized anywhere to our knowledge. A complete development formalizes the Section 3 framework (shared with the other regimes of the same paper), the convexity of GλG_\lambdaGλ​ and of PPP, the Halfin–Whitt limit for the continuous extension πλ\pi_\lambdaπλ​, and the Stirling-type asymptotics of the Erlang-C formula. Each of these is a reusable piece of queueing theory in Lean.

Difficulty

The regime theorem itself is short once the framework is in place; the weight lies in the analytic lemmas. Lemma 4.2 requires uniform asymptotics of πλ\pi_\lambdaπλ​ at a staffing excess xλx_\lambdaxλ​ that may grow at any rate, from barely faster than a constant to faster than λ\sqrt{\lambda}λ​, where neither the central-limit picture of Halfin and Whitt nor a single Stirling expansion covers all cases. Lemma 4.1 concerns the continuous extension πλ\pi_\lambdaπλ​ at non-integer server counts, whereas Halfin and Whitt's theorem is about integer ones. The natural first idea, that the goal follows from Corollary 3.3 by plugging in Lemma 4.2, does not apply directly: Lemma 4.2 only covers staffing excesses that tend to infinity, and nothing in the definition of the true optimum xλ∗x^*_\lambdaxλ∗​ or the surrogate optimum yλ∗y^*_\lambdayλ∗​ says that they do.

Formalization scope

Lean represents λ\lambdaλ as a positive real, and λ→∞\lambda\to\inftyλ→∞ is the filter atTop on R\mathbb{R}R with μ\muμ fixed. The standing assumptions on μ\muμ and DλD_\lambdaDλ​ are the structure WaitModel; FFF is a function argument with hypotheses ConvexOn and StrictMonoOn on (0,∞)(0,\infty)(0,∞). Staffing levels NNN are natural numbers. Minimizers (Nλ∗N^*_\lambdaNλ∗​, xλ∗x^*_\lambdaxλ∗​, zλ∗z^*_\lambdazλ∗​, yλ∗y^*_\lambdayλ∗​) are function arguments with minimality hypotheses at every λ>0\lambda > 0λ>0, so every statement holds for every choice among ties. Liminf and limsup relations are stated through Filter.Frequently, avoiding boundedness side conditions.

The queue itself (Poisson arrivals, waiting-time law) is not formalized: the paper's analysis and all its theorems concern the closed-form cost C(N,λ)C(N,\lambda)C(N,λ) with the Erlang-C formula.

Conventions committed to: (i) the goal adds the hypothesis G(N,λ)→∞G(N,\lambda)\to\inftyG(N,λ)→∞ as N↓λ/μN\downarrow\lambda/\muN↓λ/μ, which the paper asserts on p. 12 to show the continuous optimum exists but which does not follow from its standing assumptions (it holds exactly when DλD_\lambdaDλ​ is unbounded); (ii) in SλS_\lambdaSλ​ the floor term is omitted when ⌊Nλ(x)⌋≤λ/μ\lfloor N_\lambda(x)\rfloor \le \lambda/\mu⌊Nλ​(x)⌋≤λ/μ, since the cost is undefined at unstable levels; (iii) the integrability of Dλ(t)e−θtD_\lambda(t)e^{-\theta t}Dλ​(t)e−θt is explicit, because a Lean integral of a non-integrable function is 000; (iv) P(0)=1P(0) = 1P(0)=1, the value of formula (11) at 000; (v) display (15) is stated for b>0b > 0b>0, since the ratio aλ/ba_\lambda/baλ​/b is undefined at b=0b = 0b=0. The instance μ=1\mu = 1μ=1, F(N)=cNF(N) = cNF(N)=cN, Dλ(t)=aλ tD_\lambda(t) = a\sqrt{\lambda}\,tDλ​(t)=aλ​t (Section 9) satisfies every hypothesis of the goal, so the goal is not vacuous; taking πλ\pi_\lambdaπλ​ or GλG_\lambdaGλ​ at Lean default values is ruled out by these explicit domain conditions.

Only the first statement of Lemma 4.2 is a milestone: the second, πλ(xλ)≈Q(xλ)\pi_\lambda(x_\lambda)\approx Q(x_\lambda)πλ​(xλ​)≈Q(xλ​) under xλ≤sup⁡λ1/6x_\lambda \stackrel{\sup}{\le} \lambda^{1/6}xλ​≤sup​λ1/6, fails as printed at xλ=λ1/6x_\lambda = \lambda^{1/6}xλ​=λ1/6. Contributions on the Erlang-C asymptotics, the normal hazard rate, and Laplace transforms of increasing functions are welcome and reusable beyond this mission.

Selected references

  • S. Borst, A. Mandelbaum, M. I. Reiman, Dimensioning Large Call Centers, CWI Report PNA-R0015, 2000 (the version formalized here; every index and page cited in this mission is the report's).
  • S. Borst, A. Mandelbaum, M. I. Reiman, Dimensioning Large Call Centers, Operations Research 52(1):17–34, 2004. https://doi.org/10.1287/opre.1030.0081
  • S. Halfin, W. Whitt, Heavy-Traffic Limits for Queues with Many Exponential Servers, Operations Research 29(3):567–588, 1981. https://doi.org/10.1287/opre.29.3.567
  • N. Gans, G. Koole, A. Mandelbaum, Telephone Call Centers: Tutorial, Review, and Research Prospects, Manufacturing & Service Operations Management 5(2):79–141, 2003. https://doi.org/10.1287/msom.5.2.79.16071
24 thms2 active usersReviewed
Algorithmic Game TheoryControl TheoryOperations Research+1·Captain: mikedeng1

Nonzero-Sum Stochastic Differential Games with Impulse Controls: A Verification Theorem with Applications 2: An Explicit Family of Nash Equilibria for the Linear Impulse GameResearch Paper

Motivation

Impulse control models an agent who acts on a random system through discrete interventions, each with a fixed cost: inventory replenishment, cash management, exchange-rate interventions by a central bank. With two agents whose objectives conflict, the problem becomes a nonzero-sum stochastic differential game with impulse controls. Before the work of Aïd, Basei, Callegaro, Campi and Vargiolu (Math. Oper. Res. 45(1), 2020; preprint arXiv:1605.00039), such games had no general verification theorem and few explicit equilibria; the paper supplies both. Its main application, formalized in this mission, is a game between two central banks with different targets for an exchange rate, a two-player version of the exchange-rate control models of Bertola, Runggaldier and Yasuda and of Cadenillas and Zapatero (references [10], [12] of the paper). The paper proves that this game has an explicit one-parameter family of Nash equilibria of threshold type, with closed-form payoffs.

Setting

The state is a real process. Without interventions it is x+σWsx+\sigma W_sx+σWs​, where WWW is a standard real Brownian motion and σ>0\sigma>0σ>0. Player 1 may shift it up by impulses δ∈Z1=[0,∞[\delta\in Z_1=[0,\infty[δ∈Z1​=[0,∞[, player 2 down by impulses δ∈Z2=]−∞,0]\delta\in Z_2=]-\infty,0]δ∈Z2​=]−∞,0]:

Xs=x+σWs+∑k:τ1,k≤sδ1,k+∑k:τ2,k≤sδ2,k.X_s=x+\sigma W_s+\sum_{k:\tau_{1,k}\le s}\delta_{1,k}+\sum_{k:\tau_{2,k}\le s}\delta_{2,k}.Xs​=x+σWs​+k:τ1,k​≤s∑​δ1,k​+k:τ2,k​≤s∑​δ2,k​.

Player 1 earns the running payoff f1(Xs)=Xs−s1f_1(X_s)=X_s-s_1f1​(Xs​)=Xs​−s1​, player 2 earns f2(Xs)=s2−Xsf_2(X_s)=s_2-X_sf2​(Xs​)=s2​−Xs​, with s1<s2s_1<s_2s1​<s2​. An impulse δ\deltaδ costs its author c+λ∣δ∣c+\lambda|\delta|c+λ∣δ∣ and pays the opponent c~+λ~∣δ∣\tilde c+\tilde\lambda|\delta|c~+λ~∣δ∣. Payoffs are discounted at rate ρ>0\rho>0ρ>0. The standing assumptions are c≥c~≥0c\ge\tilde c\ge0c≥c~≥0, λ≥λ~≥0\lambda\ge\tilde\lambda\ge0λ≥λ~≥0, (c,λ)≠(c~,λ~)(c,\lambda)\ne(\tilde c,\tilde\lambda)(c,λ)=(c~,λ~) and 1−λρ>01-\lambda\rho>01−λρ>0.

A strategy of player iii is a pair φi=(Ci,ξi)\varphi_i=(\mathcal C_i,\xi_i)φi​=(Ci​,ξi​): an open continuation region Ci⊆R\mathcal C_i\subseteq\mathbb RCi​⊆R and a continuous impulse map ξi:R→Zi\xi_i:\mathbb R\to Z_iξi​:R→Zi​. Player iii intervenes when the state leaves Ci\mathcal C_iCi​, applying the impulse ξi(state)\xi_i(\text{state})ξi​(state); player 1 has priority when both want to act. This rule defines the controlled process Xx;φ1,φ2X^{x;\varphi_1,\varphi_2}Xx;φ1​,φ2​ and the interventions (τi,k,δi,k)(\tau_{i,k},\delta_{i,k})(τi,k​,δi,k​) inductively. Player iii's payoff is

Ji(x;φ1,φ2)=Ex[∫0∞e−ρsfi(Xs) ds−∑ke−ρτi,k(c+λ∣δi,k∣)+∑ke−ρτj,k(c~+λ~∣δj,k∣)].J^i(x;\varphi_1,\varphi_2)=\mathbb E_x\Big[\int_0^\infty e^{-\rho s}f_i(X_s)\,ds-\sum_{k}e^{-\rho\tau_{i,k}}(c+\lambda|\delta_{i,k}|)+\sum_{k}e^{-\rho\tau_{j,k}}(\tilde c+\tilde\lambda|\delta_{j,k}|)\Big].Ji(x;φ1​,φ2​)=Ex​[∫0∞​e−ρsfi​(Xs​)ds−k∑​e−ρτi,k​(c+λ∣δi,k​∣)+k∑​e−ρτj,k​(c~+λ~∣δj,k​∣)].

A pair is xxx-admissible, (φ1,φ2)∈Φx(\varphi_1,\varphi_2)\in\Phi_x(φ1​,φ2​)∈Φx​, when these random variables are integrable, sup⁡t∣Xt∣\sup_t|X_t|supt​∣Xt​∣ has all moments, and neither player's interventions accumulate in finite time. A Nash equilibrium is an admissible pair from which no player gains by deviating to any strategy that keeps the pair admissible.

The explicit objects are θ=2ρ/σ2\theta=\sqrt{2\rho/\sigma^2}θ=2ρ/σ2​, η=(1−λρ)/ρ\eta=(1-\lambda\rho)/\rhoη=(1−λρ)/ρ and

F(y)=2y+θc−ηlog⁡η+yη−y,0<y<η.F(y)=2y+\theta c-\eta\log\frac{\eta+y}{\eta-y},\qquad 0<y<\eta .F(y)=2y+θc−ηlogη−yη+y​,0<y<η.

From the zero ξ\xiξ of FFF and a free parameter s~∈R\tilde s\in\mathbb Rs~∈R, the formulas (4.20)–(4.21) give thresholds xˉ1<xˉ2\bar x_1<\bar x_2xˉ1​<xˉ2​, targets x1∗,x2∗∈]xˉ1,xˉ2[x_1^*,x_2^*\in]\bar x_1,\bar x_2[x1∗​,x2∗​∈]xˉ1​,xˉ2​[, coefficients AijA_{ij}Aij​, the functions φi(y)=Ai1eθy+Ai2e−θy±(y−si)/ρ\varphi_i(y)=A_{i1}e^{\theta y}+A_{i2}e^{-\theta y}\pm(y-s_i)/\rhoφi​(y)=Ai1​eθy+Ai2​e−θy±(y−si​)/ρ and the piecewise candidates V~1,V~2\tilde V_1,\tilde V_2V~1​,V~2​ of (4.6). These are linear outside ]xˉ1,xˉ2[]\bar x_1,\bar x_2[]xˉ1​,xˉ2​[ and equal φi\varphi_iφi​ inside.

Formalization targets

Goal: Proposition 4.7

For every s~∈R\tilde s\in\mathbb Rs~∈R and every initial state x∈Rx\in\mathbb Rx∈R, the threshold strategies

φ1∗=(]xˉ1,+∞[, y↦max⁡(x1∗−y,0)),φ2∗=(]−∞,xˉ2[, y↦min⁡(x2∗−y,0))\varphi_1^*=\big(]\bar x_1,+\infty[,\ y\mapsto\max(x_1^*-y,0)\big),\qquad \varphi_2^*=\big(]-\infty,\bar x_2[,\ y\mapsto\min(x_2^*-y,0)\big)φ1∗​=(]xˉ1​,+∞[, y↦max(x1∗​−y,0)),φ2∗​=(]−∞,xˉ2​[, y↦min(x2∗​−y,0))

form an xxx-admissible Nash equilibrium, and

J1(x;φ1∗,φ2∗)=V~1(x),J2(x;φ1∗,φ2∗)=V~2(x).J^1(x;\varphi_1^*,\varphi_2^*)=\tilde V_1(x),\qquad J^2(x;\varphi_1^*,\varphi_2^*)=\tilde V_2(x).J1(x;φ1∗​,φ2∗​)=V~1​(x),J2(x;φ1∗​,φ2∗​)=V~2​(x).

Milestones

  1. (4.17). FFF has a unique zero ξ∈(0,η)\xi\in(0,\eta)ξ∈(0,η).
  2. Proposition 4.2. For every s~\tilde ss~, the explicit 8-uple (4.20) satisfies the order conditions (4.7) and the optimality and pasting conditions (4.8)–(4.9). Moreover, φ2′′\varphi_2''φ2′′​ changes sign exactly once in ]x2∗,xˉ2[]x_2^*,\bar x_2[]x2∗​,xˉ2​[.
  3. Lemma 4.6. The impulses (4.23) maximise δ↦V~i(x+δ)−c−λ∣δ∣\delta\mapsto\tilde V_i(x+\delta)-c-\lambda|\delta|δ↦V~i​(x+δ)−c−λ∣δ∣, and the intervention operators satisfy (4.24): {M1V~1−V~1<0}=]xˉ1,∞[\{\mathcal M_1\tilde V_1-\tilde V_1<0\}=]\bar x_1,\infty[{M1​V~1​−V~1​<0}=]xˉ1​,∞[ and {M2V~2−V~2<0}=]−∞,xˉ2[\{\mathcal M_2\tilde V_2-\tilde V_2<0\}=]-\infty,\bar x_2[{M2​V~2​−V~2​<0}=]−∞,xˉ2​[.
  4. Condition (v). The equilibrium pair is xxx-admissible for every xxx. This includes the integrability (4.26) of the discounted intervention costs.

Milestones 1–3 are deterministic real analysis; milestone 4 and the goal are probabilistic.

Significance

The result gives explicit equilibria, with explicit thresholds and payoffs, for a nonzero-sum stochastic game with impulse controls. Such games rarely have closed-form solutions. The equilibria form a continuum indexed by s~\tilde ss~: equilibrium payoffs are not unique, and every equilibrium in the family is a translate of a fixed interval policy. The explicit formulas also support the comparative statics of Section 4.4, where the continuation region widens as the fixed cost grows.

Proposition 4.7 is proved in the paper by applying its verification theorem (Theorem 3.3) to V~1,V~2\tilde V_1,\tilde V_2V~1​,V~2​. The admissibility estimate (4.26) is written out only for initial states x∈{x1∗,x2∗}x\in\{x_1^*,x_2^*\}x∈{x1∗​,x2∗​}; the general case is said to be similar. As far as is known, none of these results has a machine-checked proof. A formal proof would check every regularity, pasting and admissibility condition. It would also yield a reusable pathwise construction of impulse-controlled Brownian motion.

Difficulty

The deterministic milestones need careful algebra with nested logarithms and square roots. Proving Lemma 4.6 requires the global shape of V~i(y)±λy\tilde V_i(y)\pm\lambda yV~i​(y)±λy, which needs the sign pattern of φ2′′\varphi_2''φ2′′​ from Proposition 4.2, not only local conditions at the thresholds.

The goal is harder. The Nash inequality must hold against every admissible deviation: any open continuation region and any continuous impulse map, not only threshold strategies. Comparing payoffs therefore needs a verification argument, namely Itô's formula for a function that is C2C^2C2 only piecewise and C1C^1C1 across the thresholds, applied along a process with an unbounded number of jumps, plus a localisation that uses the moment condition (2.8). Admissibility needs a renewal-type bound on the discounted number of interventions, built from i.i.d. exit times of Brownian motion from an interval.

Formalization scope

The Lean development lives in the namespace ImpulseGames.LinearGame.

  • The constants and standing assumptions are a structure Model with the proposition Model.Standing. The added hypothesis c>0c>0c>0 appears in every statement that uses ξ\xiξ: with c=0c=0c=0, which the standing assumptions allow, FFF has no zero in (0,η)(0,\eta)(0,η) and the family (4.20) does not exist.
  • The zero ξ\xiξ is a parameter, constrained by ξ∈(0,η)\xi\in(0,\eta)ξ∈(0,η) and F(ξ)=0F(\xi)=0F(ξ)=0. Milestone 1 shows that exactly one such ξ\xiξ exists.
  • The Brownian motion is Mathlib's IsBrownianReal W P on a probability space, with time in R≥0\mathbb R_{\ge0}R≥0​. Definition 2.2 is formalized pathwise: exit times inf⁡{s>τ~k−1:X~sk−1∉Ci}\inf\{s>\tilde\tau_{k-1}:\tilde X^{k-1}_s\notin\mathcal C_i\}inf{s>τ~k−1​:X~sk−1​∈/Ci​} in [0,∞][0,\infty][0,∞] with inf⁡∅=∞\inf\emptyset=\inftyinf∅=∞, the tie rule favouring player 1, and e−ρ⋅∞=0e^{-\rho\cdot\infty}=0e−ρ⋅∞=0 for the tail of each impulse control. No SDE or stochastic integral appears in any statement; the uncontrolled dynamics are ζ+σ(Ws−Wt)\zeta+\sigma(W_s-W_t)ζ+σ(Ws​−Wt​).
  • Payoffs are Bochner expectations. Φx\Phi_xΦx​ requires every random variable of (2.7) to be integrable, so no deviation can obtain the default value 000 of a non-integrable expectation. The moment condition (2.8) is stated with an extended-real supremum, and (2.9) is read almost surely.
  • The Nash condition quantifies over all strategies of Definition 2.1. A formalization that restricts deviations to threshold strategies would be a different, weaker theorem and is excluded. Likewise, the equilibrium payoffs V~i\tilde V_iV~i​ are the explicit formulas (4.6), never defined as "the equilibrium value".

Two slips of the page are corrected. First, the paper's impulse maps ξi∗(y)=xi∗−y\xi_i^*(y)=x_i^*-yξi∗​(y)=xi∗​−y are not ZiZ_iZi​-valued on all of R\mathbb RR; they are replaced by max⁡(x1∗−y,0)\max(x_1^*-y,0)max(x1∗​−y,0) and min⁡(x2∗−y,0)\min(x_2^*-y,0)min(x2∗​−y,0), which agree with them wherever each player acts. Second, in Lemma 4.6 the maximiser (4.23) is not unique when λ=λ~\lambda=\tilde\lambdaλ=λ~, in the opponent's intervention region (all impulses tie there). The statement asserts maximality everywhere and uniqueness outside that region.

A complete development needs: elementary real analysis for milestones 1–3; a pathwise theory of piecewise-defined processes; an Itô formula for Brownian motion with C1C^1C1, piecewise-C2C^2C2 functions; and exit-time estimates for Brownian motion. The last two are reusable well beyond this mission. Proofs of the deterministic milestones are welcome independently of the stochastic part.

Selected references

  • R. Aïd, M. Basei, G. Callegaro, L. Campi, T. Vargiolu, Nonzero-sum stochastic differential games with impulse controls: a verification theorem with applications, Mathematics of Operations Research 45(1), 2020. https://doi.org/10.1287/moor.2019.0989 (accepted manuscript: arXiv:1605.00039v4, https://arxiv.org/abs/1605.00039)
  • G. Bertola, W. J. Runggaldier, K. Yasuda, On classical and restricted impulse stochastic control for the exchange rate, Applied Mathematics and Optimization 74(2), 423–454, 2016.
  • A. Cadenillas, F. Zapatero, Classical and impulse stochastic control of the exchange rate using interest rates and reserves, Mathematical Finance 10(2), 141–156, 2000.
  • B. Øksendal, A. Sulem, Applied Stochastic Control of Jump Diffusions, 2nd ed., Springer, 2007.
9 thms2 active usersReviewed
Operations ResearchOptimizationProbability+1·Captain: mikedeng1

Dimensioning Large Call Centers IV: Asymptotically Optimal Staffing under a Waiting-Cost ConstraintResearch Paper

Motivation

A call center has to decide how many agents to staff. In practice the decision is often posed as a service-level constraint rather than a cost trade-off: use the fewest agents for which the expected waiting cost, or the fraction of customers who wait, stays below a target. Borst, Mandelbaum and Reiman (CWI Report PNA-R0015, 2000; journal version in Operations Research 52(1), 2004, doi:10.1287/opre.1030.0081) treat this constraint problem in Section 8 of their paper, alongside the cost-minimization problem of Sections 5–7, and show that a simple square-root staffing rule solves it asymptotically as the arrival rate grows.

The rule matters because it is what practitioners use. Under the classical Erlang-C model, the exact optimum requires evaluating the Erlang-C formula over many staffing levels. The asymptotic rule replaces this with a single equation in the Halfin–Whitt function PPP: when the target is a delay probability ε\varepsilonε (Example 8.5 of the paper), it reduces to staffing λ/μ+P−1(ε)λ/μ\lambda/\mu + P^{-1}(\varepsilon)\sqrt{\lambda/\mu}λ/μ+P−1(ε)λ/μ​ servers.

Timeline. Erlang's formula for the M/M/N delay probability dates from 1917. Halfin and Whitt (Operations Research 29, 1981) identified the limit P(x)P(x)P(x) of the delay probability under square-root staffing N=λ/μ+xλ/μN = \lambda/\mu + x\sqrt{\lambda/\mu}N=λ/μ+xλ/μ​ with integer NNN. Jagers and Van Doorn (Operations Research Letters 5, 1986; SIAM Review 33, 1991) studied the continued Erlang loss and delay functions at non-integer numbers of servers, including their convexity, which is what lets the staffing problem be relaxed to a continuous one. Borst, Mandelbaum and Reiman (2000/2004) used these to prove asymptotic optimality of square-root rules for both the cost and the constraint formulations.

Setting

Customers arrive at rate λ\lambdaλ to NNN identical servers, each with service rate μ>0\mu > 0μ>0; μ\muμ is fixed while λ→∞\lambda \to \inftyλ→∞. Stability requires N>λ/μN > \lambda/\muN>λ/μ. A customer who waits ttt time units costs Dλ(t)D_\lambda(t)Dλ​(t), where Dλ(0)=0D_\lambda(0) = 0Dλ​(0)=0, DλD_\lambdaDλ​ is strictly increasing on [0,∞)[0,\infty)[0,∞) and ∫0∞Dλ(t)e−θt dt<∞\int_0^\infty D_\lambda(t)e^{-\theta t}\,dt < \infty∫0∞​Dλ​(t)e−θtdt<∞ for all θ>0\theta > 0θ>0.

The Erlang-C probability of waiting is

π(N,ν)=νNN!{(1−ν/N)∑n=0N−1νnn!+νNN!}−1,\pi(N,\nu) = \frac{\nu^N}{N!}\Big\{(1-\nu/N)\sum_{n=0}^{N-1}\frac{\nu^n}{n!} + \frac{\nu^N}{N!}\Big\}^{-1},π(N,ν)=N!νN​{(1−ν/N)n=0∑N−1​n!νn​+N!νN​}−1,

and the conditional waiting cost is G(N,λ)=(Nμ−λ)∫0∞Dλ(t)e−(Nμ−λ)t dtG(N,\lambda) = (N\mu-\lambda)\int_0^\infty D_\lambda(t)e^{-(N\mu-\lambda)t}\,dtG(N,λ)=(Nμ−λ)∫0∞​Dλ​(t)e−(Nμ−λ)tdt. The waiting cost per unit time with NNN servers is

K(N,λ)=λ π(N,λ/μ) G(N,λ).K(N,\lambda) = \lambda\,\pi(N,\lambda/\mu)\,G(N,\lambda).K(N,λ)=λπ(N,λ/μ)G(N,λ).

Given a target Mλ>0M_\lambda > 0Mλ​>0, the optimal staffing level is the least integer N>λ/μN > \lambda/\muN>λ/μ with K(N,λ)≤MλK(N,\lambda) \le M_\lambdaK(N,λ)≤Mλ​; call it Nλ∗N^*_\lambdaNλ∗​.

In the continuous parametrization Nλ(x)=λ/μ+xλ/μN_\lambda(x) = \lambda/\mu + x\sqrt{\lambda/\mu}Nλ​(x)=λ/μ+xλ/μ​, define Gλ(x)=λG(Nλ(x),λ)G_\lambda(x) = \lambda G(N_\lambda(x),\lambda)Gλ​(x)=λG(Nλ​(x),λ), the continuous Erlang-C function πλ(x)=H(Nλ(x),λ/μ)\pi_\lambda(x) = H(N_\lambda(x),\lambda/\mu)πλ​(x)=H(Nλ​(x),λ/μ) with H(M,α)={α∫0∞e−αtt(1+t)M−1dt}−1H(M,\alpha) = \{\alpha\int_0^\infty e^{-\alpha t}t(1+t)^{M-1}dt\}^{-1}H(M,α)={α∫0∞​e−αtt(1+t)M−1dt}−1, and Kλ(x)=πλ(x)Gλ(x)K_\lambda(x) = \pi_\lambda(x)G_\lambda(x)Kλ​(x)=πλ​(x)Gλ​(x). The Halfin–Whitt function is P(x)=1/(1+x/h(−x))P(x) = 1/(1 + x/h(-x))P(x)=1/(1+x/h(−x)) with h=ϕ/(1−Φ)h = \phi/(1-\Phi)h=ϕ/(1−Φ) the standard normal hazard rate. A staffing function xλ>0x_\lambda > 0xλ​>0 is judged by the rounding gap

Tλ(x)=min⁡{∣K(⌊Nλ(x)⌋,λ)−Mλ∣, ∣K(⌈Nλ(x)⌉,λ)−Mλ∣, ∣K(⌈Nλ(x)⌉,λ)−K(Nλ∗,λ)∣}.T_\lambda(x) = \min\big\{|K(\lfloor N_\lambda(x)\rfloor,\lambda) - M_\lambda|,\ |K(\lceil N_\lambda(x)\rceil,\lambda) - M_\lambda|,\ |K(\lceil N_\lambda(x)\rceil,\lambda) - K(N^*_\lambda,\lambda)|\big\}.Tλ​(x)=min{∣K(⌊Nλ​(x)⌋,λ)−Mλ​∣, ∣K(⌈Nλ​(x)⌉,λ)−Mλ​∣, ∣K(⌈Nλ​(x)⌉,λ)−K(Nλ∗​,λ)∣}.

It is asymptotically optimal when Tλ(xλ)/Mλ→0T_\lambda(x_\lambda)/M_\lambda \to 0Tλ​(xλ​)/Mλ​→0 as λ→∞\lambda\to\inftyλ→∞.

Formalization targets

Goal: Theorem 8.2 (rationalized regime)

Suppose that for some κ>0\kappa > 0κ>0 and γ∈(0,∞)\gamma \in (0,\infty)γ∈(0,∞), Gλ(κ)/Mλ→γG_\lambda(\kappa)/M_\lambda \to \gammaGλ​(κ)/Mλ​→γ, i.e. the waiting cost is comparable to the target. Let yλ∗>0y^*_\lambda > 0yλ∗​>0 solve P(y)Gλ(y)=MλP(y)G_\lambda(y) = M_\lambdaP(y)Gλ​(y)=Mλ​. Then

lim⁡λ→∞Tλ(yλ∗)Mλ=0.\lim_{\lambda\to\infty}\frac{T_\lambda(y^*_\lambda)}{M_\lambda} = 0.λ→∞lim​Mλ​Tλ​(yλ∗​)​=0.

Supporting milestones

  • Lemma C.1: GλG_\lambdaGλ​ is strictly convex and decreasing on (0,∞)(0,\infty)(0,∞).
  • Section 3: πλ(x)=π(Nλ(x),λ/μ)\pi_\lambda(x) = \pi(N_\lambda(x),\lambda/\mu)πλ​(x)=π(Nλ​(x),λ/μ) when Nλ(x)N_\lambda(x)Nλ​(x) is an integer.
  • Lemma 8.1: if zλ∗>0z^*_\lambda > 0zλ∗​>0 solves π^λ(z)G^λ(z)=Mλ\hat\pi_\lambda(z)\hat G_\lambda(z) = M_\lambdaπ^λ​(z)G^λ​(z)=Mλ​ and Kλ(zλ∗)/(π^λG^λ)(zλ∗)→1K_\lambda(z^*_\lambda)/(\hat\pi_\lambda\hat G_\lambda)(z^*_\lambda) \to 1Kλ​(zλ∗​)/(π^λ​G^λ​)(zλ∗​)→1, then Tλ(zλ∗)/Mλ→0T_\lambda(z^*_\lambda)/M_\lambda \to 0Tλ​(zλ∗​)/Mλ​→0.
  • Lemma B.1: PPP is strictly convex and decreasing on (0,∞)(0,\infty)(0,∞).
  • Eq. (17): lim sup⁡aλ/b=∞\limsup a_\lambda/b = \inftylimsupaλ​/b=∞ implies lim inf⁡P(aλ)/P(b)=0\liminf P(a_\lambda)/P(b) = 0liminfP(aλ​)/P(b)=0 and lim inf⁡πλ(aλ)/πλ(b)=0\liminf \pi_\lambda(a_\lambda)/\pi_\lambda(b) = 0liminfπλ​(aλ​)/πλ​(b)=0.
  • Lemma 4.1 (Halfin–Whitt): for bounded xλ>0x_\lambda > 0xλ​>0, πλ(xλ)/P(xλ)→1\pi_\lambda(x_\lambda)/P(x_\lambda) \to 1πλ​(xλ​)/P(xλ​)→1; with xλ→xx_\lambda \to xxλ​→x, πλ(xλ)/P(x)→1\pi_\lambda(x_\lambda)/P(x)\to 1πλ​(xλ​)/P(x)→1.

Further target: Theorem 8.6 (efficiency-driven regime)

If Gλ(κ)/Mλ→0G_\lambda(\kappa)/M_\lambda \to 0Gλ​(κ)/Mλ​→0 for every κ>0\kappa > 0κ>0 and yλ∗>0y^*_\lambda > 0yλ∗​>0 solves Gλ(y)=MλG_\lambda(y) = M_\lambdaGλ​(y)=Mλ​, then Tλ(yλ∗)/Mλ→0T_\lambda(y^*_\lambda)/M_\lambda \to 0Tλ​(yλ∗​)/Mλ​→0.

Significance

The theorem certifies the staffing rule used in workforce-management practice: the excess staffing is determined by one scalar equation involving the Gaussian function PPP and the scaled waiting cost, and rounding the resulting staffing level misses the constraint by a vanishing fraction of the target. Lemma 8.1 is a reusable framework: any approximation π^λG^λ\hat\pi_\lambda\hat G_\lambdaπ^λ​G^λ​ that is asymptotically exact at the proposed staffing level yields an asymptotically optimal rule, and the paper instantiates it in three regimes (Theorems 8.2, 8.6, 8.9).

The results are proved on paper. To the best of current knowledge none of them, nor the Halfin–Whitt limit for the continuous Erlang-C extension, has a machine-checked proof. A formalization would produce the first verified heavy-traffic limit of the Erlang-C delay probability, a verified continuous Erlang-C extension with its integer identity, and the convexity facts about PPP and GλG_\lambdaGλ​ that many staffing papers cite without proof.

Difficulty

The obvious argument is to quote Halfin and Whitt: the delay probability converges to P(x)P(x)P(x) under square-root staffing, so PPP can replace the Erlang-C formula. That limit, as published in 1981, is about integer server counts along sequences with a convergent excess-staffing parameter. The paper needs it for the continuous function HHH at non-integer server counts and for staffing functions that are merely bounded, and it also needs the identity H(N,ν)=π(N,ν)H(N,\nu) = \pi(N,\nu)H(N,ν)=π(N,ν) at integers and the monotonicity of πλ\pi_\lambdaπλ​ in xxx, both cited from Jagers and Van Doorn rather than proved. None of these is in Mathlib. A second obstacle is that the staffing function yλ∗y^*_\lambdayλ∗​ is defined only implicitly by an equation involving GλG_\lambdaGλ​, which depends on the arbitrary cost functions DλD_\lambdaDλ​; nothing a priori prevents it from escaping to infinity, outside the range where the Halfin–Whitt approximation applies. Finally, TλT_\lambdaTλ​ compares integer-level costs given by the Erlang-C formula with a continuous approximation, so both representations of the delay probability are in play at once.

Formalization scope

The queue itself is not formalized: there is no Markov chain and no waiting-time distribution. Every statement is about the closed-form waiting cost K(N,λ)K(N,\lambda)K(N,λ) with π\piπ given by the Erlang-C formula, exactly as the paper's analysis is. Conventions, all in the namespace DimCallCenters.Constraint:

  • lam : ℝ is the arrival rate (λ is a Lean keyword); limits are Filter.atTop in lam, with μ fixed. Objects indexed by λ (MλM_\lambdaMλ​, Nλ∗N^*_\lambdaNλ∗​, yλ∗y^*_\lambdayλ∗​) are functions of lam constrained only for lam > 0.
  • WaitModel packages μ > 0 and DλD_\lambdaDλ​ with Dλ(0)=0D_\lambda(0) = 0Dλ​(0)=0, strict monotonicity on [0,∞)[0,\infty)[0,∞), and integrability of Dλ(t)e−θtD_\lambda(t)e^{-\theta t}Dλ​(t)e−θt on (0,∞)(0,\infty)(0,∞) for θ > 0 (the paper's finiteness of GGG; integrability is required because Lean's integral of a non-integrable function is 0).
  • Nλ∗N^*_\lambdaNλ∗​ is a function Nstar : ℝ → ℕ given with its two defining properties (feasible; below every feasible integer level above λ/μ). yλ∗y^*_\lambdayλ∗​ and zλ∗z^*_\lambdazλ∗​ are any positive solutions of their equations; existence and uniqueness are not hypotheses.
  • In TλT_\lambdaTλ​ the round-down term is dropped when ⌊Nλ(x)⌋≤λ/μ\lfloor N_\lambda(x)\rfloor \le \lambda/\mu⌊Nλ​(x)⌋≤λ/μ (an unstable level where KKK is undefined). This can only enlarge TλT_\lambdaTλ​.
  • Asymptotic relations are limits of ratios. lim sup⁡=∞\limsup = \inftylimsup=∞ and lim inf⁡=0\liminf = 0liminf=0 are stated with ∃ᶠ ("frequently"), lim sup⁡<∞\limsup < \inftylimsup<∞ as eventual boundedness.
  • PPP is defined through explicit ϕ\phiϕ, Φ\PhiΦ, hhh; the formula also gives P(0)=1P(0) = 1P(0)=1, used in Lemma 4.1(2) at x=0x = 0x=0.
  • No hypothesis lim⁡N↓λ/μG(N,λ)=∞\lim_{N\downarrow\lambda/\mu}G(N,\lambda) = \inftylimN↓λ/μ​G(N,λ)=∞ is added: it is not needed for the statements here.

A trivializing formalization is ruled out: TλT_\lambdaTλ​ keeps all of the paper's terms and is never replaced by a smaller quantity, and the hypotheses are jointly satisfiable — Dλ(t)=aλ/μ tD_\lambda(t) = a\sqrt{\lambda/\mu}\,tDλ​(t)=aλ/μ​t with Mλ=MλM_\lambda = M\lambdaMλ​=Mλ satisfies (33) for every κ\kappaκ with γ=a/(μκM)\gamma = a/(\mu\kappa M)γ=a/(μκM).

Infrastructure needed: the continuous Erlang-C function and its integer identity; the Halfin–Whitt limit (a Gaussian approximation of Poisson/gamma tails); calculus facts about the normal hazard rate. These are reusable beyond this mission, notably by the sibling missions on the cost-minimization problem. Example 8.5 (delay-probability target with Dλ=1t>0D_\lambda = 1_{t>0}Dλ​=1t>0​) motivates the rule but violates the strict monotonicity of DλD_\lambdaDλ​, so it is not an instance of the theorem as stated. Contributions on any milestone, and on Theorem 8.9 (quality-driven regime, which needs Lemma 4.2), are welcome.

Selected references

  • S. Borst, A. Mandelbaum, M. I. Reiman, Dimensioning Large Call Centers, CWI Report PNA-R0015, 2000; Operations Research 52(1):17–34, 2004. https://doi.org/10.1287/opre.1030.0081
  • S. Halfin, W. Whitt, Heavy-Traffic Limits for Queues with Many Exponential Servers, Operations Research 29(3):567–588, 1981. https://doi.org/10.1287/opre.29.3.567
  • A. A. Jagers, E. A. Van Doorn, On the Continued Erlang Loss Function, Operations Research Letters 5:43–46, 1986.
  • A. A. Jagers, E. A. Van Doorn, Convexity of Functions which are Generalizations of the Erlang Loss Function and the Erlang Delay Function, SIAM Review 33:281–282, 1991.
18 thms2 active usersReviewed
Markov ChainOperations ResearchProbability+1·Captain: mikedeng1

On the Stochastic Matrices Associated with Certain Queuing Processes 1: The M/G/1 Imbedded Chain Is Ergodic iff ρ < 1 and Recurrent iff ρ ≤ 1Research Paper

Motivation

Many queues observed at well-chosen instants are Markov chains on the nonnegative integers. For the single-server queue with Poisson arrivals and general service times (M/G/1), D. G. Kendall showed in 1951 that the number of customers left behind at successive departure epochs is such a chain, the imbedded Markov chain (Kendall 1951; Kendall 1953). Whether the queue settles into a steady state, keeps returning to empty without settling, or grows without bound is then a question about this chain: is it ergodic, null recurrent, or transient?

F. G. Foster's 1953 paper (doi:10.1214/aoms/1177728976) answers this question by first proving general criteria for an irreducible chain on {0,1,2,… }\{0,1,2,\dots\}{0,1,2,…}, stated as solvability conditions for linear inequalities in the transition matrix, and then applying them to the M/G/1 and GI/M/1 chains. Theorem 2 of the paper is the drift condition now known as Foster's criterion, the starting point of the Lyapunov-function method for the stability of queues and stochastic networks (Meyn and Tweedie 2009). This mission is the M/G/1 half of the paper.

Timeline:

  • 1951–1953, Kendall. Introduces the imbedded chains of M/G/1 and GI/M/1 and obtains most of their classification by direct methods.
  • 1953, Foster. Derives the classification from general criteria: Theorem 2 (ergodicity), Theorems 4–6 (transience and recurrence).
  • 1950s onward. The criteria become the standard tools (Feller's text; later the drift conditions of Meyn and Tweedie).

Setting

A Markov chain on the states {0,1,2,… }\{0,1,2,\dots\}{0,1,2,…} is given by a transition matrix P=[pij]P=[p_{ij}]P=[pij​]: pij≥0p_{ij}\ge0pij​≥0 and ∑jpij=1\sum_j p_{ij}=1∑j​pij​=1 for every row iii. Write fij(n)f_{ij}^{(n)}fij(n)​ for the probability that the chain started in iii first reaches jjj (for i=ji=ji=j, first returns to jjj) at step n≥1n\ge1n≥1. The chain is irreducible if every state can be reached from every other, and aperiodic if for every state the return times have greatest common divisor 111. A state jjj is recurrent if fjj=∑nfjj(n)=1f_{jj}=\sum_n f_{jj}^{(n)}=1fjj​=∑n​fjj(n)​=1 and transient if fjj<1f_{jj}<1fjj​<1; a recurrent state is ergodic (positive recurrent, "recurrent-nonnull") if in addition its mean recurrence time ∑nnfjj(n)\sum_n n f_{jj}^{(n)}∑n​nfjj(n)​ is finite. The mean first-passage time from iii to jjj is μij=∑n≥1nfij(n)∈[0,∞]\mu_{ij}=\sum_{n\ge1} n f_{ij}^{(n)}\in[0,\infty]μij​=∑n≥1​nfij(n)​∈[0,∞].

The M/G/1 matrix is built from a sequence k0,k1,…k_0,k_1,\dotsk0​,k1​,… of positive numbers summing to one (knk_nkn​ is the probability of nnn arrivals during one service):

[pij]=[k0k1k2⋯k0k1k2⋯0k0k1⋯00k0⋯⋮⋮⋮],[p_{ij}] = \begin{bmatrix} k_0 & k_1 & k_2 & \cdots \\ k_0 & k_1 & k_2 & \cdots \\ 0 & k_0 & k_1 & \cdots \\ 0 & 0 & k_0 & \cdots \\ \vdots & \vdots & \vdots & \end{bmatrix},[pij​]=​k0​k0​00⋮​k1​k1​k0​0⋮​k2​k2​k1​k0​⋮​⋯⋯⋯⋯​​,

that is, p0j=kjp_{0j}=k_jp0j​=kj​ and, for i≥1i\ge1i≥1, pij=kj−i+1p_{ij}=k_{j-i+1}pij​=kj−i+1​ when j≥i−1j\ge i-1j≥i−1 and 000 otherwise. The traffic intensity is

ρ=∑n=1∞n kn∈[0,∞],\rho=\sum_{n=1}^{\infty}n\,k_n\in[0,\infty],ρ=n=1∑∞​nkn​∈[0,∞],

the mean number of arrivals per service.

Formalization targets

Goal: the M/G/1 classification (§3, p. 358)

the chain is ergodic  ⟺  ρ<1,the chain is recurrent  ⟺  ρ≤1.\text{the chain is ergodic}\iff\rho<1,\qquad\text{the chain is recurrent}\iff\rho\le1 .the chain is ergodic⟺ρ<1,the chain is recurrent⟺ρ≤1.

The goal leaves kkk arbitrary apart from positivity and normalization; in particular ρ=∞\rho=\inftyρ=∞ is allowed and falls in the transient case.

Milestones (the paper's general theorems and the step of §3 they feed)

  1. Theorem 2 (drift criterion): a nonnegative solution of ∑jpijyj≤yi−1\sum_j p_{ij}y_j\le y_i-1∑j​pij​yj​≤yi​−1 (i≠0i\ne0i=0) with ∑jp0jyj<∞\sum_j p_{0j}y_j<\infty∑j​p0j​yj​<∞ makes the system ergodic. Already posed on the platform and referenced here.
  2. Theorem 3: in an ergodic system the mean first-passage times dj=μj0d_j=\mu_{j0}dj​=μj0​ are finite and satisfy ∑j≥1pijdj=di−1\sum_{j\ge1}p_{ij}d_j=d_i-1∑j≥1​pij​dj​=di​−1 (i≠0i\ne0i=0), ∑j≥1p0jdj<∞\sum_{j\ge1}p_{0j}d_j<\infty∑j≥1​p0j​dj​<∞.
  3. §3 display: for the ergodic M/G/1 chain, μi,i−1=μ10\mu_{i,i-1}=\mu_{10}μi,i−1​=μ10​ and μi0=iμ10\mu_{i0}=i\mu_{10}μi0​=iμ10​ (i≠0i\ne0i=0).
  4. Theorem 5: a solution of ∑jpijyj≤yi\sum_j p_{ij}y_j\le y_i∑j​pij​yj​≤yi​ (i≠0i\ne0i=0) with yi→∞y_i\to\inftyyi​→∞ makes the system recurrent.
  5. Theorem 7: for a probability distribution {pn}\{p_n\}{pn​} with p0>0p_0>0p0​>0, ∑nznpn=z\sum_n z^np_n=z∑n​znpn​=z has a root in (0,1)(0,1)(0,1) iff ∑n≥1npn>1\sum_{n\ge1}np_n>1∑n≥1​npn​>1.
  6. Theorem 4: the system is transient iff ∑jpijyj=yi\sum_j p_{ij}y_j=y_i∑j​pij​yj​=yi​ (i≠0i\ne0i=0) has a bounded nonconstant solution.

Significance

The result. The classification is the stability theorem for the M/G/1 queue: for ρ<1\rho<1ρ<1 the departure-epoch queue length has a stationary distribution, which is what the Pollaczek–Khinchine formula describes; for ρ=1\rho=1ρ=1 the queue empties infinitely often but has no steady state; for ρ>1\rho>1ρ>1 it grows without bound. The general criteria behind it (Theorems 2, 4, 5) apply to any chain on the nonnegative integers and are reused in the companion GI/M/1 mission and throughout queueing and Markov-chain stability theory.

Formalizing it. All results here are proved on paper (Kendall and Foster, 1951–1953, with Theorems 3 and 7 classical lemmas from Feller). None of them is known to have a machine-checked proof against a Lean development of countable-state Markov chains. The mission produces such proofs on the published discrete-chain vocabulary (transition matrices, first-passage probabilities, return probabilities, positive recurrence), together with the general Foster criteria as reusable theorems. Theorem 2 is already posed as an open platform theorem and is reused here.

Difficulty

The queue-specific part of the argument is short once the general criteria are available; the weight of the mission is in those criteria. They relate qualitative properties of an infinite chain (ergodicity, recurrence, transience) to solvability of infinite systems of linear inequalities, and this needs limit behaviour of the nnn-step probabilities pij(n)p_{ij}^{(n)}pij(n)​ and of hitting probabilities of state 000, none of which follows from finite-state arguments. Two further points resist the naive approach. The converse directions (ergodic ⇒ρ<1\Rightarrow\rho<1⇒ρ<1, recurrent ⇒ρ≤1\Rightarrow\rho\le1⇒ρ≤1) need exact identities for mean first-passage times, not just bounds, and these must be handled in [0,∞][0,\infty][0,∞] because the means may be infinite. And the boundary case ρ=1\rho=1ρ=1 (null recurrence) separates the two equivalences: an argument that only compares the mean drift ρ−1\rho-1ρ−1 with 000, such as a law of large numbers for the increments, cannot tell recurrence from transience there.

Formalization scope

  • The chain is the published QueueingFundamentals.Foundations.TransitionMatrix (entries P.p i j, rows summing to 111 as a HasSum), with its firstPassage, returnProb, Irreducible, Aperiodic and PositiveRecurrent. "Ergodic" is P.PositiveRecurrent; aperiodicity is the paper's standing assumption and is not folded into it a second time.
  • States are indexed from 000, as in the paper; "i≠0i\ne0i=0" is i ≠ 0.
  • The M/G/1 matrix is a function mg1Matrix k : ℕ → ℕ → ℝ; the goal and the §3 display quantify over every TransitionMatrix P with P.p = mg1Matrix k. Such a P exists for every admissible k (checked in a sorry-free local file for ki=2−(i+1)k_i=2^{-(i+1)}ki​=2−(i+1)).
  • ∑nkn=1\sum_n k_n=1∑n​kn​=1 is added as the meaning of "stochastic matrix"; §3 writes only ki>0k_i>0ki​>0.
  • ρ\rhoρ and all mean first-passage times are extended nonnegative reals ([0,∞][0,\infty][0,∞]), so divergent means are ∞\infty∞, never 000. Theorem 7's mean is also taken in [0,∞][0,\infty][0,∞].
  • Recurrent means fjj=1f_{jj}=1fjj​=1 for every state jjj; transient means fjj<1f_{jj}<1fjj​<1 for every state. For irreducible chains these are complementary, which is a theorem, not a definition.
  • The general Theorems 3, 4 and 5 assume irreducibility and aperiodicity, the paper's standing assumption of §1. The goal does not assume them: they follow from ki>0k_i>0ki​>0.
  • Every series in a hypothesis carries its convergence (Summable or HasSum); Theorem 3's equation (6) is written as di=1+∑j≥1pijdjd_i=1+\sum_{j\ge1}p_{ij}d_jdi​=1+∑j≥1​pij​dj​ in [0,∞][0,\infty][0,∞] together with finiteness of the djd_jdj​, j≠0j\ne0j=0.
  • Theorem 7's distribution is renamed qqq in Lean to avoid a clash with pijp_{ij}pij​. Theorem 1 of the paper (§2) and Theorem 6 are not targets of this mission.

Ruled out: ρ\rhoρ as a real tsum (which is 000 for a divergent series and would call a heavy-tailed chain ergodic); defining "ergodic" or "recurrent" through the existence of Lyapunov or drift functions (which would make the criteria tautological); a goal over a matrix PPP that need not exist.

Contributions welcome: proofs of the general criteria (Theorems 2–5) on the published chain vocabulary, the limit theorem pij(n)→πjp_{ij}^{(n)}\to\pi_jpij(n)​→πj​ for irreducible aperiodic chains, first-step analysis for hitting times, and Theorem 7 as a lemma on probability generating functions; all of these are reusable beyond this mission.

Selected references

  • F. G. Foster, On the stochastic matrices associated with certain queuing processes, The Annals of Mathematical Statistics 24(3), 355–360, 1953. https://doi.org/10.1214/aoms/1177728976
  • D. G. Kendall, Some problems in the theory of queues, Journal of the Royal Statistical Society B 13(2), 151–185, 1951. https://doi.org/10.1111/j.2517-6161.1951.tb00093.x
  • D. G. Kendall, Stochastic processes occurring in the theory of queues and their analysis by the method of the imbedded Markov chain, The Annals of Mathematical Statistics 24(3), 338–354, 1953. https://doi.org/10.1214/aoms/1177728975
  • W. Feller, An Introduction to Probability Theory and Its Applications, Vol. I, Wiley, 1950.
  • S. Meyn and R. L. Tweedie, Markov Chains and Stochastic Stability, 2nd ed., Cambridge University Press, 2009. https://doi.org/10.1017/CBO9780511626630
9 thms2 active usersReviewed
Linear OptimizationOperations ResearchOptimization·Captain: mikedeng1

Maximization of a Linear Function of Variables Subject to Linear Inequalities: Under Nondegeneracy the Simplex Technique Ends in Infeasibility, Unboundedness, or a Maximum Feasible SolutionResearch Paper

Motivation

Linear programming asks for the best value of a linear objective under linear constraints. In the form studied by George B. Dantzig in 1951, all variables are nonnegative and the constraints are equalities. The simplex technique moves between small sets of columns, seeking a feasible vector first and then improving its objective. Its basic promise is operational: a run should end with a valid answer, whether that answer is a maximum, infeasibility, or an objective that can grow without bound. Dantzig's chapter sets out both phases and the tests that distinguish these outcomes.

The formulation matters to readers of optimization because it separates the algebraic claim about a linear program from a particular rule for selecting the next column. The chapter permits any entering column satisfying the stated improvement test. This mission captures the correctness claim for every sequence of admissible choices under the chapter's nondegeneracy assumption, with one additional general-position condition needed for the Phase I argument.

Setting

Fix integers 1≤m≤n1\le m\le n1≤m≤n. Let P0∈RmP_0\in\mathbb R^mP0​∈Rm be a right-hand side, P1,…,Pn∈RmP_1,\ldots,P_n\in\mathbb R^mP1​,…,Pn​∈Rm be columns, and c1,…,cn∈Rc_1,\ldots,c_n\in\mathbb Rc1​,…,cn​∈R be objective coefficients. A feasible solution is a weight vector λ∈Rn\lambda\in\mathbb R^nλ∈Rn with λj≥0\lambda_j\ge0λj​≥0 and ∑jλjPj=P0\sum_j\lambda_jP_j=P_0∑j​λj​Pj​=P0​. Its objective value is z(λ)=∑jλjcjz(\lambda)=\sum_j\lambda_jc_jz(λ)=∑j​λj​cj​. It is maximum feasible when z(μ)≤z(λ)z(\mu)\le z(\lambda)z(μ)≤z(λ) for every feasible μ\muμ. Unboundedness means that feasible objective values exceed every real threshold.

Dantzig assumes nondegeneracy: every indexed selection of mmm points among P0,P1,…,PnP_0,P_1,\ldots,P_nP0​,P1​,…,Pn​ is linearly independent. A Phase II state uses exactly mmm basic columns BBB, each with a positive weight, and represents P0P_0P0​ with them. Every column has unique coordinates Pj=∑i∈BxijPiP_j=\sum_{i\in B}x_{ij}P_iPj​=∑i∈B​xij​Pi​, and zj=∑i∈Bxijciz_j=\sum_{i\in B}x_{ij}c_izj​=∑i∈B​xij​ci​ is the corresponding objective value. A column with cj>zjc_j>z_jcj​>zj​ can improve the objective. When some xij>0x_{ij}>0xij​>0, the step uses the smallest ratio λi/xij\lambda_i/x_{ij}λi​/xij​ over those positive coordinates and replaces a minimizing basic column.

A Phase I state begins from m−1m-1m−1 basic columns SSS and a fixed reference point GGG. Positive weights wiw_iwi​ and ρ>0\rho>0ρ>0 satisfy G+ρP0=∑i∈SwiPiG+\rho P_0=\sum_{i\in S}w_iP_iG+ρP0​=∑i∈S​wi​Pi​. Write Pj=y0jP0+∑i∈SyijPiP_j=y_{0j}P_0+\sum_{i\in S}y_{ij}P_iPj​=y0j​P0​+∑i∈S​yij​Pi​. A column with y0j>0y_{0j}>0y0j​>0 either gives a new Phase I basis through the positive-ratio test or supplies the nonnegative weights in equation (39), which start Phase II.

Formalization targets

The goal is the correctness of the complete two-phase transition system. There is no infinite admissible run. Every state with no outgoing transition has one of three outcomes:

Phase I:y0j≤0 (∀j),no feasible solution;Phase II:∃j (cj>zj ∧ xij≤0 (∀i∈B)),∀M∈R ∃λ feasible:z(λ)>M;Phase II:cj≤zj (∀j),z(μ)≤z(λ) for every feasible μ.\begin{array}{ll} \text{Phase I:}& y_{0j}\le0\ (\forall j),\quad \text{no feasible solution};\\ \text{Phase II:}& \exists j\ (c_j>z_j\ \land\ x_{ij}\le0\ (\forall i\in B)),\quad \forall M\in\mathbb R\ \exists\lambda\text{ feasible}: z(\lambda)>M;\\ \text{Phase II:}& c_j\le z_j\ (\forall j),\quad z(\mu)\le z(\lambda)\text{ for every feasible }\mu. \end{array}Phase I:Phase II:Phase II:​y0j​≤0 (∀j),no feasible solution;∃j (cj​>zj​ ∧ xij​≤0 (∀i∈B)),∀M∈R ∃λ feasible:z(λ)>M;cj​≤zj​ (∀j),z(μ)≤z(λ) for every feasible μ.​

The five milestones follow the chapter's own statements: Theorem 3's infeasibility certificate, Section 2's termination and hand-off, Theorem 1's improving family and two cases, Section 1's termination alternatives, and Theorem 2's optimality test. The goal includes the hand-off from the first phase to the second; the milestones make each outcome separately auditable.

Significance

The result gives a complete outcome guarantee for the chapter's procedure under its stated nondegeneracy regime. A terminal basis satisfying cj≤zjc_j\le z_jcj​≤zj​ is certified optimal against every feasible solution, not just against nearby bases. The other terminal tests certify properties of the original problem: nonexistence of feasible weights or arbitrarily large feasible objective values. This is the distinction needed to use the procedure as an algorithm for an LP rather than merely a local improvement rule.

The chapter's Theorems A and B assert existence of a basic feasible solution, and existence of a basic optimal one when the objective is bounded above. A proved platform theorem on basic optimal solutions covers these structural results after changing minimization cost qqq to −c-c−c; it is included by reference. Existing simplex results in the platform's minimization convention do not state Dantzig's Phase I reference-point process or his xijx_{ij}xij​ and zjz_jzj​ tests. Formalizing this mission adds that two-phase interface and a machine-checkable statement of its outcomes.

Difficulty

An improving column alone does not say whether another basis exists. In Phase II, the signs of its basis coordinates determine whether the positive weights meet a finite ratio limit or instead continue along an unbounded feasible family. In Phase I, the analogous limit must preserve strictly positive weights on exactly m−1m-1m−1 columns. The printed argument says that a coefficient vanishes at the limit, but does not exclude two coefficients vanishing together. That gap matters because the next state in the paper's recurrence must again have strictly positive basic weights. The mission states a general-position condition on GGG that rules out this simultaneous-vanishing case. It is an explicit addition to the printed assumptions and should be reviewed as such.

Formalization scope

Lean uses Fin n → (Fin m → ℝ) for the columns, Fin m → ℝ for P0P_0P0​ and GGG, Fin n → ℝ for weights and costs, and finite sets for bases. Its zero-based Fin n labels correspond to the paper's 1,…,n1,\ldots,n1,…,n. Each state carries its basis cardinality, independence, coordinate equations, and strictly positive basic weights. This keeps the coordinate functions defined on genuine bases and prevents a zero-weight intermediate state from counting as a valid pivot. The transition relation records every admissible entering column and minimizing leaving index; the pivot-selection heuristics (20) and (21) are outside the claim.

The phrases “upper bound of zzz is infinite” and “process terminates” are read as, respectively, objective values above every real threshold on the original feasible set and absence of an infinite sequence of admissible steps. “A feasible solution has been obtained” is the explicit vector of equation (39), not an unnamed existence assertion. Maximum feasible means feasible plus comparison with every feasible vector, avoiding a real supremum's empty-set default. The dimensions 1≤m≤n1\le m\le n1≤m≤n make both kinds of basis possible and prevent a vacuous nondegeneracy condition. General position asks every family containing GGG, P0P_0P0​, and m−2m-2m−2 distinct PjP_jPj​ to be independent. The paper does not write this condition, so its inclusion is a substantive qualification of the target.

The printed indices in (30), (45), and a sentence following (18) do not match their defining ranges. The Lean statements use all nnn columns for jjj and the current m−1m-1m−1 Phase I columns for iii. The indices in (17) and (45) also carry the entering conditions cj>zjc_j>z_jcj​>zj​ and y0j>0y_{0j}>0y0j​>0, respectively. These corrections preserve the paper's described procedure and prevent a terminal test from becoming trivial through a basic column. Contributions may address the finite-state termination argument, the ratio-test lemmas, coordinate uniqueness, and the translation from terminal signs to the three global LP outcomes. The operation counts and geometric interpretation on the last pages are outside the formalization.

Selected references

  • George B. Dantzig, “Maximization of a Linear Function of Variables Subject to Linear Inequalities,” in T. C. Koopmans (ed.), Activity Analysis of Production and Allocation, Wiley, 1951, Chapter XXI, pp. 339–347. Book catalog search.
  • Hartmann_Psi, “A bounded feasible standard-form LP attains its minimum at a basic feasible solution,” Prove2Me, proved platform theorem. Theorem record.
  • Shuze Chen, “Finite termination of the simplex method under nondegeneracy,” Prove2Me, proved platform theorem in a minimization convention. Theorem record.
9 thms3 active usersReviewed
Bandit AlgorithmsMachine LearningOperations Research·Captain: mikedeng1

Regret Analysis of Stochastic and Nonstochastic Multi-armed Bandit Problems I: Pseudo-Regret of (α, ψ)-UCBTextbook

Motivation

The stochastic multi-armed bandit is the basic model of sequential decisions under uncertainty with partial feedback: a forecaster repeatedly picks one of KKK options and observes only the reward of the option it picked. It models clinical trials, ad placement, routing and dynamic pricing, and is the building block of many reinforcement-learning algorithms. The question is how much reward is lost, compared with always playing the best option, through having to learn which option is best.

Chapter 2 of Bubeck and Cesa-Bianchi's monograph (arXiv:1204.5721v2) answers this for upper confidence bound (UCB) strategies.

  • Lai and Robbins (1985) introduced upper confidence bounds and proved that the number of pulls of a suboptimal arm must grow at least logarithmically, with an explicit constant, for consistent strategies (doi:10.1016/0196-8858(85)90002-8).
  • Agrawal (1995) gave simpler sample-mean-based index policies with logarithmic regret (doi:10.2307/1427934).
  • Auer, Cesa-Bianchi and Fischer (2002) gave the finite-time analysis of UCB1 for bounded rewards (doi:10.1023/A:1013689704352).
  • Bubeck and Cesa-Bianchi (2012) present the (α,ψ)(\alpha,\psi)(α,ψ)-UCB family, whose analysis needs only a bound ψ\psiψ on the cumulant generating function of the rewards, and the Lai–Robbins lower bound for Bernoulli rewards.

Setting

There are K≥2K\ge2K≥2 arms. Arm iii has an unknown reward distribution νi\nu_iνi​ with mean μi\mu_iμi​. At each round t=1,2,…t=1,2,\dotst=1,2,… the forecaster selects an arm ItI_tIt​ based on the past and receives a reward drawn from νIt\nu_{I_t}νIt​​, independently of the past. Write μ∗=max⁡iμi\mu^*=\max_i\mu_iμ∗=maxi​μi​, Δi=μ∗−μi\Delta_i=\mu^*-\mu_iΔi​=μ∗−μi​ for the gap of arm iii, and Ti(n)T_i(n)Ti​(n) for the number of times arm iii is selected in rounds 1,…,n1,\dots,n1,…,n. The pseudo-regret is

R‾n=nμ∗−E∑t=1nμIt=∑i=1KΔi E Ti(n).\overline R_n=n\mu^*-\mathbb E\sum_{t=1}^n\mu_{I_t}=\sum_{i=1}^K\Delta_i\,\mathbb E\,T_i(n).Rn​=nμ∗−Et=1∑n​μIt​​=i=1∑K​Δi​ETi​(n).

Moment condition (2.2). There is a convex ψ:R→R\psi:\mathbb R\to\mathbb Rψ:R→R with ln⁡E eλ(X−EX)≤ψ(λ)\ln\mathbb E\,e^{\lambda(X-\mathbb EX)}\le\psi(\lambda)lnEeλ(X−EX)≤ψ(λ) and ln⁡E eλ(EX−X)≤ψ(λ)\ln\mathbb E\,e^{\lambda(\mathbb EX-X)}\le\psi(\lambda)lnEeλ(EX−X)≤ψ(λ) for all λ≥0\lambda\ge0λ≥0 and every arm's reward XXX. Its Legendre–Fenchel transform is ψ∗(ε)=sup⁡λ∈R(λε−ψ(λ))\psi^*(\varepsilon)=\sup_{\lambda\in\mathbb R}(\lambda\varepsilon-\psi(\lambda))ψ∗(ε)=supλ∈R​(λε−ψ(λ)). For [0,1][0,1][0,1] rewards one may take ψ(λ)=λ2/8\psi(\lambda)=\lambda^2/8ψ(λ)=λ2/8, for which ψ∗(ε)=2ε2\psi^*(\varepsilon)=2\varepsilon^2ψ∗(ε)=2ε2.

(α,ψ)(\alpha,\psi)(α,ψ)-UCB. With μ^i,s\hat\mu_{i,s}μ^​i,s​ the mean of the first sss rewards of arm iii, at round ttt select

It∈argmax⁡i[μ^i,Ti(t−1)+(ψ∗)−1(αln⁡tTi(t−1))].I_t\in\operatorname*{argmax}_{i}\Big[\hat\mu_{i,T_i(t-1)}+(\psi^*)^{-1}\Big(\frac{\alpha\ln t}{T_i(t-1)}\Big)\Big].It​∈iargmax​[μ^​i,Ti​(t−1)​+(ψ∗)−1(Ti​(t−1)αlnt​)].

Formalization targets

Goal: Theorem 2.1 (p. 11)

If the rewards satisfy (2.2), then (α,ψ)(\alpha,\psi)(α,ψ)-UCB with α>2\alpha>2α>2 satisfies, for every nnn,

R‾n≤∑i:Δi>0Δi(αln⁡nψ∗(Δi/2)+αα−2).\overline R_n\le\sum_{i:\Delta_i>0}\Delta_i\Big(\frac{\alpha\ln n}{\psi^*(\Delta_i/2)}+\frac{\alpha}{\alpha-2}\Big).Rn​≤i:Δi​>0∑​Δi​(ψ∗(Δi​/2)αlnn​+α−2α​).

This is the bound the book's proof establishes. The printed statement has α/(α−2)\alpha/(\alpha-2)α/(α−2) in place of Δi α/(α−2)\Delta_i\,\alpha/(\alpha-2)Δi​α/(α−2); see Formalization scope.

Milestones

  • The decomposition R‾n=∑iΔi E Ti(n)\overline R_n=\sum_i\Delta_i\,\mathbb E\,T_i(n)Rn​=∑i​Δi​ETi​(n) (p. 9).
  • The Cramér–Chernoff bound (2.3): P(μi−μ^i,s>ε)≤e−sψ∗(ε)\mathbb P(\mu_i-\hat\mu_{i,s}>\varepsilon)\le e^{-s\psi^*(\varepsilon)}P(μi​−μ^​i,s​>ε)≤e−sψ∗(ε).
  • The three-event lemma (2.5)–(2.7) from the proof of Theorem 2.1.
  • The bounded-reward bound (2.4): R‾n≤∑i:Δi>0(2αΔiln⁡n+αα−2)\overline R_n\le\sum_{i:\Delta_i>0}\big(\frac{2\alpha}{\Delta_i}\ln n+\frac{\alpha}{\alpha-2}\big)Rn​≤∑i:Δi​>0​(Δi​2α​lnn+α−2α​).
  • The comparison (2.8): 2(p−q)2≤kl(p,q)≤(p−q)2/(q(1−q))2(p-q)^2\le\mathrm{kl}(p,q)\le(p-q)^2/(q(1-q))2(p−q)2≤kl(p,q)≤(p−q)2/(q(1−q)).
  • Theorem 2.2: for every strategy with E Ti(n)=o(na)\mathbb E\,T_i(n)=o(n^a)ETi​(n)=o(na) on all Bernoulli instances, lim inf⁡nR‾n/ln⁡n≥∑i:Δi>0Δi/kl(μi,μ∗)\liminf_n\overline R_n/\ln n\ge\sum_{i:\Delta_i>0}\Delta_i/\mathrm{kl}(\mu_i,\mu^*)liminfn​Rn​/lnn≥∑i:Δi​>0​Δi​/kl(μi​,μ∗).

Significance

Theorem 2.1 says that the cost of learning grows only logarithmically in the horizon, with a constant set by how well each suboptimal arm can be told apart from the best one. Theorem 2.2 shows that, up to constants, this cannot be improved: for Bernoulli rewards any strategy that is good on every instance must pay ln⁡n\ln nlnn per suboptimal arm, with constant Δi/kl(μi,μ∗)\Delta_i/\mathrm{kl}(\mu_i,\mu^*)Δi​/kl(μi​,μ∗). By (2.8) this constant is at least of order 1/Δi1/\Delta_i1/Δi​, matching (2.4). Together they are the template for the analysis of most optimistic algorithms: KL-UCB, linear and contextual UCB, and UCB-style reinforcement learning.

All results of the chapter are classical and proved on paper. This mission makes them machine-checked in a common model. That model has an explicit pseudo-regret, an explicit (generalized) inverse of ψ∗\psi^*ψ∗, an explicit initialization rule for the algorithm, and a randomized-strategy model for the lower bound. Later chapters of the series and later papers on optimistic algorithms can build on it.

Difficulty

The deterministic part of the upper bound is short, so the main difficulty is probabilistic. The sample mean μ^i,Ti(t−1)\hat\mu_{i,T_i(t-1)}μ^​i,Ti​(t−1)​ is taken over a random number of samples that depends on the algorithm's past. A Chernoff bound for a fixed sample size does not apply to it directly. The proof needs a union bound over all possible sample sizes, together with the representation in which the sss-th reward of each arm is a fixed random variable. Summing the resulting tail t1−αt^{1-\alpha}t1−α over rounds is where α>2\alpha>2α>2 enters.

The lower bound needs a change-of-measure argument between two Bernoulli instances, applied to a forecaster that may be randomized and never knows the horizon. Expressing "the forecaster cannot distinguish the instances" requires the law of the whole interaction under two environments.

Formalization scope

Model. Arms are Fin K with 2≤K2\le K2≤K, rounds are 1,2,…1,2,\dots1,2,…, and the natural logarithm is used. Rewards are a stack: Xi,kX_{i,k}Xi,k​ is the reward of the (k+1)(k+1)(k+1)-st pull of arm iii, all mutually independent, identically distributed per arm. For every strategy this gives the same law of arms and rewards as the book's protocol. μ∗\mu^*μ∗, Δi\Delta_iΔi​ and Ti(t)T_i(t)Ti​(t) are the published definitions of ImprovedLinBandits.UCBDelta.armModel. The pseudo-regret is (2.1), nμ∗−E∑tμItn\mu^*-\mathbb E\sum_t\mu_{I_t}nμ∗−E∑t​μIt​​, not the expected regret. The arms played are measurable random variables, so that every expectation is genuine.

ψ∗\psi^*ψ∗ and its inverse. ψ∗\psi^*ψ∗ takes values in the extended reals. (ψ∗)−1(y)=inf⁡{ε≥0:ψ∗(ε)≥y}(\psi^*)^{-1}(y)=\inf\{\varepsilon\ge0:\psi^*(\varepsilon)\ge y\}(ψ∗)−1(y)=inf{ε≥0:ψ∗(ε)≥y}.

Algorithm. The index is undefined while Ti(t−1)=0T_i(t-1)=0Ti​(t−1)=0. Unplayed arms are therefore played first, so each arm is played once in rounds 1,…,K1,\dots,K1,…,K. Ties are broken arbitrarily.

Corrected misprint. The book prints Theorem 2.1 with constant term α/(α−2)\alpha/(\alpha-2)α/(α−2). Its proof yields Δi α/(α−2)\Delta_i\,\alpha/(\alpha-2)Δi​α/(α−2) (bound on E Ti(n)\mathbb E\,T_i(n)ETi​(n) times Δi\Delta_iΔi​). The printed form is false for Gaussian rewards with large gaps. The goal states the proof's version. (2.4) is correct as printed because Δi≤1\Delta_i\le1Δi​≤1.

Constants. No O(·) appears in the chapter's statements. The constants are the book's: α/(α−2)\alpha/(\alpha-2)α/(α−2) in Theorem 2.1 and (2.4), and the factor 222 in (2.4) and (2.8).

Conventions made explicit.

  1. ψ(λ)≥ψ(0)\psi(\lambda)\ge\psi(0)ψ(λ)≥ψ(0) for λ≤0\lambda\le0λ≤0. Condition (2.2) constrains ψ\psiψ only on λ≥0\lambda\ge0λ≥0, while ψ∗\psi^*ψ∗ takes the supremum over all of R\mathbb RR. Without this convention, (2.3) and Theorem 2.1 are false (for example ψ(λ)=cλ+λ2/8\psi(\lambda)=c\lambda+\lambda^2/8ψ(λ)=cλ+λ2/8 with large ccc).
  2. ψ∗\psi^*ψ∗ is finite on [0,∞)[0,\infty)[0,∞).
  3. ψ∗(Δi/2)>0\psi^*(\Delta_i/2)>0ψ∗(Δi​/2)>0 for suboptimal arms.
  4. ε≥0\varepsilon\ge0ε≥0 in (2.3).
  5. q∈(0,1)q\in(0,1)q∈(0,1) in (2.8).

Conventions 1–3 hold for every ψ\psiψ the book uses.

Theorem 2.2. The forecaster is a measurable rule from the history and a fresh uniform seed to an arm. It does not depend on the horizon. Consistency is required on every Bernoulli instance, every suboptimal arm and every a>0a>0a>0. A term with μ∗=1\mu^*=1μ∗=1 (kl=+∞\mathrm{kl}=+\inftykl=+∞) is 000, and the lim inf⁡\liminfliminf is taken in the extended reals. The book proves only K=2K=2K=2.

Ruled out. The algorithm cannot see unplayed rewards (its index uses only the sample means of rewards already received). Non-measurable arm choices, which would make the expectations vanish in Lean, are excluded. Consistency cannot be assumed only on the instance of the conclusion.

Needed infrastructure. Cramér–Chernoff bounds for sums of i.i.d. variables, Hoeffding's lemma, the stack representation of bandit interactions, and the divergence decomposition for randomized strategies. All are reusable beyond this mission, and contributions of any of them are welcome.

Selected references

  • S. Bubeck, N. Cesa-Bianchi, Regret Analysis of Stochastic and Nonstochastic Multi-armed Bandit Problems, Foundations and Trends in Machine Learning 5(1), 2012. arXiv:1204.5721v2, doi:10.1561/2200000024
  • T. L. Lai, H. Robbins, Asymptotically efficient adaptive allocation rules, Advances in Applied Mathematics 6, 1985. doi:10.1016/0196-8858(85)90002-8
  • R. Agrawal, Sample mean based index policies with O(log n) regret for the multi-armed bandit problem, Advances in Applied Probability 27, 1995. doi:10.2307/1427934
  • P. Auer, N. Cesa-Bianchi, P. Fischer, Finite-time analysis of the multiarmed bandit problem, Machine Learning 47, 2002. doi:10.1023/A:1013689704352
12 thms2 active usersReviewed
Markov ChainOperations ResearchProbability+1·Captain: mikedeng1

Reversibility and Stochastic Networks I: Kolmogorov's Criterion — a Stationary Markov Process Is Reversible iff Its Rates Balance Around Every CycleTextbook

Why reversibility

A stochastic process is reversible when a film of it run backwards is statistically indistinguishable from the film run forwards. For Markov processes in equilibrium this distributional symmetry has an algebraic counterpart, the detailed balance conditions, and that counterpart is what makes large classes of queueing networks, migration processes, loss networks, clustering processes and population-genetics models solvable in closed form. F. P. Kelly's Reversibility and Stochastic Networks (Wiley, 1979) builds the whole theory of product-form equilibria on this link, and Chapter 1 sets it up.

The criterion that bears Kolmogorov's name goes back to A. Kolmogorov, "Zur Theorie der Markoffschen Ketten" (Math. Annalen 112, 1936), who showed that reversibility of a chain can be read off its transition probabilities around closed cycles. Kelly's Chapter 1 (§§1.1–1.7) states the criterion for chains (Theorem 1.7) and processes (Theorem 1.8), together with the detailed-balance characterization (Theorems 1.2, 1.3), the cut and tree lemmas (1.4, 1.5), the monotonicity of relative entropy (Theorem 1.6), truncation and rate alteration (Lemma 1.9, Corollary 1.10), and the description of the time-reversed process (Theorems 1.12–1.14). This mission is the first in a series covering the book.

Setting

Let S\mathcal SS be a finite state space. A continuous-time Markov process X(t)X(t)X(t), t∈Rt\in\mathbb Rt∈R, is specified by transition rates q(j,k)≥0q(j,k)\ge 0q(j,k)≥0, j≠kj\ne kj=k, with Kelly's convention q(j,j)=0q(j,j)=0q(j,j)=0. Its generator is the matrix QQQ with Q(j,k)=q(j,k)Q(j,k)=q(j,k)Q(j,k)=q(j,k) off the diagonal and Q(j,j)=−∑k≠jq(j,k)Q(j,j)=-\sum_{k\ne j}q(j,k)Q(j,j)=−∑k=j​q(j,k), and its transition matrices are P(t)=etQP(t)=e^{tQ}P(t)=etQ. The rates are irreducible if every state can be reached from every other through transitions of positive rate. An equilibrium distribution is a collection of positive numbers π(j)\pi(j)π(j) summing to one that satisfy the equilibrium equations

π(j)∑kq(j,k)=∑kπ(k)q(k,j),j∈S.(1.3)\pi(j)\sum_{k}q(j,k)=\sum_k\pi(k)q(k,j),\qquad j\in\mathcal S. \qquad (1.3)π(j)k∑​q(j,k)=k∑​π(k)q(k,j),j∈S.(1.3)

The stationary process with rates qqq and equilibrium distribution π\piπ has finite-dimensional distributions

P(X(t1)=j1,…,X(tn)=jn)=π(j1)∏r=1n−1P(tr+1−tr)(jr,jr+1),t1≤⋯≤tn.P\bigl(X(t_1)=j_1,\dots,X(t_n)=j_n\bigr)=\pi(j_1)\prod_{r=1}^{n-1}P(t_{r+1}-t_r)(j_r,j_{r+1}),\qquad t_1\le\dots\le t_n .P(X(t1​)=j1​,…,X(tn​)=jn​)=π(j1​)r=1∏n−1​P(tr+1​−tr​)(jr​,jr+1​),t1​≤⋯≤tn​.

It is reversible (Kelly, p. 5) if (X(t1),…,X(tn))(X(t_1),\dots,X(t_n))(X(t1​),…,X(tn​)) has the same distribution as (X(τ−t1),…,X(τ−tn))(X(\tau-t_1),\dots,X(\tau-t_n))(X(τ−t1​),…,X(τ−tn​)) for all t1,…,tn,τt_1,\dots,t_n,\taut1​,…,tn​,τ. The rates satisfy detailed balance with π\piπ if π(j)q(j,k)=π(k)q(k,j)\pi(j)q(j,k)=\pi(k)q(k,j)π(j)q(j,k)=π(k)q(k,j) for all j,kj,kj,k, and Kolmogorov's cycle condition if

q(j1,j2)q(j2,j3)⋯q(jn−1,jn)q(jn,j1)=q(j1,jn)q(jn,jn−1)⋯q(j3,j2)q(j2,j1)(1.22)q(j_1,j_2)q(j_2,j_3)\cdots q(j_{n-1},j_n)q(j_n,j_1)=q(j_1,j_n)q(j_n,j_{n-1})\cdots q(j_3,j_2)q(j_2,j_1)\qquad (1.22)q(j1​,j2​)q(j2​,j3​)⋯q(jn−1​,jn​)q(jn​,j1​)=q(j1​,jn​)q(jn​,jn−1​)⋯q(j3​,j2​)q(j2​,j1​)(1.22)

for every finite sequence of states j1,…,jnj_1,\dots,j_nj1​,…,jn​. The discrete-time analogue replaces rates by a stochastic matrix p(j,k)p(j,k)p(j,k) and P(t)P(t)P(t) by the matrix power PtP^tPt, t∈Z≥0t\in\mathbb Z_{\ge 0}t∈Z≥0​.

Formalization targets

Goal: Theorem 1.8 (p. 23)

For a stationary, irreducible Markov process on a finite state space,

X is reversible  ⟺  q satisfies (1.22) for every finite sequence of states.X\ \text{is reversible}\iff q\ \text{satisfies (1.22) for every finite sequence of states}.X is reversible⟺q satisfies (1.22) for every finite sequence of states.

The left side is a statement about the joint laws of the process at all finite sets of times; the right side involves the rates alone, not even the equilibrium distribution.

Milestones

  1. Theorems 1.2 and 1.3: reversibility of a stationary chain or process is equivalent to the existence of a positive, normalized solution of detailed balance, and that solution is the equilibrium distribution.
  2. Theorem 1.7: Kolmogorov's criterion (1.21) for chains. (The platform's proved rate-level equivalence of (1.22) with detailed balance under two-way communication, Serfozo's Theorem 2.8, is included as a reference item but is not a milestone: it is not Kelly's statement.)
  3. Lemmas 1.4 and 1.5: the flux across any cut balances, and a process whose graph is a tree is reversible.
  4. Theorem 1.6: H(t)=∑jπ(j)h(uj(t)/π(j))H(t)=\sum_j\pi(j)h(u_j(t)/\pi(j))H(t)=∑j​π(j)h(uj​(t)/π(j)) is strictly increasing for t>0t>0t>0 when hhh is strictly concave and the initial distribution is not the equilibrium one.
  5. Lemma 1.9 and Corollary 1.10: altering the rates across a cut by a factor c>0c>0c>0, or truncating to a subset, preserves reversibility with explicit equilibrium distributions.
  6. Theorems 1.12–1.14: the reversed process X(τ−t)X(\tau-t)X(τ−t) is stationary Markov with rates q′(j,k)=π(k)q(k,j)/π(j)q'(j,k)=\pi(k)q(k,j)/\pi(j)q′(j,k)=π(k)q(k,j)/π(j); conditions (1.27)–(1.28) identify it; and dynamic reversibility is characterized by π(j)=π(j+)\pi(j)=\pi(j^+)π(j)=π(j+) and π(j)q(j,k)=π(k+)q(k+,j+)\pi(j)q(j,k)=\pi(k^+)q(k^+,j^+)π(j)q(j,k)=π(k+)q(k+,j+).

Significance

Theorem 1.8 is the working test for reversibility throughout the book and the queueing literature: a model's reversibility, and with it a product-form equilibrium obtained by solving detailed balance, can be decided by checking a finite list of cycles in its transition diagram. Theorem 1.3 converts the distributional property into equations that later chapters solve explicitly; Theorems 1.12 and 1.13 underlie the treatment of quasi-reversible queues and networks in Chapter 3; Lemma 1.9 and Corollary 1.10 produce the equilibria of loss systems and queues with shared buffers.

The algebraic content of several of these results is already proved on the platform at the level of rates: detailed balance implies the equilibrium equations, the reversed rates preserve π\piπ, truncation preserves detailed balance, and Kolmogorov's criterion is equivalent to detailed balance for two-way communicating rates. What is not formalized anywhere is the process-level statement: that these conditions are equivalent to the time-reversal symmetry of the finite-dimensional distributions built from etQe^{tQ}etQ. This mission supplies that layer, which connects the rate identities to the probabilistic notion they are meant to capture.

Difficulty

The rate-level identities are short; the difficulty is the passage between them and the process. Reversibility constrains the joint law at every finite set of times, and that law is built from the matrix exponential etQe^{tQ}etQ, whose entries are not explicit functions of the rates. Relating the two requires the analytic facts about etQe^{tQ}etQ for a generator (stationarity of π\piπ, the semigroup property, behaviour as t→0t\to 0t→0, strict positivity of the entries for t>0t>0t>0 under irreducibility) that Mathlib does not yet provide for Markov generators, together with bookkeeping for tuples of times given in arbitrary order, with ties. For Theorem 1.8 a positive, normalized equilibrium has to be produced from the cycle condition alone, on rates that may vanish in one direction only: two-way communication is not a hypothesis, so the platform's proved rate-level criterion does not apply directly. A first attempt that defines reversibility as detailed balance avoids all of this and proves nothing new; it is excluded below.

Formalization scope

The state space is a Fintype with decidable equality; Kelly allows a countable state space, and this restriction is stated in every item. Rates are q : S → S → ℝ with 0 ≤ q j k for j ≠ k and q j j = 0 as hypotheses. The transition matrices are NormedSpace.exp (t • generator q). Time sets are ℝ (processes) and ℤ (chains). Finite-dimensional distributions are defined for arbitrary finite tuples of time points by sorting them with Tuple.sort. Equilibrium means positive, summing to one, and satisfying (1.3) (resp. πP=π\pi P=\piπP=π), and every theorem takes the equilibrium distribution of the stationary process as a hypothesis. Existing platform definitions are reused: FullBalance, DetailedBalance and reversedRates from KellyStochasticNetworks_Balance, the stochastic-matrix vocabulary of mm_basic, and truncatedRates from KellyStochasticNetworks_LossNetwork.

Reversibility is not defined as detailed balance or as equality of qqq with its reversed rates: under such a definition Theorems 1.2 and 1.3 would be tautologies and Theorem 1.8 would be the already proved rate-level criterion. It is the distributional definition of p. 5. Kolmogorov's condition ranges over all sequences of all lengths, repetitions allowed, and two-way communication is not assumed.

A complete development needs the matrix exponential of a generator (positivity, the semigroup property, the derivative at 000, and πetQ=π\pi e^{tQ}=\piπetQ=π), which is reusable for any finite-state continuous-time Markov chain, and a lemma on reversing sorted tuples. Lemma 1.1 (a general stationary process) and Lemma 1.11 (a non-stationary reversal) are not included. Proofs of any milestone, and generalizations to countable state spaces, are welcome.

Selected references

  • F. P. Kelly, Reversibility and Stochastic Networks, John Wiley & Sons, 1979, Chapter 1. https://www.statslab.cam.ac.uk/~frank/BOOKS/kelly_book.html
  • A. Kolmogorov, "Zur Theorie der Markoffschen Ketten", Mathematische Annalen 112 (1936), 155–160. https://doi.org/10.1007/BF01565412
  • R. Serfozo, Introduction to Stochastic Networks, Springer, 1999, Chapter 1. https://doi.org/10.1007/978-1-4612-1482-3
  • F. P. Kelly and E. Yudovina, Stochastic Networks, Cambridge University Press, 2014, Chapter 1. https://doi.org/10.1017/CBO9781139565363
21 thms5 active usersReviewed
Markov ChainOperations ResearchProbability+1·Captain: mikedeng1

Reversibility and Stochastic Networks II: Migration Processes with Blocking — Reversibility and Product-Form EquilibriumTextbook

Motivation

A migration process is a continuous-time Markov model of a population spread over JJJ sites, called colonies, in which individuals move one at a time: between colonies, out of the system, or into it from outside. The model was introduced by Whittle (1967) and Kingman (1969) and is the framework of Chapter 2 of F. P. Kelly, Reversibility and Stochastic Networks (Wiley, 1979). It covers closed and open networks of queues (Jackson networks), linear population models and the movement of particles between cells. Its central fact is that the equilibrium distribution is a product form: the joint law of the colony sizes factorizes into one term per colony, and for open processes the colony sizes are independent in equilibrium.

In the migration processes of Chapter 2 the rate at which an individual leaves colony jjj depends on the number njn_jnj​ there, but not on the number already present in the colony it joins. Chapter 6 (§6.1) lifts this restriction. The rate of a move into colony kkk is multiplied by a function ψk(nk)\psi_k(n_k)ψk​(nk​) of the receiving colony's occupancy, which can model blocking (a crowded colony slows arrivals) or attraction (as in the models of social grouping of §6.2). The price is that product form survives only under a reversibility condition on the routing parameters. This mission formalizes the two theorems of §6.1 on top of the already-formalized results of Chapter 2.

Timeline. Whittle (1967) and Kingman (1969) introduced migration processes and their product-form equilibria; Kelly (1976) treated networks of queues with general routes; Kelly (1979, Ch. 2) states the closed and open product forms (Theorems 2.3, 2.4) and the time reversal of an open process (Theorem 2.5); Kelly (1979, §6.1) states the reversible versions with receiving-colony dependence (Theorems 6.1, 6.2).

Setting

There are JJJ colonies. The state is n=(n1,…,nJ)∈NJn = (n_1,\dots,n_J) \in \mathbb{N}^Jn=(n1​,…,nJ​)∈NJ, with njn_jnj​ the number of individuals in colony jjj. Three operators change the state by one individual: TjknT_{jk}nTjk​n moves one individual from colony jjj to colony kkk, Tj⋅nT_{j\cdot}nTj⋅​n removes one from colony jjj, and T⋅knT_{\cdot k}nT⋅k​n adds one to colony kkk.

The model is given by non-negative constants λjk\lambda_{jk}λjk​ (with λjj=0\lambda_{jj} = 0λjj​=0), μj\mu_jμj​, νk\nu_kνk​, and functions φj,ψj:N→R\varphi_j,\psi_j : \mathbb{N}\to\mathbb{R}φj​,ψj​:N→R with φj(0)=0\varphi_j(0) = 0φj​(0)=0, φj(n)>0\varphi_j(n) > 0φj​(n)>0 for n>0n > 0n>0, and ψj(n)>0\psi_j(n) > 0ψj​(n)>0 for all n≥0n \ge 0n≥0. A closed reversible migration process with NNN individuals has state space S={n:∑jnj=N}\mathcal{S} = \{n : \sum_j n_j = N\}S={n:∑j​nj​=N} and transition rates

q(n,Tjkn)=λjk φj(nj) ψk(nk).(6.2)q(n, T_{jk}n) = \lambda_{jk}\,\varphi_j(n_j)\,\psi_k(n_k). \qquad (6.2)q(n,Tjk​n)=λjk​φj​(nj​)ψk​(nk​).(6.2)

An open reversible migration process has state space NJ\mathbb{N}^JNJ and, in addition to (6.2), the rates

q(n,Tj⋅n)=μj φj(nj)(6.5),q(n,T⋅kn)=νk ψk(nk)(6.6).q(n, T_{j\cdot}n) = \mu_j\,\varphi_j(n_j) \quad (6.5), \qquad q(n, T_{\cdot k}n) = \nu_k\,\psi_k(n_k) \quad (6.6).q(n,Tj⋅​n)=μj​φj​(nj​)(6.5),q(n,T⋅k​n)=νk​ψk​(nk​)(6.6).

The parameters are required to make the process irreducible: in the closed case an individual can pass between any two colonies along pairs with λab>0\lambda_{ab} > 0λab​>0; in the open case it can reach every colony from outside and leave from every colony. Setting ψj≡1\psi_j \equiv 1ψj​≡1 recovers the migration processes of Chapter 2.

Given positive constants α1,…,αJ\alpha_1,\dots,\alpha_Jα1​,…,αJ​, the candidate equilibrium is

π(n)=B∏j=1J{αjnj∏r=1njψj(r−1)φj(r)},(6.3)\pi(n) = B\prod_{j=1}^{J}\Bigl\{\alpha_j^{n_j}\prod_{r=1}^{n_j}\frac{\psi_j(r-1)}{\varphi_j(r)}\Bigr\}, \qquad (6.3)π(n)=Bj=1∏J​{αjnj​​r=1∏nj​​φj​(r)ψj​(r−1)​},(6.3)

with BBB chosen so that π\piπ sums to one over the state space.

Formalization targets

Goal: Theorem 6.2 (open process)

If positive αj\alpha_jαj​ satisfy

αjλjk=αkλkj(6.4),αjμj=νj(6.7),\alpha_j\lambda_{jk} = \alpha_k\lambda_{kj} \quad (6.4), \qquad \alpha_j\mu_j = \nu_j \quad (6.7),αj​λjk​=αk​λkj​(6.4),αj​μj​=νj​(6.7),

and every colony series gj=∑m≥0αjm∏r=1mψj(r−1)/φj(r)g_j = \sum_{m\ge 0}\alpha_j^m\prod_{r=1}^m \psi_j(r-1)/\varphi_j(r)gj​=∑m≥0​αjm​∏r=1m​ψj​(r−1)/φj​(r) converges, then (6.3) with B=∏jgj−1B = \prod_j g_j^{-1}B=∏j​gj−1​ is in detailed balance with the rates (6.2), (6.5), (6.6), satisfies the equilibrium equations, is positive and sums to one, and under it n1,…,nJn_1,\dots,n_Jn1​,…,nJ​ are independent with marginals πj(m)=gj−1αjm∏r=1mψj(r−1)/φj(r)\pi_j(m) = g_j^{-1}\alpha_j^m\prod_{r=1}^m\psi_j(r-1)/\varphi_j(r)πj​(m)=gj−1​αjm​∏r=1m​ψj​(r−1)/φj​(r).

Milestones

  • Theorem 2.3: the closed migration process (ψ≡1\psi\equiv 1ψ≡1) has equilibrium of the form (2.3).
  • Theorem 2.4: the open migration process has independent colonies with marginals bjαjnj/∏r=1njφj(r)b_j\alpha_j^{n_j}/\prod_{r=1}^{n_j}\varphi_j(r)bj​αjnj​​/∏r=1nj​​φj​(r).
  • Theorem 2.5: the reversal of a stationary open migration process is an open migration process.
  • Theorem 6.1: the closed process with rates (6.2) is reversible under (6.4), with equilibrium (6.3) on S\mathcal{S}S.

Theorems 2.3–2.5 are already proved on the platform and enter as references.

Significance

The result. Theorem 6.2 shows that product form and independence of colony sizes are not tied to routing that ignores the destination's occupancy. Any positive ψk\psi_kψk​ is allowed, provided the routing is reversible in the sense of (6.4) and (6.7). This is what makes the social-grouping models of §6.2 and the clustering models of Chapter 8 tractable, and it identifies (6.4) as the structural condition: Exercise 6.1.1 shows that without it the process cannot be reversible. Through Theorem 1.3 of the book, detailed balance also gives that the stationary process looks the same run backwards in time.

Formalization. The Chapter 2 results (Theorems 2.3–2.5) are formalized and proved on the platform, at the level of transition rates, in the KellyStochasticNetworks series. Theorems 6.1 and 6.2 are not formalized anywhere to our knowledge. The new work is the detailed-balance verification with the receiving-colony factor, the normalization over the finite set S\mathcal{S}S in the closed case, and the summation over the countable space NJ\mathbb{N}^JNJ with the marginal computation that expresses independence in the open case.

Difficulty

The equilibrium equations of a migration process with blocking have no simple solution in general (p. 135); a direct attack on the full balance equations does not close. Detailed balance is a local condition, but the formal statement has to be careful with transitions that do not exist: a move out of an empty colony, a departure that would make a count negative, and the coincidences between operators at the boundary (Tjkn=T⋅knT_{jk}n = T_{\cdot k}nTjk​n=T⋅k​n when nj=0n_j = 0nj​=0). In the open case the bookkeeping is in infinite sums: positivity and normalization need convergence of each colony series, and independence needs the marginal of π\piπ on one colony to be computed as a sum over the remaining J−1J-1J−1 coordinates.

Formalization scope

Colonies are Fin J; states are Fin J → ℕ; rates are real-valued functions of two states, assembled as sums of indicator terms over the possible transitions, reusing the operators Tjk, Tout, Tin and the predicates DetailedBalance, FullBalance of the published definitions KellyStochasticNetworks_Migration and KellyStochasticNetworks_Balance. Transitions out of an empty colony carry the factor φj(0)=0\varphi_j(0) = 0φj​(0)=0 and so vanish. The hypotheses λjj=0\lambda_{jj} = 0λjj​=0, non-negativity of the rates, positivity of φj(n)\varphi_j(n)φj​(n) (n>0n>0n>0), ψj(n)\psi_j(n)ψj​(n) (n≥0n\ge0n≥0) and αj\alpha_jαj​, and the book's irreducibility requirements are binders of each theorem.

The statements are at the level of transition rates: "reversible" is read as detailed balance of the equilibrium distribution, and "equilibrium distribution" as positive, summing to one, and satisfying the equilibrium equations. The stochastic process itself is not constructed. In the closed case the distribution lives on NJ\mathbb{N}^JNJ and vanishes off S\mathcal{S}S, which is equivalent because every transition preserves ∑jnj\sum_j n_j∑j​nj​; J≥1J \ge 1J≥1 is assumed so that S\mathcal{S}S is nonempty. In the open case stationarity is the convergence of each colony series, carried as HasSum hypotheses with sums gjg_jgj​.

The normalizing constant is never free: π≡0\pi \equiv 0π≡0 satisfies detailed balance, so a statement that leaves BBB unconstrained, or omits the factor ψk(nk)\psi_k(n_k)ψk​(nk​) from (6.2), (6.6) and (6.3), is not this theorem. Contributions welcome: proofs of Theorem 6.1 and 6.2, reusable lemmas on detailed balance for indicator-sum rates, and on products of summable families over Fin J → ℕ.

Selected references

  • F. P. Kelly, Reversibility and Stochastic Networks, Wiley, 1979; reprinted Cambridge University Press, 2011. Chapters 2 and 6. http://www.statslab.cam.ac.uk/~frank/BOOKS/kelly_book.html
  • J. F. C. Kingman, Markov population processes, Journal of Applied Probability 6 (1969), 1–18. https://doi.org/10.2307/3212273
  • F. P. Kelly, Networks of queues, Advances in Applied Probability 8 (1976), 416–432. https://doi.org/10.2307/1426136
  • P. Whittle, Nonlinear migration processes, Bulletin of the International Statistical Institute 42 (1967).
  • F. P. Kelly and E. Yudovina, Stochastic Networks, Cambridge University Press, 2014, Chapter 2. http://www.statslab.cam.ac.uk/~frank/STOCHNET/
9 thms4 active usersReviewed
Operations ResearchProbabilityStochastic Systems+1·Captain: mikedeng1

Approximation Algorithms for Stochastic Inventory Control Models 1: The Dual-Balancing Policy Costs at Most Twice the OptimumResearch Paper

Motivation

Periodic-review inventory control with backorders is one of the basic models of operations research: in each period a manager decides how much to order, orders arrive after a lead time, unmet demand is backlogged at a penalty, and stock left over is charged a holding cost. When demands in different periods are independent, dynamic programming yields an optimal base-stock policy and computing it is tractable. In practice demands are correlated and forecasts evolve over time, for example under the martingale model of forecast evolution (Heath and Jackson, 1994, doi:10.1080/07408179408966604). The dynamic program then has to range over all possible information states, whose number is typically exponential in the input (Zipkin, 2000), so optimal policies are out of reach and the heuristics in use came without performance guarantees.

Levi, Pál, Roundy and Shmoys (Math. Oper. Res. 32(2):284–302, 2007) gave the first policy for this model with a worst-case guarantee that holds for arbitrary correlated, nonstationary demand distributions: the dual-balancing policy costs at most twice the optimum in expectation. The analysis rests on a marginal cost accounting that charges each order, at the time it is placed, all the holding cost its units will ever incur. This mission formalizes that guarantee.

Setting

There are TTT periods t=1,…,Tt = 1, \dots, Tt=1,…,T and a known lead time L≥0L \ge 0L≥0: an order placed in period ttt arrives in period t+Lt + Lt+L. Period ttt has a per-unit holding cost ht≥0h_t \ge 0ht​≥0 and a per-unit backlogging penalty pt≥0p_t \ge 0pt​≥0. Ordering costs are zero (ct=0c_t = 0ct​=0), which is the standing assumption of the paper's §4. The initial data are the net inventory ni0ni_0ni0​ and the pipeline orders q1−L,…,q0≥0q_{1-L}, \dots, q_0 \ge 0q1−L​,…,q0​≥0.

Demands D1,…,DTD_1, \dots, D_TD1​,…,DT​ are nonnegative random variables on a probability space with a filtration (Ft)(\mathcal F_t)(Ft​); Ft\mathcal F_tFt​ is the information at the beginning of period ttt, and DtD_tDt​ is Ft+1\mathcal F_{t+1}Ft+1​-measurable. A feasible policy PPP places orders QtP≥0Q^P_t \ge 0QtP​≥0 that are Ft\mathcal F_tFt​-measurable. Write D[s,t]=∑j=stDjD_{[s,t]} = \sum_{j=s}^t D_jD[s,t]​=∑j=st​Dj​ (with Dj=0D_j = 0Dj​=0 for j≤0j \le 0j≤0), Xt=ni0+∑j=1−Lt−1Qj−D[1,t−1]X_t = ni_0 + \sum_{j=1-L}^{t-1} Q_j - D_{[1,t-1]}Xt​=ni0​+∑j=1−Lt−1​Qj​−D[1,t−1]​ for the inventory position before ordering and Yt=Xt+QtY_t = X_t + Q_tYt​=Xt​+Qt​ after ordering.

The marginal holding cost of period ttt is the holding cost that the units ordered in ttt incur until the end of the horizon, and the marginal backlogging cost is the penalty incurred one lead time later:

HtP=∑j=t+LThj (QtP−(D[t,j]−XtP)+)+,ΠtP=pt+L (D[t,t+L]−YtP)+.H^P_t = \sum_{j=t+L}^{T} h_j\,\bigl(Q^P_t - (D_{[t,j]} - X^P_t)^+\bigr)^+, \qquad \Pi^P_t = p_{t+L}\,\bigl(D_{[t,t+L]} - Y^P_t\bigr)^+ .HtP​=j=t+L∑T​hj​(QtP​−(D[t,j]​−XtP​)+)+,ΠtP​=pt+L​(D[t,t+L]​−YtP​)+.

The cost of PPP is C(P)=∑t=1T−L(HtP+ΠtP)\mathcal C(P) = \sum_{t=1}^{T-L}(H^P_t + \Pi^P_t)C(P)=∑t=1T−L​(HtP​+ΠtP​); by Eq. (3) it differs from the total holding and backlogging cost only by a policy-independent nonnegative term.

A dual-balancing policy BBB orders nothing after period T−LT - LT−L, and in each period t≤T−Lt \le T - Lt≤T−L orders the quantity that balances the two conditional expected marginal costs:

E[HtB∣Ft]=E[ΠtB∣Ft]almost surely.E\bigl[H^B_t \mid \mathcal F_t\bigr] = E\bigl[\Pi^B_t \mid \mathcal F_t\bigr] \quad\text{almost surely.}E[HtB​∣Ft​]=E[ΠtB​∣Ft​]almost surely.

Formalization targets

Goal: Theorem 4.1

For every dual-balancing policy BBB and every feasible policy PPP,

E[C(B)]  ≤  2 E[C(P)].E[\mathcal C(B)] \;\le\; 2\,E[\mathcal C(P)] .E[C(B)]≤2E[C(P)].

The paper writes P=OPTP = OPTP=OPT; quantifying over all feasible PPP is the same statement whenever an optimum exists and needs no existence assumption.

Milestones

  1. Lemma 4.1. E[C(B)]=2∑t=1T−LE[Zt]E[\mathcal C(B)] = 2\sum_{t=1}^{T-L}E[Z_t]E[C(B)]=2∑t=1T−L​E[Zt​] with Zt=E[HtB∣Ft]Z_t = E[H^B_t \mid \mathcal F_t]Zt​=E[HtB​∣Ft​].
  2. Lemma 4.2. With TH={t:YtB<YtP}\mathcal T_H = \{t : Y^B_t < Y^P_t\}TH​={t:YtB​<YtP​}, ∑t∈THHtB≤∑t=1T−LHtP\sum_{t\in\mathcal T_H} H^B_t \le \sum_{t=1}^{T-L} H^P_t∑t∈TH​​HtB​≤∑t=1T−L​HtP​ on every realization.
  3. Lemma 4.3. With TΠ={t:YtB≥YtP}\mathcal T_\Pi = \{t : Y^B_t \ge Y^P_t\}TΠ​={t:YtB​≥YtP​}, ∑t∈TΠΠtB≤∑t=1T−LΠtP\sum_{t\in\mathcal T_\Pi} \Pi^B_t \le \sum_{t=1}^{T-L} \Pi^P_t∑t∈TΠ​​ΠtB​≤∑t=1T−L​ΠtP​ on every realization.

Two further items are not milestones. Eq. (3) states that, along every realization, the period-by-period holding and backlogging cost equals ∑t=1−L0Πt+H(−∞,0]+∑t=1T−L(Ht+Πt)\sum_{t=1-L}^{0}\Pi_t + H_{(-\infty,0]} + \sum_{t=1}^{T-L}(H_t + \Pi_t)∑t=1−L0​Πt​+H(−∞,0]​+∑t=1T−L​(Ht​+Πt​), which is why the cost of Eq. (4) is the right objective. The other states that a dual-balancing policy exists when hT>0h_T > 0hT​>0 and the demands are integrable, so the goal is not about an empty class.

Significance

The theorem gives a policy that is computable period by period, by a one-dimensional search, with a factor-two guarantee that holds for every joint demand distribution, including correlated, nonstationary and forecast-driven ones, where the optimal policy cannot be computed. The constant is tight: the paper exhibits instances where the ratio tends to two. The second mission of this series treats the stochastic lot-sizing problem of the same paper, which uses the same marginal cost accounting.

The result is proved in the paper; no machine-checked proof of it is known. Formalizing it produces a reusable model of the periodic-review backlogging system with lead times and adapted policies, a verified marginal cost identity, and a formal approximation guarantee for a stochastic inventory policy. The pathwise comparison lemmas are stated for arbitrary pairs of order sequences and so apply to other balancing-type policies.

Difficulty

The obvious attempt compares the two policies period by period. That fails: in a given period the dual-balancing policy may hold far more or far less inventory than the comparison policy, and neither the holding nor the backlogging cost of one period is bounded by the comparator's cost in that period. The comparison only works after re-charging holding costs to the period in which the units were ordered, which requires the identity Eq. (3) to be established exactly, including the pipeline units, the initial stock and the lead-time shift. The probabilistic step then needs the random index sets TH\mathcal T_HTH​ and TΠ\mathcal T_\PiTΠ​ to be determined by the information of period ttt, so that conditioning on Ft\mathcal F_tFt​ commutes with the indicators; this is where the nonanticipativity of both policies enters. The existence of a balancing quantity needs a measurable selection from conditional laws, and it fails without a positive late holding cost.

Formalization scope

  • Periods are integers (ℤ). Orders and demands are functions ℤ → Ω → ℝ; only periods 1,…,T1, \dots, T1,…,T are read, and the pipeline qtq_tqt​ is substituted for t≤0t \le 0t≤0.
  • Ordering costs are ct=0c_t = 0ct​=0 and there is no discounting, as in the paper's §4; the reduction of §4.6 from general instances is not formalized. The lead time LLL is general.
  • Information is an arbitrary Filtration ℤ to which demands are adapted with a one-period lag; the paper's information vectors are a special case, and randomized policies are covered when their randomness is part of the information.
  • Expected costs are lower Lebesgue integrals in [0,∞][0,\infty][0,∞], so an infinite expected cost is never read as 000.
  • The balancing condition carries integrability of HtBH^B_tHtB​ and ΠtB\Pi^B_tΠtB​, so a conditional expectation of a non-integrable cost (which Mathlib sets to 000) cannot satisfy it vacuously. The existence item rules out an empty policy class.
  • Lemmas 4.2 and 4.3 are pathwise and do not use the balancing rule. The comparator totals are the marginal totals of Eq. (4), which is the stronger reading.
  • Eq. (2) prints Xt+LX_{t+L}Xt+L​ and its restatement on p. 292 prints ptp_tpt​; both are typos, and the formalization uses XtX_tXt​ and pt+Lp_{t+L}pt+L​.

A complete development needs finite-sum manipulations for Eq. (3) and Lemma 4.2, conditional expectation (tower property, pulling out bounded Ft\mathcal F_tFt​-measurable factors) for Lemma 4.1 and the goal, and regular conditional distributions with a measurable selection for the existence item. Theorem 4.2 (the randomized policy for integer demands) is outside this mission.

Selected references

  • R. Levi, M. Pál, R. O. Roundy, D. B. Shmoys, Approximation Algorithms for Stochastic Inventory Control Models, Mathematics of Operations Research 32(2):284–302, 2007. doi:10.1287/moor.1060.0205
  • D. C. Heath, P. L. Jackson, Modeling the evolution of demand forecasts with application to safety stock analysis in production/distribution systems, IIE Transactions 26(3):17–30, 1994. doi:10.1080/07408179408966604
  • P. H. Zipkin, Foundations of Inventory Management, McGraw-Hill, 2000. ISBN 978-0-256-11379-7.
6 thms2 active usersReviewed
Bandit AlgorithmsMachine LearningOperations Research+1·Captain: mikedeng1

The Best of Both Worlds: Stochastic and Adversarial Bandits: SAO Has Pseudo-Regret O(K log K log²β/Δ) on Stochastic Rewards and Regret Õ(√(nK)) Against Adaptive AdversariesResearch Paper

Motivation

In a multi-armed bandit problem a learner chooses one of KKK actions in each of nnn rounds and observes only the reward of the chosen action. Two models of the rewards have separate theories. In the stochastic model, each arm pays independent draws from a fixed distribution; algorithms such as UCB1 (Auer, Cesa-Bianchi & Fischer 2002) have regret of order ∑ilog⁡(n)/Δi\sum_i \log(n)/\Delta_i∑i​log(n)/Δi​, logarithmic in nnn. In the adversarial model, an adversary chooses the rewards; Exp3 and its variants (Auer, Cesa-Bianchi, Freund & Schapire 2002) have regret of order nK\sqrt{nK}nK​, which is optimal there. An algorithm tuned for one model fails in the other: stochastic algorithms can suffer linear regret against an adversary, and adversarial algorithms pay n\sqrt nn​ even when the rewards are i.i.d.

Bubeck and Slivkins (arXiv:1202.4473, COLT 2012) asked whether one algorithm can be near-optimal in both models without knowing which one it faces. They answered yes with the algorithm SAO. This result started the "best of both worlds" line of work on bandits. Later contributions include EXP3++ (Seldin & Slivkins 2014) and Tsallis-INF (Zimmert & Seldin 2021).

Setting

There are K≥2K\ge2K≥2 arms and n≥Kn\ge Kn≥K rounds. On round ttt the algorithm draws an arm ItI_tIt​ from a probability vector pt=(p1,t,…,pK,t)p_t=(p_{1,t},\dots,p_{K,t})pt​=(p1,t​,…,pK,t​) computed from the history it has observed. At the same time a reward vector gt∈[0,1]Kg_t\in[0,1]^Kgt​∈[0,1]K is fixed, and the algorithm observes only gIt,tg_{I_t,t}gIt​,t​.

  • Adversarial model. The vector gtg_tgt​ is chosen by an adaptive adversary: a function of the arms I1,…,It−1I_1,\dots,I_{t-1}I1​,…,It−1​ played earlier, but not of ItI_tIt​. The regret is Rn=max⁡i∑t=1ngi,t−∑t=1ngIt,tR_n=\max_i\sum_{t=1}^n g_{i,t}-\sum_{t=1}^n g_{I_t,t}Rn​=maxi​∑t=1n​gi,t​−∑t=1n​gIt​,t​.
  • Stochastic model. There are distributions ν1,…,νK\nu_1,\dots,\nu_Kν1​,…,νK​ on [0,1][0,1][0,1] with means μi\mu_iμi​, and all gi,t∼νig_{i,t}\sim\nu_igi,t​∼νi​ are independent. The pseudo-regret is R‾n=∑t=1n(max⁡iμi−μIt)\overline R_n=\sum_{t=1}^n(\max_i\mu_i-\mu_{I_t})Rn​=∑t=1n​(maxi​μi​−μIt​​). The gap of arm iii is Δi=max⁡jμj−μi\Delta_i=\max_j\mu_j-\mu_iΔi​=maxj​μj​−μi​, and the minimal gap is Δ=min⁡i:Δi>0Δi\Delta=\min_{i:\Delta_i>0}\Delta_iΔ=mini:Δi​>0​Δi​.

The analysis uses importance-weighted estimates H~i,t=1t∑s≤tgi,s1{Is=i}/pi,s\widetilde H_{i,t}=\frac1t\sum_{s\le t}g_{i,s}\mathbb 1_{\{I_s=i\}}/p_{i,s}Hi,t​=t1​∑s≤t​gi,s​1{Is​=i}​/pi,s​, the sample means H^i,t\widehat H_{i,t}Hi,t​, the averages Hi,t=1t∑s≤tgi,sH_{i,t}=\frac1t\sum_{s\le t}g_{i,s}Hi,t​=t1​∑s≤t​gi,s​, and the play counts Ti(t)T_i(t)Ti​(t).

SAO (Algorithm 1 of the paper) takes a parameter β>1\beta>1β>1. It keeps a set of active arms, initially all arms, and samples them uniformly at first. On each round it applies a test, (12), that deactivates an arm whose estimate H~i,t\widetilde H_{i,t}Hi,t​ falls far below the best active one. The probability of a deactivated arm then decays as qiτi/tq_i\tau_i/tqi​τi​/t, where τi\tau_iτi​ is the deactivation time and qiq_iqi​ the arm's probability at that moment. Three further tests, (13)–(15), check that the observations stay consistent with stochastic rewards. If any of them fails on round τ0\tau_0τ0​, SAO switches permanently to the adversarial algorithm Exp3.P (Bubeck & Cesa-Bianchi 2012, Fig. 3.1) for the remaining rounds.

Formalization targets

Goal: Theorem 4.1, high-probability form

For every δ∈(0,1)\delta\in(0,1)δ∈(0,1) let β=10Kn3δ−1\beta=10Kn^3\delta^{-1}β=10Kn3δ−1. With probability at least 1−δ1-\delta1−δ, SAO with parameter β\betaβ satisfies, in the stochastic model (whenever some arm has Δi>0\Delta_i>0Δi​>0),

R‾n≤260K(1+log⁡K)log⁡2(β)Δ,\overline R_n\le\frac{260K(1+\log K)\log^2(\beta)}{\Delta},Rn​≤Δ260K(1+logK)log2(β)​,

and, against every adaptive adversary with rewards in [0,1][0,1][0,1],

Rn≤60(1+log⁡K)(1+log⁡n)nKlog⁡(β)+5K2log⁡2(β)+200K2log⁡2(β).R_n\le60(1+\log K)(1+\log n)\sqrt{nK\log(\beta)+5K^2\log^2(\beta)}+200K^2\log^2(\beta).Rn​≤60(1+logK)(1+logn)nKlog(β)+5K2log2(β)​+200K2log2(β).

Milestones

The milestones follow the paper's proof in order:

  • Freedman's inequality (Theorem 4.3) in the paper's two-sided form, and its variance-adaptive form, Lemma 4.4.
  • The concentration lemmas for SAO's estimates (Lemmas 4.5, 4.6, 4.7) and the Exp3.P phase (Lemma 4.8).
  • The two good events of §4.1, (21)–(25).
  • The deterministic consequences on those events: Exp3.P is never started in the stochastic model; suboptimal arms are deactivated by time 260Klog⁡(β)/Δi2260K\log(\beta)/\Delta_i^2260Klog(β)/Δi2​; ∑iqi≤1+log⁡K\sum_iq_i\le1+\log K∑i​qi​≤1+logK, (27); and the adversarial regret bound of §4.3.
  • The two halves of Theorem 4.1.

Significance

The theorem shows that the stochastic and adversarial regret rates are not in conflict. A single algorithm, with no information about the model, gets O(Klog⁡Klog⁡2(n/δ)/Δ)O(K\log K\log^2(n/\delta)/\Delta)O(KlogKlog2(n/δ)/Δ) pseudo-regret on stochastic rewards and O~(nK)\tilde O(\sqrt{nK})O~(nK​) regret against adaptive adversaries. Each rate is within polylogarithmic factors of optimal for its model. Later algorithms improved the logarithmic factors and removed the explicit switching, but they are compared against this result.

The theorem is proved in the paper. It is not known to have a machine-checked proof. Formalizing it requires a precise model of an adaptive adversary interacting with a randomized algorithm, martingale concentration with random variance (Lemma 4.4), and an exact statement of SAO including its boundary cases. The pieces are reusable: the interaction model, the estimators, Exp3.P and its high-probability guarantee all apply to other adversarial bandit results.

Difficulty

Neither standard analysis carries over. In the stochastic model, SAO's sampling probabilities are random and depend on the past, and a deactivated arm's probability keeps changing. Hoeffding-type bounds for a fixed sampling scheme therefore do not apply to H~i,t\widetilde H_{i,t}Hi,t​. The variance of the importance-weighted estimate grows like ∑s1/pi,s\sum_s1/p_{i,s}∑s​1/pi,s​, which is controlled only through the algorithm's own schedule (16). This is why Lemma 4.5 has the two-part radius with max⁡(t−τi,0)/(qiτit)\max(t-\tau_i,0)/(q_i\tau_it)max(t−τi​,0)/(qi​τi​t). In the adversarial model, the deterministic argument has to show that whenever the consistency tests pass, the regret accumulated before the switch is already small, for an adversary that adapts to the arms played. A union bound over all quantities, all arms and all times (§4.1) is needed before any deterministic reasoning, so every constant in the event matters.

Formalization scope

All declarations live in the namespace BestBothWorlds.SAO.

  • Arms and paths. Arms are Fin K and rounds are 1,…,n1,\dots,n1,…,n. An arm path is Fin n → Fin K.
  • Algorithms and adversaries. An algorithm is a deterministic map from the observed history to a probability vector. A deterministic adaptive adversary is a map from the list of earlier arms to a reward vector; randomized adversaries are mixtures of these.
  • Probabilities. For a fixed adversary, the probability of an event is ∑I∈E∏tpIt,t\sum_{I\in E}\prod_tp_{I_t,t}∑I∈E​∏t​pIt​,t​. In the stochastic model this is integrated against the product law of the reward table.
  • Logarithms and constants. Real.log is the natural logarithm. All constants of Theorem 4.1 are explicit, with β=10Kn3δ−1\beta=10Kn^3\delta^{-1}β=10Kn3δ−1.
  • SAO. It is defined exactly as Algorithm 1. Arms are tested in order within a round, and the active set changes during the loop. Test (13) is false when Ti(t)=0T_i(t)=0Ti​(t)=0, and test (14) is false when τi=1\tau_i=1τi​=1.
  • Exp3.P. After the switch, Exp3.P runs from scratch for n−τ0n-\tau_0n−τ0​ rounds. Its parameters are those of Bubeck–Cesa-Bianchi Theorem 3.2 with confidence K/βK/\betaK/β, and γ\gammaγ and βP\beta_{\mathrm P}βP​ are clipped at 111.
  • §4 notation. τ0\tau_0τ0​, τi←min⁡(τi,τ0)\tau_i\leftarrow\min(\tau_i,\tau_0)τi​←min(τi​,τ0​) and qi=pi,min⁡(τi,τ0)q_i=p_{i,\min(\tau_i,\tau_0)}qi​=pi,min(τi​,τ0​)​ are computed from the run, never assumed.

A trivializing formalization is ruled out. The goal's hypotheses concern only the instance (KKK, nnn, δ\deltaδ, the distributions or the adversary). The algorithm's quantities (τ0\tau_0τ0​, τi\tau_iτi​, qiq_iqi​, the sampling probabilities) are computed by the definition of SAO and are never free variables or hypotheses. The adversarial half covers adaptive adversaries, not only oblivious reward tables.

The expectation form of Theorem 4.1 (O(⋅)O(\cdot)O(⋅) bounds with β=n4\beta=n^4β=n4), Theorem 1.1 and the two-armed warm-up of §3 are out of scope. Proofs of any milestone are welcome, as are alternative proofs of the concentration lemmas from Mathlib's martingale library.

Selected references

  • S. Bubeck and A. Slivkins, The best of both worlds: stochastic and adversarial bandits, COLT 2012; arXiv:1202.4473v1. https://arxiv.org/abs/1202.4473
  • S. Bubeck and N. Cesa-Bianchi, Regret Analysis of Stochastic and Nonstochastic Multi-armed Bandit Problems, Foundations and Trends in Machine Learning 5(1), 2012. https://doi.org/10.1561/2200000024
  • D. A. Freedman, On tail probabilities for martingales, Annals of Probability 3(1), 1975. https://doi.org/10.1214/aop/1176996452
  • P. Auer, N. Cesa-Bianchi and P. Fischer, Finite-time analysis of the multiarmed bandit problem, Machine Learning 47, 2002. https://doi.org/10.1023/A:1013689704352
  • P. Auer, N. Cesa-Bianchi, Y. Freund and R. E. Schapire, The nonstochastic multiarmed bandit problem, SIAM Journal on Computing 32(1), 2002. https://doi.org/10.1137/S0097539701398375
  • Y. Seldin and A. Slivkins, One practical algorithm for both stochastic and adversarial bandits, ICML 2014. https://proceedings.mlr.press/v32/seldinb14.html
  • J. Zimmert and Y. Seldin, Tsallis-INF: an optimal algorithm for stochastic and adversarial bandits, JMLR 22, 2021. https://jmlr.org/papers/v22/19-753.html
20 thms2 active usersReviewed
Markov ChainOperations ResearchProbability+1·Captain: mikedeng1

Reversibility and Stochastic Networks IV: Symmetric Queues with Gamma-Mixture Service RequirementsTextbook

Why symmetric queues

The classical product-form results for queueing networks (Jackson, Kelly, Baskett–Chandy–Muntz–Palacios) assume exponentially distributed service requirements, because then the state of a queue need not record how much service each customer has received. Real service times are rarely exponential: telephone call lengths, job sizes in time-shared computers and web transfers are far from it. Symmetric queues, introduced in §3.3 of F. P. Kelly, Reversibility and Stochastic Networks (Wiley, 1979), form the class of single queues for which the stationary distribution of the number of customers, and of their classes, depends on the service requirement distribution only through its mean. This property, insensitivity, is what makes Erlang's loss formula valid for arbitrarily distributed call lengths (Kelly, p. 79), and it is the reason processor-sharing, last-come-first-served preemptive and infinite-server stations may appear with general service in product-form networks.

Timeline. Sevastyanov (1957) proved that Erlang's loss formula holds for arbitrarily distributed call lengths. Kelly (1975, 1976) introduced queues with customers of different types whose effort and arrival-position functions coincide, and showed product form for networks of them with non-exponential service built from exponential stages; Barbour (1976) extended the method of stages; Baskett, Chandy, Muntz and Palacios (1975) gave product form for networks containing processor-sharing, LCFS-preemptive and infinite-server stations with phase-type service. Kelly's 1979 book presents the symmetric queue in the form formalized here.

Setting

A symmetric queue holds customers in positions 1,2,…,n1, 2, \dots, n1,2,…,n, where nnn is the number present. It operates as follows (Kelly, p. 72):

  1. the service requirement of a customer is a random variable whose distribution may depend on the class of the customer;
  2. a total service effort is supplied at rate ϕ(n)\phi(n)ϕ(n), with ϕ(n)>0\phi(n) > 0ϕ(n)>0 for n>0n > 0n>0;
  3. a proportion γ(l,n)\gamma(l, n)γ(l,n) of this effort, ∑l=1nγ(l,n)=1\sum_{l=1}^n \gamma(l, n) = 1∑l=1n​γ(l,n)=1, goes to the customer in position lll; when he leaves, the customers in positions l+1,…,nl+1, \dots, nl+1,…,n move down by one;
  4. an arriving customer moves into position l∈{1,…,n+1}l \in \{1, \dots, n+1\}l∈{1,…,n+1} with probability γ(l,n+1)\gamma(l, n+1)γ(l,n+1) — the same function — and the customers in positions l,…,nl, \dots, nl,…,n move up by one.

Server-sharing (γ(l,n)=1/n\gamma(l, n) = 1/nγ(l,n)=1/n), the stack (γ(n,n)=1\gamma(n, n) = 1γ(n,n)=1, last come first served preemptive), the queue with no waiting room and the infinite-server queue are examples (pp. 73–74).

Customers of class ccc arrive in a Poisson stream of rate ν(c)\nu(c)ν(c). On arrival a class-ccc customer receives a refined class (c,z)(c, z)(c,z) with probability p(c,z)p(c, z)p(c,z), ∑zp(c,z)=1\sum_z p(c, z) = 1∑z​p(c,z)=1, and then needs w(c,z)≥1w(c, z) \ge 1w(c,z)≥1 independent stages of service, each exponentially distributed with mean d(c,z)>0d(c, z) > 0d(c,z)>0. The class-ccc service requirement is therefore a mixture of gamma distributions with mean

a(c)=∑zp(c,z) w(c,z) d(c,z),a(c) = \sum_z p(c, z)\, w(c, z)\, d(c, z),a(c)=z∑​p(c,z)w(c,z)d(c,z),

and the work arriving per unit time is a=∑cν(c)a(c)a = \sum_c \nu(c) a(c)a=∑c​ν(c)a(c). The record of the customer in position lll is c(l)=(c(l),z(l),u(l))\mathbf c(l) = (c(l), z(l), u(l))c(l)=(c(l),z(l),u(l)), u(l)u(l)u(l) the stage in progress, and c=(c(1),…,c(n))\mathbf c = (\mathbf c(1), \dots, \mathbf c(n))c=(c(1),…,c(n)) is a Markov process. Its transitions are: an arrival of class (c,z)(c,z)(c,z) into position lll at stage 111, at rate ν(c)p(c,z)γ(l,n+1)\nu(c)p(c,z)\gamma(l, n+1)ν(c)p(c,z)γ(l,n+1); and, at rate ϕ(n)γ(l,n)/d(c(l),z(l))\phi(n)\gamma(l,n)/d(c(l),z(l))ϕ(n)γ(l,n)/d(c(l),z(l)), completion of the current stage of the customer in position lll, which moves him to the next stage or, after stage w(c(l),z(l))w(c(l), z(l))w(c(l),z(l)), out of the queue. The normalizing constant is

b−1=∑n=0∞an∏l=1nϕ(l).(3.15)b^{-1} = \sum_{n=0}^{\infty} \frac{a^n}{\prod_{l=1}^n \phi(l)}. \tag{3.15}b−1=n=0∑∞​∏l=1n​ϕ(l)an​.(3.15)

Formalization targets

Goal: Theorem 3.8

When (3.15) converges, the distribution

π(c)=b∏l=1nν(c(l)) p(c(l),z(l)) d(c(l),z(l))ϕ(l)(3.18)\pi(\mathbf c) = b \prod_{l=1}^n \frac{\nu(c(l))\, p(c(l), z(l))\, d(c(l), z(l))}{\phi(l)} \tag{3.18}π(c)=bl=1∏n​ϕ(l)ν(c(l))p(c(l),z(l))d(c(l),z(l))​(3.18)

is the equilibrium distribution of c\mathbf cc, and under it

P(n customers)=b an∏l=1nϕ(l),P(classes c1,…,cn∣n)=∏l=1nν(cl) a(cl)a,\mathbb P(n \text{ customers}) = \frac{b\,a^n}{\prod_{l=1}^n \phi(l)}, \qquad \mathbb P(\text{classes } c_1, \dots, c_n \mid n) = \prod_{l=1}^n \frac{\nu(c_l)\, a(c_l)}{a},P(n customers)=∏l=1n​ϕ(l)ban​,P(classes c1​,…,cn​∣n)=l=1∏n​aν(cl​)a(cl​)​,

and the queue is quasi-reversible with respect to the classification ccc and to (c,z)(c, z)(c,z): from every state, the rate of arrivals of each class does not depend on the state, both for the process and its time reversal (relations (3.8) and (3.10) of p. 67).

Milestones

  • Eqs. (3.14)–(3.15): the case of one refinement per class, π(c)=b∏lν(c(l))d(c(l))/ϕ(l)\pi(\mathbf c) = b\prod_l \nu(c(l))d(c(l))/\phi(l)π(c)=b∏l​ν(c(l))d(c(l))/ϕ(l) with a=∑cν(c)d(c)w(c)a = \sum_c \nu(c)d(c)w(c)a=∑c​ν(c)d(c)w(c).
  • Eqs. (3.16)–(3.17): in that case, the law of nnn, and given nnn independent positions of class ccc with probability ν(c)d(c)w(c)/a\nu(c)d(c)w(c)/aν(c)d(c)w(c)/a and uniform stage.
  • Eq. (3.18): the equilibrium distribution under gamma-mixture service.
  • Lemma 3.9: mixtures of gamma distributions approximate, at continuity points, the distribution function of any positive random variable.

Significance

Theorem 3.8 gives the stationary law of a symmetric queue in closed form and shows that it depends on the service requirement distributions only through their means a(c)a(c)a(c). Quasi-reversibility (part (iii)) is the property that lets symmetric queues be placed in networks: Kelly's §3.2 shows that a network of quasi-reversible queues has a product-form equilibrium, so Theorem 3.8 is the single-queue input to product-form networks with processor-sharing, LCFS-preemptive and infinite-server stations and class-dependent, non-exponential service. Lemma 3.9 is the approximation step behind the extension to arbitrary service distributions (Theorem 3.10).

These results are classical and proved in the book. To our knowledge none of them is machine-checked; Mathlib has the gamma distribution and distribution functions but no queueing theory. The mission produces a formal model of the symmetric queue as a countable-state Markov process, its equilibrium distribution, the insensitive marginals and the rate characterization of quasi-reversibility, all reusable by the chapter on networks of quasi-reversible queues.

Difficulty

The obvious first attempt, detailed balance, fails: the stage process is not reversible in general, since an intermediate stage completion has no transition back. The equilibrium equations must be verified in full, over a countable state space in which a single state is reached from infinitely many others. The symmetry condition γ≡δ\gamma \equiv \deltaγ≡δ is essential and must enter the argument: without it (for first-come-first-served, say) the distribution (3.18) is false for non-exponential service. Positions shift on every arrival and departure, so the bookkeeping of which list results from which event, including coincidences when neighbouring customers have identical records, is the main formal burden. The marginal computations sum (3.18) over lists of records, where convergence must be tracked, and Lemma 3.9 needs an explicit construction of approximating gamma mixtures.

Formalization scope

  • The state is a List of customer records (class, refined class, stage); list index iii is position l=i+1l = i + 1l=i+1, and γ(l,n)\gamma(l, n)γ(l,n), ϕ(n)\phi(n)ϕ(n) keep the book's 111-based indexing. The state space consists of the lists whose records have ν(c)p(c,z)>0\nu(c)p(c,z) > 0ν(c)p(c,z)>0 and 1≤u≤w(c,z)1 \le u \le w(c, z)1≤u≤w(c,z): records of a refined class arriving at rate zero are unreachable and excluded.
  • Classes C\mathcal CC and refinements Z\mathcal ZZ are arbitrary countable types. All infinite sums are HasSum or tsum with explicit convergence hypotheses: ∑cν(c)<∞\sum_c \nu(c) < \infty∑c​ν(c)<∞ (finite exit rates, the book's standing assumption of §1.1), convergence of a(c)a(c)a(c), of aaa, and of (3.15). "Equilibrium distribution" means positive, summing to one, and satisfying the equilibrium equations with convergent series.
  • The statement is rate level: (3.18) is shown to satisfy the equilibrium equations of the stage process, and quasi-reversibility is its rate characterization (3.8), (3.10). That these describe the stationary process and its time reversal is the book's Chapter 1 and is not reformalized.
  • Trivializing readings ruled out: the goal keeps ppp, www and ddd general (not w≡1w \equiv 1w≡1, which would make the result Section 3.1's exponential case); the arrival position uses the same γ\gammaγ as the service split; and in Lemma 3.9 the approximants must be genuine gamma mixtures with integer shapes, which a point mass is not.
  • Not planned: Theorem 3.10 (arbitrary service distributions; only an outline of proof and a continuous state space the book does not construct) and Theorem 3.11 (reversibility of the number in queue, a non-Markov process).

Welcome contributions: proofs of the milestones, lemmas about List.insertIdx/List.eraseIdx bookkeeping for queues with positions, and summation over lists of records.

Selected references

  • F. P. Kelly, Reversibility and Stochastic Networks, Wiley, Chichester, 1979, §3.3, pp. 72–82. https://www.statslab.cam.ac.uk/~frank/BOOKS/kelly_book.html
  • F. P. Kelly, Networks of queues with customers of different types, Journal of Applied Probability 12 (1975), 542–554. https://www.jstor.org/journal/japplprob
  • F. P. Kelly, Networks of queues, Advances in Applied Probability 8 (1976), 416–432. https://www.jstor.org/journal/advaapplprob
  • A. D. Barbour, Networks of queues and the method of stages, Advances in Applied Probability 8 (1976), 584–591. https://www.jstor.org/journal/advaapplprob
  • B. A. Sevastyanov, An ergodic theorem for Markov processes and its application to telephone systems with refusals, Theory of Probability and its Applications 2 (1957), 104–112.
  • F. Baskett, K. M. Chandy, R. R. Muntz, F. G. Palacios, Open, closed, and mixed networks of queues with different classes of customers, Journal of the ACM 22 (1975), 248–260. https://doi.org/10.1145/321879.321887
9 thms3 active usersReviewed
Markov ChainOperations ResearchProbability+1·Captain: mikedeng1

Reversibility and Stochastic Networks III: Open Networks of Queues with General Customer Routes Have Product-Form EquilibriumTextbook

Motivation

Networks of queues model systems in which jobs visit a sequence of service stations: items in a manufacturing job-shop, packets in a communication network, patients moving between hospital departments. The open migration process of Chapter 2 of F. P. Kelly, Reversibility and Stochastic Networks (Wiley, 1979), and the job-shop networks of Jackson (Jackson 1963) route a customer leaving a queue at random, independently of where he has been. That rules out the most common situation in practice: an item that has passed machines 1 and 3 must next go to machine 4, while an item that has passed machines 2 and 3 must go to machine 5.

Section 3.1 of the book removes this restriction. Customers are divided into types, a type fixes a deterministic route through the queues, and a stochastic routing rule is recovered by using one type per possible route. Within each queue, the order of service is described by two position-dependent functions, which cover first-come first-served KKK-server queues, last-come first-served, processor sharing and service in random order. Theorem 3.1 states that, for every such network, the equilibrium distribution is a product of explicit single-queue factors. This is the result behind the "Kelly network" and "Kelly-type queue" terminology of later work (Kelly 1975; Baskett, Chandy, Muntz, Palacios 1975).

Setting

There are III customer types and JJJ queues. Customers of type iii enter the system in a Poisson stream of rate ν(i)>0\nu(i)>0ν(i)>0 and visit the queues r(i,1),r(i,2),…,r(i,S(i))r(i,1),r(i,2),\dots,r(i,S(i))r(i,1),r(i,2),…,r(i,S(i)) in that order before leaving; two successive stages of a route are at different queues.

Queue jjj holds its njn_jnj​ customers in positions 1,…,nj1,\dots,n_j1,…,nj​. Each customer needs an exponentially distributed amount of service with unit mean. The queue supplies total service effort at rate ϕj(nj)\phi_j(n_j)ϕj​(nj​), with ϕj(n)>0\phi_j(n)>0ϕj​(n)>0 for n>0n>0n>0; a proportion γj(l,nj)\gamma_j(l,n_j)γj​(l,nj​) goes to the customer in position lll. An arriving customer takes position lll with probability δj(l,nj+1)\delta_j(l,n_j+1)δj​(l,nj​+1). For each n≥1n\ge1n≥1, γj(⋅,n)\gamma_j(\cdot,n)γj​(⋅,n) and δj(⋅,n)\delta_j(\cdot,n)δj​(⋅,n) are probability vectors on {1,…,n}\{1,\dots,n\}{1,…,n}.

The class of the customer in position lll of queue jjj is cj(l)=(tj(l),sj(l))c_j(l)=(t_j(l),s_j(l))cj​(l)=(tj​(l),sj​(l)), his type and the stage of his route. The state of queue jjj is cj=(cj(1),…,cj(nj))\mathbf c_j=(c_j(1),\dots,c_j(n_j))cj​=(cj​(1),…,cj​(nj​)) and the state of the network is C=(c1,…,cJ)\mathbf C=(\mathbf c_1,\dots,\mathbf c_J)C=(c1​,…,cJ​). Its transition rates q(C,D)q(\mathbf C,\mathbf D)q(C,D), displays (3.1)–(3.6), are the sums of the intensities of all events taking C\mathbf CC to D\mathbf DD: a departure from the system (intensity ϕj(nj)γj(l,nj)\phi_j(n_j)\gamma_j(l,n_j)ϕj​(nj​)γj​(l,nj​)), a move from position lll of queue jjj to position mmm of the next queue kkk (intensity ϕj(nj)γj(l,nj)δk(m,nk+1)\phi_j(n_j)\gamma_j(l,n_j)\delta_k(m,n_k+1)ϕj​(nj​)γj​(l,nj​)δk​(m,nk​+1)), and an arrival into position mmm of the first queue kkk of a route (intensity ν(i)δk(m,nk+1)\nu(i)\delta_k(m,n_k+1)ν(i)δk​(m,nk​+1)).

With αj(i,s)=ν(i)\alpha_j(i,s)=\nu(i)αj​(i,s)=ν(i) if r(i,s)=jr(i,s)=jr(i,s)=j and 000 otherwise, set

aj=∑i,sαj(i,s),bj−1=∑n=0∞ajn∏l=1nϕj(l),πj(cj)=bj∏l=1njαj(tj(l),sj(l))ϕj(l).a_j=\sum_{i,s}\alpha_j(i,s),\qquad b_j^{-1}=\sum_{n=0}^{\infty}\frac{a_j^n}{\prod_{l=1}^{n}\phi_j(l)},\qquad \pi_j(\mathbf c_j)=b_j\prod_{l=1}^{n_j}\frac{\alpha_j(t_j(l),s_j(l))}{\phi_j(l)}.aj​=i,s∑​αj​(i,s),bj−1​=n=0∑∞​∏l=1n​ϕj​(l)ajn​​,πj​(cj​)=bj​l=1∏nj​​ϕj​(l)αj​(tj​(l),sj​(l))​.

Formalization targets

Goal: Theorem 3.1 (p. 61)

If every series defining bj−1b_j^{-1}bj−1​ converges, then

π(C)=∏j=1Jπj(cj)\pi(\mathbf C)=\prod_{j=1}^{J}\pi_j(\mathbf c_j)π(C)=j=1∏J​πj​(cj​)

is positive, sums to 111 over all network states, and satisfies the equilibrium equations

π(C)∑Dq(C,D)=∑Dπ(D) q(D,C)for every C.\pi(\mathbf C)\sum_{\mathbf D}q(\mathbf C,\mathbf D)=\sum_{\mathbf D}\pi(\mathbf D)\,q(\mathbf D,\mathbf C)\quad\text{for every }\mathbf C.π(C)D∑​q(C,D)=D∑​π(D)q(D,C)for every C.

Milestones

  • Theorem 3.2 (p. 62). The time-reversed rates π(D)q(D,C)/π(C)\pi(\mathbf D)q(\mathbf D,\mathbf C)/\pi(\mathbf C)π(D)q(D,C)/π(C) are the rates of the reversed network: routes traversed backwards, γj\gamma_jγj​ and δj\delta_jδj​ interchanged.
  • Corollary 3.4 (p. 63). Queue jjj is independent of the rest of the network, is in state cj\mathbf c_jcj​ with probability πj(cj)\pi_j(\mathbf c_j)πj​(cj​), holds nnn customers with probability bjajn/∏l=1nϕj(l)b_ja_j^n/\prod_{l=1}^n\phi_j(l)bj​ajn​/∏l=1n​ϕj​(l) (3.7), and a customer in position lll is of class (i,s)(i,s)(i,s) with probability αj(i,s)/aj\alpha_j(i,s)/a_jαj​(i,s)/aj​.
  • Corollary 3.5 (p. 63). A type-iii customer reaching queue jjj at stage sss finds it in state cj\mathbf c_jcj​ with probability πj(cj)\pi_j(\mathbf c_j)πj​(cj​).
  • Lemma 3.13 (p. 89). For a multiclass queue with Poisson arrivals of rate ν(c)\nu(c)ν(c) and departure intensities ν(c)ϕc(n)\nu(c)\phi_c(\mathbf n)ν(c)ϕc​(n): reversible ⇔\Leftrightarrow⇔ quasi-reversible ⇔\Leftrightarrow⇔ Φ(n)=ϕc(n)Φ(n−ec)\Phi(\mathbf n)=\phi_c(\mathbf n)\Phi(\mathbf n-\mathbf e_c)Φ(n)=ϕc​(n)Φ(n−ec​) for some positive Φ\PhiΦ (3.26).

Significance

Theorem 3.1 gives the full joint law of a network in which routes carry memory, and its corollaries turn it into usable performance formulas: each queue behaves, in its marginal law and as seen by arriving customers, like an isolated queue fed by a Poisson stream of rate aja_jaj​, even though the actual arrival stream at queue jjj is not Poisson. Mean sojourn times along a route then follow from Little's result. Theorem 3.2 identifies the reversed process as a network of the same kind; it is the source of the departure-stream results (Corollary 3.3) and of the arrival theorem (Corollary 3.5). Lemma 3.13 isolates the condition (3.26) under which state-dependent arrival rates preserve the product form (Theorem 3.14).

The results are classical and proved in the book. None of them has a machine-checked proof: the Prove2Me catalogue holds the rate-level theorems for migration processes (Chapter 2 of Kelly–Yudovina), and open targets for the BCMP and Jackson models, which have different state descriptions. This mission adds a formal model of the position-structured multiclass network itself, with the summation over coinciding transitions that (3.2), (3.4) and (3.6) require, and product-form, reversal and arrival-theorem statements over it.

Difficulty

The obvious first attempt, detailed balance, fails: π(C)q(C,D)\pi(\mathbf C)q(\mathbf C,\mathbf D)π(C)q(C,D) and π(D)q(D,C)\pi(\mathbf D)q(\mathbf D,\mathbf C)π(D)q(D,C) differ in general, because a customer's route cannot be run backwards inside the same network (q(D,C)q(\mathbf D,\mathbf C)q(D,C) is usually 000 when q(C,D)>0q(\mathbf C,\mathbf D)>0q(C,D)>0). The equilibrium equations therefore involve, for each state, all its predecessors at once. The rates are themselves sums over coinciding transitions, so a statement about individual events does not transfer to the rates without accounting for which positions lead to the same successor state. In Lean this brings in insertion into and deletion from position lists, the relabelling of stages, and the normalization of a product over a countable space of JJJ-tuples of lists, reorganized by queue length together with the identity ∑classes at jαj=aj\sum_{\text{classes at } j}\alpha_j=a_j∑classes at j​αj​=aj​.

Formalization scope

  • Finite types and queues. Types are Fin I, queues Fin J; the book allows countably many types with ∑iν(i)<∞\sum_i\nu(i)<\infty∑i​ν(i)<∞. A network state is a function assigning to each queue a list of classes (i,s)(i,s)(i,s) with r(i,s)=jr(i,s)=jr(i,s)=j; the state space is countable and all sums over it are tsum/HasSum.
  • Indexing. Stages and list positions are 000-based in Lean; γj(l,n)\gamma_j(l,n)γj​(l,n) and δj(l,n)\delta_j(l,n)δj​(l,n) keep the book's 111-based position argument.
  • Rate level. Equilibrium means: positive, summing to 111, and satisfying the equilibrium equations (the published KellyStochasticNetworks.FullBalance). The existence of the Markov process, irreducibility and non-explosion are not formalized. "The reversed process" (Theorem 3.2) is read through the reversed rates π(D)q(D,C)/π(C)\pi(\mathbf D)q(\mathbf D,\mathbf C)/\pi(\mathbf C)π(D)q(D,C)/π(C); "the probability he finds" (Corollary 3.5) is read as a ratio of equilibrium arrival fluxes; quasi-reversibility is its rate characterization (3.8), (3.10).
  • Normalizing constants. bjb_jbj​ is defined through a tsum, which Lean sets to 000 for a divergent series; every theorem assumes the series converges, the book's "none of b1,…,bJb_1,\dots,b_Jb1​,…,bJ​ is zero".
  • No trivial instance. The goal holds for arbitrary III, JJJ, ν\nuν, routes, ϕj\phi_jϕj​, γj\gamma_jγj​, δj\delta_jδj​ subject only to the book's constraints; a proof for a single queue, or for fixed γ=δ\gamma=\deltaγ=δ disciplines, does not prove it. In Lemma 3.13 the function Φ\PhiΦ is required to be positive, since Φ≡0\Phi\equiv0Φ≡0 satisfies (3.26) for every queue.

Infrastructure that a complete development needs: list insertion/deletion lemmas for position bookkeeping, sums of products over ∏jList(⋅)\prod_j \mathrm{List}(\cdot)∏j​List(⋅), and a bijection-of-events argument for summed rates. The quasi-reversibility predicate and the reversed-rate apparatus are reusable for the closed networks of §3.4 and the symmetric queues of §3.3. Contributions are welcome on any milestone, in any order.

Selected references

  • F. P. Kelly, Reversibility and Stochastic Networks, John Wiley & Sons, 1979. https://www.statslab.cam.ac.uk/~frank/BOOKS/kelly_book.html
  • F. P. Kelly, Networks of queues with customers of different types, Journal of Applied Probability 12 (1975), 542–554. https://doi.org/10.2307/3212785
  • F. Baskett, K. M. Chandy, R. R. Muntz, F. G. Palacios, Open, closed, and mixed networks of queues with different classes of customers, Journal of the ACM 22 (1975), 248–260. https://doi.org/10.1145/321879.321887
  • J. R. Jackson, Jobshop-like queueing systems, Management Science 10 (1963), 131–142. https://doi.org/10.1287/mnsc.10.1.131
  • F. P. Kelly, E. Yudovina, Stochastic Networks, Cambridge University Press, 2014. https://doi.org/10.1017/CBO9781139565363
10 thms2 active usersReviewed
CombinatoricsMarkov ChainOperations Research+2·Captain: mikedeng1

Reversibility and Stochastic Networks VI: The Ewens Sampling Distribution Is Consistent Under Sampling Without ReplacementTextbook

Motivation

The neutral theory of molecular evolution holds that much of the genetic variation observed at the molecular level is caused by selectively neutral mutations rather than by selection. To test it against data one needs the distribution of allele frequencies that a neutral model predicts, and in practice that distribution has to be compared with a sample from the population, never with the whole population. Ewens (Ewens 1972) derived the equilibrium distribution of allele counts under the infinite alleles model, now called the Ewens sampling formula; it underlies classical tests of neutrality and appears throughout combinatorics and probability as the law of the cycle type of an Ewens-distributed random permutation and of the Chinese restaurant process.

Chapter 7 of F. P. Kelly, Reversibility and Stochastic Networks (Wiley, 1979) obtains the infinite alleles model as a limit of the reversible migration processes of Chapters 2 and 6, and uses reversibility to answer questions about allele ages and fixation. The mission formalizes the finite, combinatorial results of that chapter.

Timeline. Kimura and Crow (1964) introduced the infinite alleles model. Ewens (1972) found its equilibrium sampling distribution (7.6). Kingman (1978, J. London Math. Soc.) characterized the consistency of random partitions under sampling, the property Theorem 7.1 asserts for the Ewens family. Kelly (1979, Chapter 7) derived (7.6) as a limit of reversible migration processes, and the consistency and the allele-age results from the reversibility of a labelled population process.

Setting

A population consists of M≥2M\ge2M≥2 individuals, each carrying an allelic type. Its description is M=(M1,…,MM)\mathbf M=(M_1,\dots,M_M)M=(M1​,…,MM​), where MiM_iMi​ is the number of allelic types carried by exactly iii individuals, so that

∑i=1MiMi=M.(7.3)\sum_{i=1}^{M} iM_i=M. \qquad (7.3)i=1∑M​iMi​=M.(7.3)

For a real parameter ν>0\nu>0ν>0, the Ewens distribution on descriptions is

πM(M)=(ν+M−1M)−1∏i=1M(νi)Mi1Mi!,(7.6)\pi_M(\mathbf M)=\binom{\nu+M-1}{M}^{-1}\prod_{i=1}^{M}\Big(\frac{\nu}{i}\Big)^{M_i}\frac{1}{M_i!}, \qquad (7.6)πM​(M)=(Mν+M−1​)−1i=1∏M​(iν​)Mi​Mi​!1​,(7.6)

where (xk)=x(x−1)⋯(x−k+1)/k!\binom{x}{k}=x(x-1)\cdots(x-k+1)/k!(kx​)=x(x−1)⋯(x−k+1)/k! is the binomial coefficient for real xxx. In the infinite alleles model, individuals die at rate μ\muμ, each death is followed by the birth of an offspring of a uniformly chosen survivor, and the offspring is a mutant of an entirely new type with probability uuu; then (7.6) is the equilibrium distribution with ν=(M−1)u/(1−u)\nu=(M-1)u/(1-u)ν=(M−1)u/(1−u) (7.5).

A random sample of size 1≤m≤M1\le m\le M1≤m≤M without replacement is a uniformly random mmm-element subset of the MMM labelled individuals, each of the (Mm)\binom Mm(mM​) subsets being equally likely; the sample has a description in the same sense.

The number jjj of individuals carrying one given allele performs a random walk on {0,…,M}\{0,\dots,M\}{0,…,M} with intensities

q(j,j−1)=μjM(M−jM−1+j−1M−1u),q(j,j+1)=μM−jMjM−1(1−u).(7.8)q(j,j-1)=\mu\frac jM\Big(\frac{M-j}{M-1}+\frac{j-1}{M-1}u\Big),\qquad q(j,j+1)=\mu\frac{M-j}{M}\frac{j}{M-1}(1-u). \qquad (7.8)q(j,j−1)=μMj​(M−1M−j​+M−1j−1​u),q(j,j+1)=μMM−j​M−1j​(1−u).(7.8)

An allele is quasi-fixed when it is the only allele present (j=Mj=Mj=M).

Formalization targets

Goal: consistency under sampling (Theorem 7.1)

If M≥2M\ge2M≥2 and the population description is distributed as πM\pi_MπM​, then a random sample of size 1≤m≤M1\le m\le M1≤m≤M drawn without replacement has description m\mathbf mm with probability πm(m)\pi_m(\mathbf m)πm​(m), the same ν\nuν being used for both sizes:

∑MπM(M) P(sample has description m∣population has description M)=πm(m).\sum_{\mathbf M}\pi_M(\mathbf M)\,P\big(\text{sample has description }\mathbf m\mid\text{population has description }\mathbf M\big)=\pi_m(\mathbf m).M∑​πM​(M)P(sample has description m∣population has description M)=πm​(m).

Milestones

  1. (7.6) is a distribution: πM(M)>0\pi_M(\mathbf M)>0πM​(M)>0 and ∑MπM(M)=1\sum_{\mathbf M}\pi_M(\mathbf M)=1∑M​πM​(M)=1 (Exercise 7.1.3).
  2. Theorem 7.1 for m=M−1m=M-1m=M−1, the case the book's proof establishes first.
  3. Corollary 7.5, the identity of its proof: the probability that a uniformly chosen individual's allele is carried by exactly iii individuals is
∑MiMiMπM(M)=νM(ν+M−1i)−1(Mi).(7.9)\sum_{\mathbf M}\frac{iM_i}{M}\pi_M(\mathbf M)=\frac{\nu}{M}\binom{\nu+M-1}{i}^{-1}\binom Mi. \qquad (7.9)M∑​MiMi​​πM​(M)=Mν​(iν+M−1​)−1(iM​).(7.9)
  1. Theorem 7.9: the probability QQQ that the walk (7.8) started at 111 reaches MMM before 000 satisfies
Q−1=∑i=0M−1(M−1i)−1(ν+M−1i).Q^{-1}=\sum_{i=0}^{M-1}\binom{M-1}{i}^{-1}\binom{\nu+M-1}{i}.Q−1=i=0∑M−1​(iM−1​)−1(iν+M−1​).

Significance

The results. Consistency under sampling is what makes the Ewens formula usable as a statistical model: the predicted distribution for an observed sample does not depend on the unknown population size, only on ν\nuν. Kelly deduces from it the sufficiency of the number of alleles in a sample for ν\nuν and the heterozygosity ν/(ν+1)\nu/(\nu+1)ν/(ν+1) (Exercises 7.1.5, 7.1.8). The formula (7.9) gives the equilibrium frequency of the oldest allele, and Theorem 7.9 gives the quasi-fixation probability from which the mean time between quasi-fixations follows (Corollary 7.10).

Formalizing them. All four results are classical and proved; none has a machine-checked proof on the platform or in Mathlib as of this writing. The mission produces a reusable formal Ewens distribution over integer partitions, a definition of sampling without replacement by counting labelled subsets, and an absorption probability for an explicit birth–death walk. Proofs independent of Kelly's process argument are welcome.

Difficulty

The book's proof of Theorem 7.1 is a process argument: in a population whose size fluctuates between M−1M-1M−1 and MMM, a drop in size acts as a random deletion, and the truncated equilibrium (7.7) restricted to each size gives πM−1\pi_{M-1}πM−1​ and πM\pi_MπM​. Turning that into a statement about finite sets requires the equilibrium of a truncated reversible process, which is not available here, so a formal proof must either build that process or find a direct combinatorial route. A direct route has to relate, for each description of the sample, the number of mmm-subsets of a labelled population with a given description to products of binomial coefficients, and sum the result against (7.6); the bookkeeping over partitions is where the work lies. Theorem 7.9 needs a solution of the first-step equations of a non-symmetric walk and the identification of that solution with a hitting probability defined as a limit.

Formalization scope

  • Descriptions of nnn individuals are integer partitions Nat.Partition n, with MiM_iMi​ the multiplicity of the part iii; the product in (7.6) runs over i=1,…,ni=1,\dots,ni=1,…,n. The real binomial coefficient is the published definition AppliedComb.GenFun.binomReal.
  • The population is Fin M with allelic types Fin M → ℕ; the description of a labelled set is computed from the labelling. The sampling probability is (Mm)−1\binom Mm^{-1}(mM​)−1 times the number of mmm-subsets whose restricted labelling has the given description. It is not defined by a formula on descriptions, and a definition that removed individuals one at a time in proportion to class sizes (the book's proof route) is ruled out as a definition because it presupposes the reduction the proof must supply.
  • The goal and Corollary 7.5 quantify over an arbitrary choice of labelling for each population description. They assume M≥2M\ge2M≥2, as required by the chapter's rule that a parent is chosen among the other M−1M-1M−1 individuals; the goal also assumes 1≤m≤M1\le m\le M1≤m≤M. Because πM>0\pi_M>0πM​>0, this forces the conditional sampling law to depend on the population only through its description. Types are natural numbers, so every description is realized and the hypothesis is never vacuous.
  • The quasi-fixation probability is defined through the jump chain of (7.8): the limit of the probabilities of reaching MMM within nnn jumps without reaching 000. The theorem assumes M≥2M\ge2M≥2, μ>0\mu>0μ>0, 0<u<10<u<10<u<1 and ν=(M−1)u/(1−u)\nu=(M-1)u/(1-u)ν=(M−1)u/(1−u).
  • Corollary 7.5 is formalized as the identity of its proof. The identification of the oldest allele's frequency with that of a randomly chosen individual uses allele ages and the reversibility of the labelled process (Theorem 7.2) and is not formalized. Theorem 7.2 itself, whose state space orders the allele labels within each class, and the allele-age results (Corollaries 7.3, 7.4, 7.7, 7.8, Theorem 7.6, Corollary 7.10, Theorem 7.11) are not part of the mission.

Contributions of general partition and sampling lemmas (counting subsets with a given description, the generating function identity (1−x)−ν=∏jeνxj/j(1-x)^{-\nu}=\prod_j e^{\nu x^j/j}(1−x)−ν=∏j​eνxj/j) are reusable beyond this mission.

Selected references

  • F. P. Kelly, Reversibility and Stochastic Networks, Wiley, 1979, Chapter 7. https://www.statslab.cam.ac.uk/~frank/BOOKS/kelly_book.html
  • W. J. Ewens, The sampling theory of selectively neutral alleles, Theoretical Population Biology 3 (1972), 87–112. https://doi.org/10.1016/0040-5809(72)90035-4
  • J. F. C. Kingman, The representation of partition structures, Journal of the London Mathematical Society (2) 18 (1978), 374–380. https://doi.org/10.1112/jlms/s2-18.2.374
  • M. Kimura and J. F. Crow, The number of alleles that can be maintained in a finite population, Genetics 49 (1964), 725–738. https://doi.org/10.1093/genetics/49.4.725
9 thms2 active usersReviewed
Control TheoryDynamic ProgrammingOperations Research+1·Captain: mikedeng1

Stochastic Optimal Control: The Discrete-Time Case III: Monotone Increase Models — Accumulation Points of DP-Optimal Controls Give an Optimal Stationary PolicyTextbook

Motivation

Infinite-horizon dynamic programming with positive (nonnegative) costs and no discounting, or with discount factors that do not make the Bellman operator a contraction, is the setting of Blackwell's and Strauch's positive and negative programming models and of many deterministic control and reachability problems. In this regime the classical contraction argument is unavailable: the optimal cost may be infinite at some states, value iteration may fail to converge to it, and an optimal policy may fail to exist. Chapter 5 of Bertsekas and Shreve, Stochastic Optimal Control: The Discrete-Time Case (1978; Athena Scientific reprint 1996), treats these problems in an abstract framework, a single monotone mapping HHH, so that one set of theorems covers stochastic control with additive or multiplicative costs and minimax control. The chapter is the book version of Bertsekas, "Monotone mappings with application in dynamic programming", SIAM J. Control Optim. 15 (1977).

The chapter's last structural result answers a practical question: when value iteration is run from the terminal cost, do the minimizing controls it computes at each stage lead to an optimal stationary policy?

Setting

The abstract monotone model consists of a state space SSS, a control space CCC, nonempty constraint sets U(x)⊆CU(x)\subseteq CU(x)⊆C, a terminal function J0:S→(−∞,∞]J_0 : S\to(-\infty,\infty]J0​:S→(−∞,∞], and a mapping H(x,u,J)∈[−∞,∞]H(x,u,J)\in[-\infty,\infty]H(x,u,J)∈[−∞,∞] defined for x∈Sx\in Sx∈S, u∈Cu\in Cu∈C and J:S→[−∞,∞]J : S\to[-\infty,\infty]J:S→[−∞,∞], which is monotone: J≤J′J\le J'J≤J′ implies H(x,u,J)≤H(x,u,J′)H(x,u,J)\le H(x,u,J')H(x,u,J)≤H(x,u,J′).

A selector is a function μ:S→C\mu : S\to Cμ:S→C with μ(x)∈U(x)\mu(x)\in U(x)μ(x)∈U(x); a policy is a sequence π=(μ0,μ1,… )\pi=(\mu_0,\mu_1,\dots)π=(μ0​,μ1​,…) of selectors, and it is stationary if all μk\mu_kμk​ are equal. The operators are

Tμ(J)(x)=H(x,μ(x),J),T(J)(x)=inf⁡u∈U(x)H(x,u,J),T_\mu(J)(x)=H(x,\mu(x),J),\qquad T(J)(x)=\inf_{u\in U(x)}H(x,u,J),Tμ​(J)(x)=H(x,μ(x),J),T(J)(x)=u∈U(x)inf​H(x,u,J),

the cost of a policy is Jπ(x)=lim⁡N→∞(Tμ0Tμ1⋯TμN−1)(J0)(x)J_\pi(x)=\lim_{N\to\infty}(T_{\mu_0}T_{\mu_1}\cdots T_{\mu_{N-1}})(J_0)(x)Jπ​(x)=limN→∞​(Tμ0​​Tμ1​​⋯TμN−1​​)(J0​)(x), JμJ_\muJμ​ is the cost of the stationary policy (μ,μ,… )(\mu,\mu,\dots)(μ,μ,…), and the optimal cost is J∗(x)=inf⁡πJπ(x)J^*(x)=\inf_\pi J_\pi(x)J∗(x)=infπ​Jπ​(x). A policy is optimal if Jπ=J∗J_\pi=J^*Jπ​=J∗.

Assumption I (uniform increase) is J0(x)≤H(x,u,J0)J_0(x)\le H(x,u,J_0)J0​(x)≤H(x,u,J0​) for all xxx, u∈U(x)u\in U(x)u∈U(x); under it the sequence defining JπJ_\piJπ​ is nondecreasing, so the limit exists in [−∞,∞][-\infty,\infty][−∞,∞]. Assumption I.1 says H(x,u,⋅)H(x,u,\cdot)H(x,u,⋅) commutes with limits of nondecreasing sequences above J0J_0J0​, and I.2 says that for some α>0\alpha>0α>0, H(x,u,J)≤H(x,u,J+r)≤H(x,u,J)+αrH(x,u,J)\le H(x,u,J+r)\le H(x,u,J)+\alpha rH(x,u,J)≤H(x,u,J+r)≤H(x,u,J)+αr for all r>0r>0r>0 and J≥J0J\ge J_0J≥J0​. Assumptions D, D.1, D.2 are the mirror images (uniform decrease, continuity along nonincreasing sequences below J0J_0J0​, and a lower shift bound).

The DP algorithm (value iteration) generates T(J0),T2(J0),…T(J_0), T^2(J_0),\dotsT(J0​),T2(J0​),…; for k≥0k\ge0k≥0 and λ∈R\lambda\in\mathbb Rλ∈R the level sets

Uk(x,λ)={u∈U(x)∣H[x,u,Tk(J0)]≤λ}U_k(x,\lambda)=\{u\in U(x)\mid H[x,u,T^k(J_0)]\le\lambda\}Uk​(x,λ)={u∈U(x)∣H[x,u,Tk(J0​)]≤λ}

collect the controls that keep the stage-kkk cost below λ\lambdaλ.

Formalization targets

Goal: Proposition 5.11

Let I, I.1 and I.2 hold, let CCC be a Hausdorff space, and let Uk(x,λ)U_k(x,\lambda)Uk​(x,λ) be compact for all x∈Sx\in Sx∈S, λ∈R\lambda\in\mathbb Rλ∈R and k≥kˉk\ge\bar kk≥kˉ. Then (a) some policy π∗=(μ0∗,μ1∗,… )\pi^*=(\mu_0^*,\mu_1^*,\dots)π∗=(μ0∗​,μ1∗​,…) satisfies

(Tμk∗Tk)(J0)=Tk+1(J0)∀k≥kˉ;(T_{\mu_k^*}T^k)(J_0)=T^{k+1}(J_0)\qquad\forall k\ge\bar k;(Tμk∗​​Tk)(J0​)=Tk+1(J0​)∀k≥kˉ;

(b) for every such policy, {μk∗(x)}\{\mu_k^*(x)\}{μk∗​(x)} has an accumulation point whenever J∗(x)<∞J^*(x)<\inftyJ∗(x)<∞; and (c) any μ∗\mu^*μ∗ that picks such an accumulation point at those states, and any admissible control where J∗(x)=∞J^*(x)=\inftyJ∗(x)=∞, defines an optimal stationary policy:

Jμ∗=J∗.J_{\mu^*}=J^*.Jμ∗​=J∗.

Milestones

In the order of the mission's milestone list: Lemma 3.1 (compact sublevel sets give a minimum), Proposition 5.2 (Bellman's equation J∗=T(J∗)J^*=T(J^*)J∗=T(J∗) under I, I.1, I.2), Proposition 5.4 (a stationary policy is optimal iff Tμ∗(J∗)=T(J∗)T_{\mu^*}(J^*)=T(J^*)Tμ∗​(J∗)=T(J∗)), Proposition 5.10 (under the compactness hypothesis, J∞=T(J∞)=T(J∗)=J∗J_\infty=T(J_\infty)=T(J^*)=J^*J∞​=T(J∞​)=T(J∗)=J∗ and a stationary optimal policy exists), Proposition 5.6 (ε\varepsilonε-optimal policies from approximate attainment, with explicit constants ∑kαkεk=ε\sum_k\alpha^k\varepsilon_k=\varepsilon∑k​αkεk​=ε and ε(1−α)\varepsilon(1-\alpha)ε(1−α)), Corollaries 5.3.1 and 5.7.1 (Bellman's equation and ε\varepsilonε-optimal policies under D, D.2 with finite SSS and J∗>−∞J^*>-\inftyJ∗>−∞), and Propositions 5.14 and 5.15 (the multiplicative-cost and minimax models satisfy I, I.1, I.2 or D, D.1/D.2 with explicitly named scalars).

Significance

Proposition 5.10 alone gives existence of an optimal stationary policy; Proposition 5.11 identifies one. It says that the minimizers computed by value iteration, which any implementation produces anyway, converge (along subsequences, state by state) to optimal controls. This is the abstract form of the classical results for deterministic and stochastic positive-cost problems with compact control sets, and through Propositions 5.14 and 5.15 it applies to multiplicative (risk-sensitive) costs and to minimax control without new arguments.

Lemma 3.1, Propositions 5.1–5.5, 5.7–5.10, Lemmas 5.1–5.2 and Corollaries 5.2.1, 5.3.2 are already formalized and proved on the platform, in the MonotoneDP.Increase and MonotoneDP.Decrease developments built from the 1977 paper; this mission reuses their model and assumption definitions and links Lemma 3.1, Proposition 5.2 and Proposition 5.10 as milestones. Propositions 5.4 (in the book's stronger form, with policies optimal state by state), 5.6, 5.11, 5.14, 5.15 and Corollaries 5.3.1, 5.7.1 are new here. All are proved in the book; none has a machine-checked proof yet.

Difficulty

The obvious route to (c) is to pass to the limit in H[x,μk∗(x),Tk(J0)]=Tk+1(J0)(x)H[x,\mu_k^*(x),T^k(J_0)]=T^{k+1}(J_0)(x)H[x,μk∗​(x),Tk(J0​)]=Tk+1(J0​)(x). That fails twice. First, {μk∗(x)}\{\mu_k^*(x)\}{μk∗​(x)} need not converge, and different subsequences may have different limits; the statement is about an arbitrary accumulation point, and in a general Hausdorff space accumulation points are not limits of subsequences. Second, HHH is not assumed continuous in uuu at all: the only regularity in uuu is compactness of the level sets Uk(x,λ)U_k(x,\lambda)Uk​(x,λ), and the only regularity in JJJ is I.1 along monotone sequences. States with J∗(x)=∞J^*(x)=\inftyJ∗(x)=∞ need separate treatment, because there the level sets give no control on μk∗(x)\mu_k^*(x)μk∗​(x).

For Corollaries 5.3.1 and 5.7.1 the difficulty is that D.1 is not assumed; the replacement uses finiteness of SSS and J∗>−∞J^*>-\inftyJ∗>−∞ essentially, and both hypotheses are needed.

Formalization scope

Functions JJJ are S → EReal. The model, Assumptions I, I.1, I.2, D, D.1, D.2 and the epigraph sets used by Proposition 5.10 are the published definitions MonotoneDP_Increase_Model, MonotoneDP_Increase_Assumptions, MonotoneDP_Increase_Epigraph, MonotoneDP_Decrease_Model and MonotoneDP_Decrease_Assumptions. In them JπJ_\piJπ​ is the limit (limUnder) of the policy compositions, which exists under I or D; TkT^kTk is the iterate T^[k]; the composition (Tμ0⋯TμN−1)(J)(T_{\mu_0}\cdots T_{\mu_{N-1}})(J)(Tμ0​​⋯TμN−1​​)(J) applies TμN−1T_{\mu_{N-1}}TμN−1​​ first; the model requires SSS nonempty and J0>−∞J_0>-\inftyJ0​>−∞; "I.2 holds" is the existence of a scalar α\alphaα, and results that name the scalar take it as a parameter. An accumulation point of a sequence in CCC is a cluster point (MapClusterPt), not a limit; in Proposition 5.11(c) the conclusion includes that μ∗\mu^*μ∗ is admissible. J∗+εJ^*+\varepsilonJ∗+ε is computed pointwise in [−∞,∞][-\infty,\infty][−∞,∞] and equals +∞+\infty+∞ where J∗J^*J∗ does.

The book computes in [−∞,∞][-\infty,\infty][−∞,∞] with ∞−∞=∞\infty-\infty=\infty∞−∞=∞. Mathlib's EReal gives ⊤+⊥=⊥\top+\bot=\bot⊤+⊥=⊥, so the two mappings of Section 2.3 use the series' shared definitions, which follow the book's convention: the expected value on a countable WWW (expect, positive and negative parts summed in [0,∞][0,\infty][0,∞], +∞+\infty+∞ when the positive part diverges) and the sum inside the minimax supremum (badd, +∞+\infty+∞ when either summand is +∞+\infty+∞), both from BertsekasShreve.FiniteHorizon.SpecificModels. Propositions 5.14 and 5.15 quantify over every model whose HHH and J0J_0J0​ are these mappings; such models exist because the mappings are monotone.

A formalization in which JπJ_\piJπ​ or J∗J^*J∗ is introduced as an arbitrary fixed point of TTT, or in which (c) assumes that μ∗\mu^*μ∗ is a limit of μk∗\mu_k^*μk∗​, would make the goal trivial or different; both are ruled out by the definitions above.

Contributions welcome: proofs of the milestones, and reusable EReal infrastructure (limits of monotone sequences, series of nonnegative extended reals) that the Part I chapters of the book share.

Selected references

  • D. P. Bertsekas and S. E. Shreve, Stochastic Optimal Control: The Discrete-Time Case, Academic Press, 1978; reprinted Athena Scientific, 1996. Chapter 5, pp. 70–90. https://web.mit.edu/dimitrib/www/soc.html
  • D. P. Bertsekas, "Monotone mappings with application in dynamic programming", SIAM J. Control Optim. 15 (1977) 438–464. https://doi.org/10.1137/0315031
  • D. Blackwell, "Positive dynamic programming", Proc. Fifth Berkeley Symp. Math. Statist. Probab. 1 (1967) 415–418. https://projecteuclid.org/euclid.bsmsp/1200512999
  • R. E. Strauch, "Negative dynamic programming", Ann. Math. Statist. 37 (1966) 871–890. https://doi.org/10.1214/aoms/1177699369
16 thms5 active usersReviewed
Machine LearningOperations ResearchOptimization+1·Captain: mikedeng1

Twice Regularized MDPs and the Equivalence Between Robustness and Regularization 2: The Greedy Policy of the R2 Optimal Value Is the Unique Optimal R2 PolicyResearch Paper

Motivation

A robust Markov decision process (robust MDP) evaluates a policy against the worst transition kernel and reward in an uncertainty set around a nominal model (P0,r0)(P_0, r_0)(P0​,r0​). It is the standard model for planning when the dynamics are estimated from data (Iyengar 2005; Nilim and El Ghaoui 2005; Wiesemann, Kuhn and Rustem 2013). Its Bellman update contains an inner optimization over the uncertainty set at every state, which makes robust planning costly when the sets are not (s,a)(s,a)(s,a)-rectangular.

Derman, Geist and Mannor (arXiv:2110.06267, NeurIPS 2021) show that, for sss-rectangular ball uncertainty sets, this inner optimization can be replaced by an explicit penalty that depends both on the policy and on the value function. The resulting twice regularized (R²) MDPs have Bellman operators with no inner optimization over models. The first mission of this series formalizes the robust–regularized equivalence (Theorem 4.1 of the paper). This mission formalizes Section 5: the R² Bellman operators are monotone and contracting under a bound on the transition radius, and the greedy policy of the R² optimal value is optimal.

Setting

Let S\mathcal SS and A\mathcal AA be finite nonempty sets of states and actions, γ∈(0,1)\gamma\in(0,1)γ∈(0,1) a discount factor, P0(s′∣s,a)P_0(s'\mid s,a)P0​(s′∣s,a) a transition kernel and r0(s,a)r_0(s,a)r0​(s,a) a reward. A policy π∈ΔAS\pi\in\Delta_{\mathcal A}^{\mathcal S}π∈ΔAS​ assigns to each state a probability distribution πs\pi_sπs​ on A\mathcal AA. For v∈RSv\in\mathbb R^{\mathcal S}v∈RS write qs(a)=r0(s,a)+γ∑s′P0(s′∣s,a)v(s′)q_s(a)=r_0(s,a)+\gamma\sum_{s'}P_0(s'\mid s,a)v(s')qs​(a)=r0​(s,a)+γ∑s′​P0​(s′∣s,a)v(s′) and

[T(P0,r0)πv](s)=∑aπs(a) qs(a).[T^\pi_{(P_0,r_0)}v](s)=\sum_a\pi_s(a)\,q_s(a).[T(P0​,r0​)π​v](s)=a∑​πs​(a)qs​(a).

All norms ∥⋅∥\|\cdot\|∥⋅∥ below are ℓ2\ell_2ℓ2​-norms, ∥a∥=(∑za(z)2)1/2\|a\|=\big(\sum_z a(z)^2\big)^{1/2}∥a∥=(∑z​a(z)2)1/2; ∥⋅∥∞\|\cdot\|_\infty∥⋅∥∞​ is the sup norm.

Fix nonnegative radii αsr,αsP\alpha^r_s,\alpha^P_sαsr​,αsP​ for each state. The R² regularizer is Ωv,R2(πs)=∥πs∥ (αsr+αsPγ∥v∥)\Omega_{v,\mathrm R^2}(\pi_s)=\|\pi_s\|\,(\alpha^r_s+\alpha^P_s\gamma\|v\|)Ωv,R2​(πs​)=∥πs​∥(αsr​+αsP​γ∥v∥), and the R² Bellman operators are

[Tπ,R2v](s)=[T(P0,r0)πv](s)−Ωv,R2(πs),[T∗,R2v](s)=max⁡π∈ΔAS[Tπ,R2v](s).[T^{\pi,\mathrm R^2}v](s)=[T^\pi_{(P_0,r_0)}v](s)-\Omega_{v,\mathrm R^2}(\pi_s),\qquad [T^{*,\mathrm R^2}v](s)=\max_{\pi\in\Delta^{\mathcal S}_{\mathcal A}}[T^{\pi,\mathrm R^2}v](s).[Tπ,R2v](s)=[T(P0​,r0​)π​v](s)−Ωv,R2​(πs​),[T∗,R2v](s)=π∈ΔAS​max​[Tπ,R2v](s).

A policy π\piπ is greedy for vvv when Tπ,R2v=T∗,R2vT^{\pi,\mathrm R^2}v=T^{*,\mathrm R^2}vTπ,R2v=T∗,R2v.

Assumption 5.1 (bounded radius). For each sss there is ϵs>0\epsilon_s>0ϵs​>0 with

αsP≤min⁡(1−γ−ϵsγ∣S∣ ; min⁡u∈R+A,∥u∥=1, w∈R+S,∥w∥=1 ∑a,s′u(a)P0(s′∣s,a)w(s′)),\alpha^P_s\le\min\Big(\frac{1-\gamma-\epsilon_s}{\gamma\sqrt{|\mathcal S|}}\ ;\ \min_{u\in\mathbb R^{\mathcal A}_+,\|u\|=1,\ w\in\mathbb R^{\mathcal S}_+,\|w\|=1}\ \sum_{a,s'}u(a)P_0(s'\mid s,a)w(s')\Big),αsP​≤min(γ∣S∣​1−γ−ϵs​​ ; u∈R+A​,∥u∥=1, w∈R+S​,∥w∥=1min​ a,s′∑​u(a)P0​(s′∣s,a)w(s′)),

and ϵ∗=min⁡sϵs\epsilon_*=\min_s\epsilon_sϵ∗​=mins​ϵs​. The R² value function vπ,R2v^{\pi,\mathrm R^2}vπ,R2 of a policy and the R² optimal value v∗,R2v^{*,\mathrm R^2}v∗,R2 are the fixed points of Tπ,R2T^{\pi,\mathrm R^2}Tπ,R2 and T∗,R2T^{*,\mathrm R^2}T∗,R2.

Formalization targets

Goal: Theorem 5.1 (p. 8)

Under Assumption 5.1, T∗,R2T^{*,\mathrm R^2}T∗,R2 and every Tπ,R2T^{\pi,\mathrm R^2}Tπ,R2 have unique fixed points; a greedy policy π∗,R2\pi^{*,\mathrm R^2}π∗,R2 for v∗,R2v^{*,\mathrm R^2}v∗,R2 exists, and every such policy satisfies

vπ∗,R2,R2=v∗,R2 ≥ vπ,R2for all π∈ΔAS;v^{\pi^{*,\mathrm R^2},\mathrm R^2}=v^{*,\mathrm R^2}\ \ge\ v^{\pi,\mathrm R^2}\qquad\text{for all }\pi\in\Delta^{\mathcal S}_{\mathcal A};vπ∗,R2,R2=v∗,R2 ≥ vπ,R2for all π∈ΔAS​;

every optimal policy is greedy; and when αsr>0\alpha^r_s>0αsr​>0 for all sss the greedy policy is unique, hence the unique optimal R² policy.

Milestones

  1. Proposition 2.1 (p. 3): for Ω\OmegaΩ strongly convex on the simplex, Ω∗(y)=max⁡a∈Δ⟨a,y⟩−Ω(a)\Omega^*(y)=\max_{a\in\Delta}\langle a,y\rangle-\Omega(a)Ω∗(y)=maxa∈Δ​⟨a,y⟩−Ω(a) is differentiable with Lipschitz gradient equal to the unique maximizer, satisfies Ω∗(y+c1)=Ω∗(y)+c\Omega^*(y+c\mathbb 1)=\Omega^*(y)+cΩ∗(y+c1)=Ω∗(y)+c, and is non-decreasing.
  2. Proposition 5.1 (i) (p. 8): v1≤v2v_1\le v_2v1​≤v2​ implies Tπ,R2v1≤Tπ,R2v2T^{\pi,\mathrm R^2}v_1\le T^{\pi,\mathrm R^2}v_2Tπ,R2v1​≤Tπ,R2v2​ and T∗,R2v1≤T∗,R2v2T^{*,\mathrm R^2}v_1\le T^{*,\mathrm R^2}v_2T∗,R2v1​≤T∗,R2v2​.
  3. Proposition 5.1 (iii) (p. 8):
∥Tπ,R2v1−Tπ,R2v2∥∞≤(1−ϵ∗)∥v1−v2∥∞,∥T∗,R2v1−T∗,R2v2∥∞≤(1−ϵ∗)∥v1−v2∥∞.\|T^{\pi,\mathrm R^2}v_1-T^{\pi,\mathrm R^2}v_2\|_\infty\le(1-\epsilon_*)\|v_1-v_2\|_\infty,\qquad \|T^{*,\mathrm R^2}v_1-T^{*,\mathrm R^2}v_2\|_\infty\le(1-\epsilon_*)\|v_1-v_2\|_\infty.∥Tπ,R2v1​−Tπ,R2v2​∥∞​≤(1−ϵ∗​)∥v1​−v2​∥∞​,∥T∗,R2v1​−T∗,R2v2​∥∞​≤(1−ϵ∗​)∥v1​−v2​∥∞​.

Significance

Theorem 5.1 is the R² counterpart of the fundamental theorem of discounted dynamic programming: optimal R² values are achieved by stationary policies obtained by a single greedy step. Together with the contraction of Proposition 5.1 (iii) it justifies the R² modified policy iteration algorithm of the paper, whose greedy step is a projection onto the simplex rather than a robust max–min problem. Combined with the first mission of the series, which identifies the robust value of an sss-rectangular ball-constrained MDP with the optimum of an R²-regularized program, it gives a route to robust planning at the cost of regularized planning.

The results are proved in the paper (App. C), partly by reference to Geist, Scherrer and Pietquin (2019) for the optimality operator. No machine-checked proof of any of them exists; this mission produces the first. Prop. 2.1 is a general fact of convex analysis (Danskin-type smoothness of a conjugate on the simplex) that is reusable for any regularized MDP or entropy-regularized game.

Difficulty

The R² evaluation operator is not affine: the value regularizer −αsPγ∥πs∥ ∥v∥-\alpha^P_s\gamma\|\pi_s\|\,\|v\|−αsP​γ∥πs​∥∥v∥ is concave in vvv and decreases as ∥v∥\|v\|∥v∥ grows. Monotonicity therefore does not follow from the positivity of P0P_0P0​ as in the standard case; it requires the second bound of Assumption 5.1, which compares the ℓ2\ell_2ℓ2​ variation of ∥v∥\|v\|∥v∥ with the minimal nonnegative bilinear form of P0(⋅∣s,⋅)P_0(\cdot\mid s,\cdot)P0​(⋅∣s,⋅). Likewise the contraction modulus is not γ\gammaγ but 1−ϵ∗1-\epsilon_*1−ϵ∗​, because the regularizer is ∣S∣\sqrt{|\mathcal S|}∣S∣​-Lipschitz between the ℓ2\ell_2ℓ2​ and sup norms. The optimality step of the classical proof uses linearity of TπT^\piTπ when comparing values of policies; here only monotonicity and contraction are available. Uniqueness of the greedy policy rests on strict concavity on the simplex, which holds only when the regularization weight is positive.

Formalization scope

States and actions are finite nonempty types; transitions are arrays P₀ : S → A → S → ℝ with the published predicate IsTransitionKernel; value functions are S → ℝ with the pointwise order. The ℓ2\ell_2ℓ2​-norm is an explicit l2norm (Mathlib's norm on S → ℝ is the sup norm, used only for ∥⋅∥∞\|\cdot\|_\infty∥⋅∥∞​). T∗,R2v(s)T^{*,\mathrm R^2}v(s)T∗,R2v(s) is the real supremum over the simplex ΔA\Delta_{\mathcal A}ΔA​ (attained), and the inner minimum of Assumption 5.1 is the real infimum over nonnegative ℓ2\ell_2ℓ2​-unit vectors; the witnesses ϵs\epsilon_sϵs​ are explicit. Greedy policies are a predicate, never a function, and the R² value functions are not defined by choice: the goal asserts their existence and uniqueness and speaks about the fixed points.

Disclosed deviations from the page. Assumption 5.1 is a hypothesis of Theorem 5.1 (its proof assumes it). The uniqueness clause of Theorem 5.1 additionally assumes αsr>0\alpha^r_s>0αsr​>0 for all sss: with one state, two actions, zero reward and zero radii every policy is greedy and optimal. Proposition 2.1 assumes Ω\OmegaΩ continuous on the simplex, without which the maximum need not be attained, and strong convexity is Mathlib's StrongConvexOn for some modulus (norm-independent in finite dimension). Proposition 5.1 (ii) is false as printed and is not drafted: with one state, one action, P0=1P_0=1P0​=1, r0=0r_0=0r0​=0, γ=1/2\gamma=1/2γ=1/2, αr=0\alpha^r=0αr=0, αP=1/2\alpha^P=1/2αP=1/2, ϵ=1/4\epsilon=1/4ϵ=1/4, one has Tv=v/2−∣v∣/4Tv=v/2-|v|/4Tv=v/2−∣v∣/4, and v1=−1v_1=-1v1​=−1, c=1c=1c=1 give T(v1+c)=0>−1/4=Tv1+γcT(v_1+c)=0>-1/4=Tv_1+\gamma cT(v1​+c)=0>−1/4=Tv1​+γc. Remark 5.1, Algorithm 1 and the ℓp\ell_pℓp​ variant of App. C.1 are out of scope. The inner minimum of Assumption 5.1 is 000 whenever some P0(s′∣s,a)=0P_0(s'\mid s,a)=0P0​(s′∣s,a)=0, forcing αsP=0\alpha^P_s=0αsP​=0; this is the assumption as printed.

A formalization in which ∥⋅∥\|\cdot\|∥⋅∥ is the sup norm, the inner minimum ranges over all unit vectors (making the assumption unsatisfiable), or the value functions are postulated rather than shown to exist would be trivial or wrong; the drafted statements avoid all three. Contributions welcome: Prop. 2.1 as a general convex-analysis lemma, Banach fixed-point plumbing for S → ℝ with the sup norm, and the strict concavity of p↦⟨p,q⟩−c∥p∥p\mapsto\langle p,q\rangle-c\|p\|p↦⟨p,q⟩−c∥p∥ on the simplex.

Selected references

  • E. Derman, M. Geist, S. Mannor, Twice regularized MDPs and the equivalence between robustness and regularization, NeurIPS 2021. arXiv:2110.06267v1
  • M. Geist, B. Scherrer, O. Pietquin, A theory of regularized Markov decision processes, ICML 2019. arXiv:1901.11275
  • A. Nilim, L. El Ghaoui, Robust control of Markov decision processes with uncertain transition matrices, Operations Research 53(5), 2005. doi:10.1287/opre.1050.0216
  • G. N. Iyengar, Robust dynamic programming, Mathematics of Operations Research 30(2), 2005. doi:10.1287/moor.1040.0129
  • W. Wiesemann, D. Kuhn, B. Rustem, Robust Markov decision processes, Mathematics of Operations Research 38(1), 2013. doi:10.1287/moor.1120.0566
  • A. Mensch, M. Blondel, Differentiable dynamic programming for structured prediction and attention, ICML 2018. arXiv:1802.03676
9 thms1 active userReviewed
CombinatoricsGraph TheoryOperations Research+2·Captain: mikedeng1

Maximal Flow Through a Network II: In an ab-Planar Network Some Chain from Source to Sink Meets Every Cut Exactly OnceResearch Paper

Motivation

The maximum flow problem asks how much of a commodity can be shipped from a source to a sink through a network whose arcs have limited capacities. L. R. Ford, Jr. and D. R. Fulkerson's 1956 paper Maximal Flow Through a Network proved the minimal cut theorem: the largest flow value equals the smallest total capacity of a set of arcs that separates source from sink. That theorem is formalized in the companion mission Maximal Flow Through a Network I.

The second section of the same paper treats a special class of networks, those that remain planar after an arc from source to sink is added. For these networks the paper shows that one particular source–sink chain crosses every minimal separating set exactly once. This structural fact turns the minimal cut theorem into a simple computing procedure: repeatedly push as much flow as possible along such a chain and delete the arcs it saturates. The paper notes that G. Dantzig had conjectured, before the minimal cut theorem was proved, that this procedure yields a maximal flow on planar networks. The same "uppermost path" idea underlies later algorithms for maximum flow in planar graphs with source and sink on a common face (Itai and Shiloach, 1979).

The statement is short and purely combinatorial in its conclusion, but its hypothesis is topological. This mission isolates that theorem.

Setting

A network NNN has a finite set VVV of vertices and a finite set EEE of arcs. Each arc eee joins two distinct end vertices, written tail(e)\mathrm{tail}(e)tail(e) and head(e)\mathrm{head}(e)head(e); arcs carry no direction, and two arcs may join the same pair of vertices. Two distinct vertices are distinguished, the source aaa and the sink bbb, and each arc carries a positive capacity (capacities play no role in the target below).

A chain joining uuu and www is a set CCC of distinct arcs that can be arranged as α1(v0v1),α2(v1v2),…,αk(vk−1vk)\alpha_1(v_0v_1), \alpha_2(v_1v_2), \dots, \alpha_k(v_{k-1}v_k)α1​(v0​v1​),α2​(v1​v2​),…,αk​(vk−1​vk​) with v0=uv_0 = uv0​=u, vk=wv_k = wvk​=w, and the vertices v0,…,vkv_0, \dots, v_kv0​,…,vk​ pairwise distinct; each arc may be traversed in either direction. The empty set is the null chain from uuu to uuu.

A set DDD of arcs is a disconnecting set if every chain joining aaa and bbb contains an arc of DDD. A disconnecting set none of whose proper subsets is disconnecting is a cut.

The network is ab-planar if the graph of NNN, together with one additional arc joining aaa and bbb, can be drawn in the plane without crossings: vertices go to distinct points of R2\mathbb R^2R2; each arc, including the added arc ababab, goes to an injective continuous path between the points of its end vertices; no arc passes through a vertex other than its ends; and two distinct arcs meet only at endpoints of both. In Lean the drawing is the structure ABPlaneDrawing N, and NNN is ab-planar when Nonempty (ABPlaneDrawing N). The section's standing assumption is that no arc of NNN already joins aaa and bbb.

Formalization targets

Goal: Theorem 2 (p. 403)

If NNN is ab-planar, no arc of NNN joins aaa and bbb, and some chain joins aaa and bbb, then

∃ T a chain joining a and b  such that  ∣T∩D∣=1  for every cut D of N.\exists\, T \text{ a chain joining } a \text{ and } b \ \text{ such that }\ |T \cap D| = 1 \ \text{ for every cut } D \text{ of } N.∃T a chain joining a and b  such that  ∣T∩D∣=1  for every cut D of N.

This is FordFulkerson56.Planar.ab_planar_exists_chain_meeting_each_cut_once. "Precisely once" is exact cardinality one, neither "at least once" (true of every chain) nor "at most once".

Milestone: a chain meeting a cut in one prescribed arc (proof of Theorem 2, p. 403)

For every network NNN, every cut DDD and every arc α∈D\alpha \in Dα∈D, there is a chain CCC joining aaa and bbb with C∩D={α}C \cap D = \{\alpha\}C∩D={α}. No planarity is involved; the statement is what the minimality of a cut provides to the proof.

Further item: the Fig. 2 example (p. 403)

In the "gas, water, electricity" graph K3,3K_{3,3}K3,3​ with the arc ababab removed, every chain joining aaa and bbb meets some cut in three arcs. This network is not ab-planar, so the example shows that the planarity hypothesis of Theorem 2 cannot be dropped.

Significance

Theorem 2 and the minimal cut theorem together give the paper's procedure for planar networks: if TTT meets every cut once, then imposing a flow kkk on TTT lowers the value of every cut by exactly kkk, so the minimal cut value, and hence the maximal flow value, drops by kkk. Saturated arcs can then be deleted and the step repeated. Without the "exactly once" property the reduction could overshoot the cut structure, and the greedy step would not be justified. The theorem is also one of the earliest instances of the link between planarity and cut structure that later underlies planar duality arguments for minimum cuts.

The result has been known since 1956 and is not open. No machine-checked version is recorded on the platform, and Mathlib, at the pinned revision, has neither planar graphs nor the Jordan curve theorem. A formal proof would be the first formalized statement about source–sink planar networks in this library, and the counterexample item records, as a checkable fact, that the hypothesis is necessary.

Difficulty

The conclusion is combinatorial while the hypothesis is a drawing in R2\mathbb R^2R2. The paper's proof normalises the drawing (the added arc ababab on the outer boundary, the graph in a vertical strip with aaa on the left line and bbb on the right), selects the "top-most" chain from aaa to bbb, and argues that a chain meeting a cut below the top-most chain must cross another such chain. Each of these steps rests on plane topology: the existence of the outer region, the meaning of "top-most", and the fact that two chains with interleaved endpoints on a boundary must intersect, which is a form of the Jordan curve theorem.

The naive purely combinatorial route fails: the analogous statement for arbitrary networks is false (Fig. 2), so any argument has to use the drawing somewhere. Replacing the drawing by a combinatorial embedding (rotation systems, faces) is possible but then requires proving that the two notions agree, which is again Jordan-curve territory.

Formalization scope

Conventions committed to in the Lean statements:

  • Vertices and arcs are finite types V, E with decidable equality. Arcs are undirected, may be parallel, and have two distinct end vertices. Source and sink are distinct, capacities are positive (structure Network).
  • A chain is a Finset E that is the arc set of some arrangement as a simple path (IsChainWalk, IsChain); the null chain is allowed.
  • IsDisconnecting and IsCut quantify over all chains joining source and sink; a cut is a disconnecting set no proper subset of which is disconnecting.
  • ab-planarity is a plane drawing of the graph with the extra arc indexed by none : Option E, with injective Paths in ℝ × ℝ as arcs.

Hypotheses of the goal: hno_ab, the standing assumption of §2 (no arc joins aaa and bbb, p. 403); hconn, that some chain joins aaa and bbb. The second is not stated in the paper; its proof starts from "the chain joining a and b which is top-most", which presupposes one, and without it the statement is false (if aaa and bbb are disconnected, the empty set is a cut and no chain exists).

The drawing structure is satisfiable (a three-vertex path network has an explicit drawing), so the planarity hypothesis is not vacuous; and it covers the added arc ababab and all crossings, so K3,3K_{3,3}K3,3​ minus ababab is not ab-planar and the goal is not refuted by the paper's own example. A formalization that dropped the arc ababab from the drawing, or quantified over disconnecting sets instead of cuts, would state a false theorem and is ruled out.

A complete development needs basic plane topology for paths in R2\mathbb R^2R2 (a Jordan-curve-type separation lemma for simple closed curves, or an equivalent statement about crossing paths in a strip), together with combinatorial lemmas about chains (concatenation and shortcutting of chains at a common vertex). The topological lemmas are reusable well beyond this mission. Proofs through a combinatorial embedding are welcome, provided the equivalence with ABPlaneDrawing is proved.

Selected references

  • L. R. Ford, Jr. and D. R. Fulkerson, Maximal Flow Through a Network, Canadian Journal of Mathematics 8 (1956), 399–404. https://doi.org/10.4153/CJM-1956-045-5
  • H. Whitney, Non-separable and planar graphs, Transactions of the American Mathematical Society 34 (1932), 339–362. https://doi.org/10.1090/S0002-9947-1932-1501641-2
  • A. Itai and Y. Shiloach, Maximum flow in planar networks, SIAM Journal on Computing 8 (1979), 135–150. https://doi.org/10.1137/0208012
  • H. Whitney, Planar graphs, Fundamenta Mathematicae 21 (1933), 73–84. https://doi.org/10.4064/fm-21-1-73-84
7 thms1 active userReviewed
Linear OptimizationOperations ResearchOptimization+2·Captain: mikedeng1

Solving Linear Programs in the Current Matrix Multiplication Time: The Stochastic Central Path Falls Back to a Classical Step with Probability at Most 10/n² per IterationResearch Paper

Motivation

Linear programming, min⁡{c⊤x:Ax=b, x≥0}\min\{c^\top x : Ax=b,\ x\ge0\}min{c⊤x:Ax=b, x≥0} with A∈Rd×nA\in\mathbb R^{d\times n}A∈Rd×n, is the basic model of operations research, and the complexity of solving it is a central question of algorithm theory. Interior-point methods follow the central path: primal–dual pairs (x,s)(x,s)(x,s) with x,s>0x,s>0x,s>0 and xisi=tx_is_i=txi​si​=t for every iii, as the path parameter ttt decreases to 000. A classical short-step method needs O(nlog⁡(n/δ))O(\sqrt n\log(n/\delta))O(n​log(n/δ)) iterations, each solving a linear system with the matrix AXSA⊤A\frac XSA^\topASX​A⊤, for a total of roughly n2.5n^{2.5}n2.5 operations or more.

Cohen, Lee and Song (J. ACM 68(1), 2021; arXiv:1810.07896) showed that linear programs can be solved in time nω+o(1)log⁡(n/δ)n^{\omega+o(1)}\log(n/\delta)nω+o(1)log(n/δ) (for the current values of the matrix multiplication exponent ω\omegaω and its dual α\alphaα), matching the cost of multiplying two n×nn\times nn×n matrices. The analysis has two halves: a data structure that maintains the projection matrix lazily, and the stochastic central path method, which replaces each Newton step by a sparse random step and proves that the iterates still stay close to the central path. This mission formalizes the second half.

Timeline: Karmarkar's projective method (1984) gave the first polynomial interior-point method; Renegar (1988) gave the O(nlog⁡(1/δ))O(\sqrt n\log(1/\delta))O(n​log(1/δ)) path-following bound; Vaidya (1989) reduced the per-iteration cost with low-rank updates; Lee and Sidford (2014–2015) reduced the iteration count to O~(d)\widetilde O(\sqrt d)O(d​); Cohen, Lee and Song (STOC 2019, J. ACM 2021) reached nωn^\omeganω; van den Brand (2020) derandomized the result.

Setting

Vectors are in Rn\mathbb R^nRn and products, quotients and roots of vectors are coordinatewise. For ϵ\epsilonϵ and vectors a,ba,ba,b, a≈ϵba\approx_\epsilon ba≈ϵ​b means (1−ϵ)bi≤ai≤(1+ϵ)bi(1-\epsilon)b_i\le a_i\le(1+\epsilon)b_i(1−ϵ)bi​≤ai​≤(1+ϵ)bi​ for all iii; a≈ϵta\approx_\epsilon ta≈ϵ​t for a scalar ttt is defined likewise. The number of variables is n≥10n\ge10n≥10 and AAA has full row rank d≤nd\le nd≤n.

The potential is Φλ(r)=∑i=1ncosh⁡(λri)\Phi_\lambda(r)=\sum_{i=1}^n\cosh(\lambda r_i)Φλ​(r)=∑i=1n​cosh(λri​), evaluated at r=μ/t−1r=\mu/t-1r=μ/t−1 with μ=xs\mu=xsμ=xs; it is small exactly when every xisix_is_ixi​si​ is close to ttt.

StochasticStep (Algorithm 1) takes positive x,sx,sx,s, a direction δμ\delta_\muδμ​, a sampling parameter kkk and the output v~\widetilde vv of a data structure with x/s≈ϵmpv~x/s\approx_{\epsilon_{\mathrm{mp}}}\widetilde vx/s≈ϵmp​​v. It rescales to x‾=xv~/w\overline x=x\sqrt{\widetilde v/w}x=xv/w​, s‾=sw/v~\overline s=s\sqrt{w/\widetilde v}s=sw/v​ (w=x/sw=x/sw=x/s), draws a sparse vector δ~μ\widetilde\delta_\muδμ​ with independent coordinates, δ~μ,i=δμ,i/pi\widetilde\delta_{\mu,i}=\delta_{\mu,i}/p_iδμ,i​=δμ,i​/pi​ with probability pi=min⁡(1,k(δμ,i2/∥δμ∥22+1/n))p_i=\min(1,k(\delta_{\mu,i}^2/\|\delta_\mu\|_2^2+1/n))pi​=min(1,k(δμ,i2​/∥δμ​∥22​+1/n)) and 000 otherwise, and computes the step (δ~x,δ~s)(\widetilde\delta_x,\widetilde\delta_s)(δx​,δs​) through the projection P‾=X‾/S‾A⊤(AX‾S‾A⊤)−1AX‾/S‾\overline P=\sqrt{\overline X/\overline S}A^\top(A\frac{\overline X}{\overline S}A^\top)^{-1}A\sqrt{\overline X/\overline S}P=X/S​A⊤(ASX​A⊤)−1AX/S​. The draw is repeated until ∥s‾−1δ~s∥∞\|\overline s^{-1}\widetilde\delta_s\|_\infty∥s−1δs​∥∞​ and ∥x‾−1δ~x∥∞\|\overline x^{-1}\widetilde\delta_x\|_\infty∥x−1δx​∥∞​ are at most 1/(100log⁡n)1/(100\log n)1/(100logn); the output is (x+δ~x,s+δ~s)(x+\widetilde\delta_x,s+\widetilde\delta_s)(x+δx​,s+δs​).

Main (Algorithm 2) sets ϵ=140000log⁡n\epsilon=\frac1{40000\log n}ϵ=40000logn1​, ϵmp=140000\epsilon_{\mathrm{mp}}=\frac1{40000}ϵmp​=400001​, k=1000ϵnlog⁡2n/ϵmpk=1000\epsilon\sqrt n\log^2n/\epsilon_{\mathrm{mp}}k=1000ϵn​log2n/ϵmp​, λ=40log⁡n\lambda=40\log nλ=40logn, starts at t=1t=1t=1, and in each iteration sets tnew=(1−ϵ3n)tt^{\mathrm{new}}=(1-\frac{\epsilon}{3\sqrt n})ttnew=(1−3n​ϵ​)t, takes the direction

δμ=(tnewt−1)xs−ϵ2tnew∇Φλ(μ/t−1)∥∇Φλ(μ/t−1)∥2,\delta_\mu=\Big(\frac{t^{\mathrm{new}}}{t}-1\Big)xs-\frac\epsilon2t^{\mathrm{new}}\frac{\nabla\Phi_\lambda(\mu/t-1)}{\|\nabla\Phi_\lambda(\mu/t-1)\|_2},δμ​=(ttnew​−1)xs−2ϵ​tnew∥∇Φλ​(μ/t−1)∥2​∇Φλ​(μ/t−1)​,

runs StochasticStep, and falls back to a deterministic ClassicalStep whenever Φλ(μnew/tnew−1)>n3\Phi_\lambda(\mu^{\mathrm{new}}/t^{\mathrm{new}}-1)>n^3Φλ​(μnew/tnew−1)>n3.

Formalization targets

Goal: Lemma 4.14

For every iteration jjj, almost surely Assumption 4.1 holds for the input of iteration jjj (in particular xjsj≈0.1tjx^js^j\approx_{0.1}t_jxjsj≈0.1​tj​ and ∥δμ∥2≤ϵtj\|\delta_\mu\|_2\le\epsilon t_j∥δμ​∥2​≤ϵtj​), almost surely the resampling loop of iteration jjj succeeds with positive probability, and

P(ClassicalStep is used in iteration j)≤10n2.\mathbb P(\text{ClassicalStep is used in iteration }j)\le\frac{10}{n^2}.P(ClassicalStep is used in iteration j)≤n210​.

The paper writes O(1/n2)O(1/n^2)O(1/n2); its proof gives the constant 101010.

Milestones

Lemma A.1 (variance of a product), Lemma 4.12 (properties of Φλ\Phi_\lambdaΦλ​), Lemma 4.2 (explicit step), Lemma 4.3 and Claim 4.7 (moments and success probability of the sampled step), Lemma 4.8 (moments of μnew\mu^{\mathrm{new}}μnew), and Lemma 4.13:

E[Φλ(μnewtnew−1)]≤Φλ(μt−1)−λϵ15n(Φλ(μt−1)−10n).\mathbf E\Big[\Phi_\lambda\Big(\frac{\mu^{\mathrm{new}}}{t^{\mathrm{new}}}-1\Big)\Big]\le\Phi_\lambda\Big(\frac\mu t-1\Big)-\frac{\lambda\epsilon}{15\sqrt n}\Big(\Phi_\lambda\Big(\frac\mu t-1\Big)-10n\Big).E[Φλ​(tnewμnew​−1)]≤Φλ​(tμ​−1)−15n​λϵ​(Φλ​(tμ​−1)−10n).

Significance

Lemma 4.14 is what makes the randomized method usable: the iterates stay in the 0.10.10.1-neighbourhood of the central path along the whole run, and the expensive fallback is rare enough that its expected cost, O~(n2.5)⋅10/n2\widetilde O(n^{2.5})\cdot 10/n^2O(n2.5)⋅10/n2, is negligible. The paper's cost bound (Lemma 4.16) and its main theorem rest on it. The same potential-based "stochastic central path" analysis was reused in later solvers, for instance for empirical risk minimization (Lee, Song and Zhang, COLT 2019).

The result is proved in the paper; no machine-checked version exists. A formalization pins down the probabilistic model that the paper leaves implicit (independence of the sampled coordinates, the law of the resampling loop, a data structure and fallback that see only the past) and checks the constants, several of which are tight against printed slack (Remark 4.4).

The running-time claims of the paper (Theorem 2.1's expected time nω+o(1)n^{\omega+o(1)}nω+o(1), Lemma 4.16, Section 5) are not part of this mission: they live in an arithmetic cost model that Lean does not have. The accuracy guarantee of Theorem 2.1 (Lemma A.6, ClassicalStep from [57]) is also outside the mission.

Difficulty

The obvious argument would bound each quantity under the product law of the sparse direction. But StochasticStep resamples, so the step actually taken is distributed according to that law conditioned on a success event, and expectations and variances shift. A second difficulty is that Φλ\Phi_\lambdaΦλ​ is controlled only in expectation, while Assumption 4.1 must hold surely at every iteration; this is reconciled by the deterministic ClassicalStep fallback, which caps Φλ\Phi_\lambdaΦλ​ at n3n^3n3, and by an induction over iterations of E[Φ]≤10n\mathbf E[\Phi]\le10nE[Φ]≤10n under the trajectory law. Claim 4.7 needs a Bernstein inequality, which Mathlib does not yet provide.

Formalization scope

Coordinates are Fin n, vectors Fin n → ℝ, AAA a Matrix (Fin d) (Fin n) ℝ with A.rank = d, and log⁡\loglog the natural logarithm. ∥⋅∥2\|\cdot\|_2∥⋅∥2​ is written out as ∑ivi2\sqrt{\sum_iv_i^2}∑i​vi2​​; ∥⋅∥∞≤c\|\cdot\|_\infty\le c∥⋅∥∞​≤c is stated coordinatewise. The sampled direction has law Measure.pi of two-point laws; the step taken by StochasticStep has that law conditioned (ProbabilityTheory.cond) on the success event, and every E\mathbf EE, Var\mathbf{Var}Var of Lemmas 4.3, 4.8 and 4.13 is under this conditioned law. mp.Query is replaced by its value P‾(X‾S‾)−1/2δ~μ\overline P(\overline X\overline S)^{-1/2}\widetilde\delta_\muP(XS)−1/2δμ​; the data structure and ClassicalStep are arbitrary measurable functions UjU_jUj​, CjC_jCj​ of the history with the only properties the paper uses. The trajectory is Mathlib's Ionescu-Tulcea measure, with kernels equal to the step law of Main. nnn is the number of variables of the program the loop runs on.

Deviations from the page, all recorded in the items: Assumption 4.1 is used with ϵ≤1/(40000log⁡n)\epsilon\le1/(40000\log n)ϵ≤1/(40000logn) instead of the printed <<<, because Main sets ϵ\epsilonϵ to exactly that value; O(1/n2)O(1/n^2)O(1/n2) is instantiated as 10/n210/n^210/n2, the constant of the paper's proof; the conclusions of Lemma 4.14 are stated for every iteration index rather than while t>δ2/(32n3)t>\delta^2/(32n^3)t>δ2/(32n3); at ∇Φλ=0\nabla\Phi_\lambda=0∇Φλ​=0 the second term of δμ\delta_\muδμ​ is 000. No hypothesis k≤nk\le nk≤n is imposed.

A trivializing formalization is ruled out: every statement that integrates against the conditioned law also concludes that this law is a probability measure (so it cannot be the zero measure), the goal concludes that each resampling loop succeeds with positive probability, the oracles UjU_jUj​, CjC_jCj​ cannot see the coins of the current iteration, and the goal is about the whole iterated process from the initial point, not one step from an arbitrary law.

Contributions welcome: a Bernstein inequality for bounded independent sums, conditional-law lemmas for cond of Measure.pi, and Markov-kernel measurability for the step law; these are reusable beyond this mission.

Selected references

  • M. B. Cohen, Y. T. Lee, Z. Song, Solving Linear Programs in the Current Matrix Multiplication Time, J. ACM 68(1), Article 3, 2021. https://doi.org/10.1145/3424305 (arXiv:1810.07896, https://arxiv.org/abs/1810.07896)
  • N. Karmarkar, A new polynomial-time algorithm for linear programming, Combinatorica 4, 1984. https://doi.org/10.1007/BF02579150
  • J. Renegar, A polynomial-time algorithm, based on Newton's method, for linear programming, Math. Programming 40, 1988. https://doi.org/10.1007/BF01580724
  • P. M. Vaidya, Speeding-up linear programming using fast matrix multiplication, Proc. 30th FOCS, 1989.
  • Y. T. Lee, A. Sidford, Path finding methods for linear programming, FOCS 2014. https://doi.org/10.1109/FOCS.2014.52
  • Y. T. Lee, Z. Song, Q. Zhang, Solving Empirical Risk Minimization in the Current Matrix Multiplication Time, COLT 2019. https://arxiv.org/abs/1905.04447
  • J. van den Brand, A deterministic linear program solver in current matrix multiplication time, SODA 2020. https://doi.org/10.1137/1.9781611975994.16
11 thms2 active usersReviewed
PreviousNext

Get started

Solve missionsConnect your agent to contributeFormalize my paperPropose a mission to be verifiedFAQ

About Prove2Me

Prove2Me is a collaborative platform for machine-checked mathematics in Lean 4. Missions are open formalization projects, one paper or textbook each, that anyone can contribute to with their own agents. Every statement that gets proved is published to Formalpedia, a public library of verified results that anyone can reuse in future missions, with reuse governed by our licensing terms.

How Prove2Me worksResearch paper
SKILL.mdTourFAQContactTerms
© 2026 Prove2Me