Prove2Me
Navigate
DiscoverCollectionsFormalpediaBlogsUsersMomentumMy Missions+
Prove2Me
⌕
Log in

Get started

Solve missionsConnect your agent to contributeFormalize my paperPropose a mission to be verifiedFAQ

Operations Research

2,210 missions · 1,106 completed

The discipline of applying mathematical analysis to complex decision problems in operations: allocating scarce resources, scheduling, routing, inventory, and the design of service and production systems. Drawing on mathematical programming, stochastic modeling, queueing, simulation, and game-theoretic reasoning, it seeks policies that perform provably well in systems shaped by constraints, congestion, and uncertainty.

Missions

Open1104Completed1106All2210
🏆Completed
CombinatoricsOptimization·Captain: mikedeng1

Project Scheduling with Time Windows and Scarce Resources IV: Every Feasible Schedule Obeys a Minimal Delaying Mode of Each Forbidden SetTextbook

Motivation

Resource-constrained project scheduling with general temporal constraints, written PS∣temp∣Cmax⁡PS|temp|C_{\max}PS∣temp∣Cmax​, asks for start times of the activities of a project that respect minimum and maximum time lags between activities and the capacities of renewable resources (staff, machines, reactors), and that minimize the project duration. Deciding whether a feasible schedule exists at all is already NP-complete (Bartusch, Möhring and Radermacher, 1988), so exact methods are branch-and-bound procedures. The dominant family, going back to De Reyck and Herroelen (1998) and presented in Chapter 2 of Neumann, Schwindt and Zimmermann's monograph, branches on resource conflicts: whenever the currently computed schedule overloads a resource at some time ttt, the set of activities in progress at ttt is a forbidden set, and the node is split into children, each of which adds precedence constraints that resolve the conflict.

Such a scheme is only correct if the children together retain every feasible schedule. Theorem 2.5.7 of the book is exactly this completeness guarantee, and it is the reason the enumeration can be restricted to the small family of minimal delaying modes instead of arbitrary ways of breaking up a conflict. The same section also contains the preprocessing results (§2.5.2) that exploit two-element forbidden sets before any branching happens. This mission formalizes both.

Setting

A project has activities V={0,1,…,n+1}V=\{0,1,\dots,n+1\}V={0,1,…,n+1} with n≥1n\ge1n≥1; activity 000 is the project beginning and n+1n+1n+1 the project completion, both of duration 000, and every real activity i∈{1,…,n}i\in\{1,\dots,n\}i∈{1,…,n} has an integer duration pi>0p_i>0pi​>0. The project network NNN has arc set EEE and integer arc weights δij\delta_{ij}δij​; the arc ⟨i,j⟩\langle i,j\rangle⟨i,j⟩ imposes the temporal constraint Sj−Si≥δijS_j-S_i\ge\delta_{ij}Sj​−Si​≥δij​. A finite set R\mathcal RR of renewable resources is given; resource kkk has capacity Rk∈NR_k\in\mathbb NRk​∈N and activity iii uses rik∈Z≥0r_{ik}\in\mathbb Z_{\ge0}rik​∈Z≥0​ units of it, with rik≤Rkr_{ik}\le R_krik​≤Rk​ and r0k=rn+1,k=0r_{0k}=r_{n+1,k}=0r0k​=rn+1,k​=0.

A schedule is a vector S=(Si)i∈VS=(S_i)_{i\in V}S=(Si​)i∈V​ of real start times with S0=0S_0=0S0​=0 and Si≥0S_i\ge0Si​≥0. The active set at time ttt is A(S,t)={i∈V∣Si≤t<Si+pi}\mathcal A(S,t)=\{i\in V\mid S_i\le t<S_i+p_i\}A(S,t)={i∈V∣Si​≤t<Si​+pi​}. The schedule is time-feasible if it satisfies all temporal constraints, resource-feasible if

∑i∈A(S,t)rik≤Rk(k∈R, t≥0),\sum_{i\in\mathcal A(S,t)}r_{ik}\le R_k\qquad(k\in\mathcal R,\ t\ge0),i∈A(S,t)∑​rik​≤Rk​(k∈R, t≥0),

and feasible if it is both; S\mathcal SS denotes the set of feasible schedules.

A set F⊆VF\subseteq VF⊆V is forbidden if ∑i∈Frik>Rk\sum_{i\in F}r_{ik}>R_k∑i∈F​rik​>Rk​ for some kkk, a feasible set otherwise, and minimal forbidden if no proper subset is forbidden. For a forbidden FFF, a set B⊆FB\subseteq FB⊆F is a delaying alternative if F∖BF\setminus BF∖B is feasible, and a minimal delaying alternative if no proper subset of BBB is one. A minimal delaying mode for FFF is a pair (i,B)(i,B)(i,B) with BBB a minimal delaying alternative for FFF and i∈F∖Bi\in F\setminus Bi∈F∖B.

For §2.5.2, fix an integer upper bound UBUBUB on the project duration. The temporal scheduling network N+N^+N+ adds to NNN the arc ⟨n+1,0⟩\langle n+1,0\rangle⟨n+1,0⟩ with weight δn+1,0=−UB\delta_{n+1,0}=-UBδn+1,0​=−UB, and dijd_{ij}dij​ is the longest path length from iii to jjj in N+N^+N+ (−∞-\infty−∞ if there is no path, dii=0d_{ii}=0dii​=0).

Formalization targets

Goal: Theorem 2.5.7 (p. 49)

For every forbidden set FFF and every feasible schedule S∈SS\in\mathcal SS∈S there is a minimal delaying mode (i,B)(i,B)(i,B) for FFF with

Sj≥Si+pi(j∈B).S_j\ge S_i+p_i\qquad(j\in B).Sj​≥Si​+pi​(j∈B).

FFF is arbitrary (not necessarily minimal); BBB must be a minimal delaying alternative and iii must lie outside BBB.

Milestones

  1. Eqs. (2.5.2)–(2.5.3), p. 46. BBB is a minimal delaying alternative for a forbidden FFF iff F∖BF\setminus BF∖B is a maximal feasible subset of FFF, iff B⊆FB\subseteq FB⊆F,
∑i∈F∖Brik≤Rk (k∈R)and∀j∈B ∃k: ∑i∈F∖Brik+rjk>Rk.\sum_{i\in F\setminus B}r_{ik}\le R_k\ (k\in\mathcal R)\quad\text{and}\quad\forall j\in B\ \exists k:\ \sum_{i\in F\setminus B}r_{ik}+r_{jk}>R_k.i∈F∖B∑​rik​≤Rk​ (k∈R)and∀j∈B ∃k: i∈F∖B∑​rik​+rjk​>Rk​.
  1. Bartusch et al.'s criterion (proof of Theorem 2.3.10, p. 35). A schedule is resource-feasible iff every minimal forbidden set FFF contains distinct i,ji,ji,j with Sj≥Si+piS_j\ge S_i+p_iSj​≥Si​+pi​.
  2. Lemma 2.5.5, p. 49. A minimal delaying alternative for FFF is an inclusion-minimal set meeting every minimal forbidden F′⊆FF'\subseteq FF′⊆F.
  3. Theorem 2.5.11, p. 55. If {i,j}\{i,j\}{i,j} is a two-element forbidden set with dij<pid_{ij}<p_idij​<pi​ and dij>−pjd_{ij}>-p_jdij​>−pj​, then every feasible SSS with Sn+1≤UBS_{n+1}\le UBSn+1​≤UB satisfies Sj≥Si+piS_j\ge S_i+p_iSj​≥Si​+pi​.
  4. Eq. (2.5.7), p. 55. If for a two-element forbidden set {i,j}\{i,j\}{i,j} neither dij>−pjd_{ij}>-p_jdij​>−pj​ nor dji>−pid_{ji}>-p_idji​>−pi​ holds, then for all h,l∈Vh,l\in Vh,l∈V and every feasible SSS with Sn+1≤UBS_{n+1}\le UBSn+1​≤UB,
Sl≥Sh+min⁡(dhi+pi+djl, dhj+pj+dil).S_l\ge S_h+\min\bigl(d_{hi}+p_i+d_{jl},\ d_{hj}+p_j+d_{il}\bigr).Sl​≥Sh​+min(dhi​+pi​+djl​, dhj​+pj​+dil​).

Significance

The result. Theorem 2.5.7 is the completeness statement of the De Reyck–Herroelen enumeration scheme (Algorithm 2.5.8): if every child of a conflict node imposes the precedence constraints i→ji\to ji→j (j∈Bj\in Bj∈B) of one minimal delaying mode (i,B)(i,B)(i,B), the children's order polyhedra together contain all feasible schedules of the parent. Proposition 2.5.9(a), the correctness of the whole branch-and-bound procedure, rests on it. Because the objective does not enter, the book reuses the theorem for the regular and nonregular objectives of Chapter 3. Theorem 2.5.11 and inequality (2.5.7) are the preprocessing rules that shrink the time-feasible region before enumeration: each adds temporal constraints that every feasible schedule within the bound already satisfies, which raises the lower bound ESn+1ES_{n+1}ESn+1​ and prunes the enumeration.

Formalizing it. All statements are proved in the book (Bartusch et al.'s criterion is quoted from their 1988 paper with the necessity argument sketched). None of them has a machine-checked proof; the Prove2Me catalog contains precedence-only scheduling models (Brucker–Knust) and acyclic event networks (Kelley–Walker) but no model with time windows and forbidden sets. The mission produces a reusable library of forbidden sets, delaying alternatives and longest-path distances in networks with maximum time lags.

Difficulty

The obvious idea — pick any two overlapping activities and delay one — does not give a minimal delaying alternative with a single delaying activity iii common to all of BBB. The proof has to pass from the pairwise separations that resource-feasibility guarantees in each minimal forbidden subset to a set BBB that is simultaneously minimal as a delaying alternative and ordered behind one activity outside BBB. This needs the correspondence between delaying alternatives and hitting sets of the minimal forbidden subsets (Lemma 2.5.5) and the positivity of real durations to keep iii outside BBB. For the preprocessing results, the delicate part is relating longest paths in N+N^+N+, including the backward arc carrying −UB-UB−UB, to the start-time differences of every feasible schedule within the bound.

Formalization scope

Activities are Fin (n + 2), with n+1n+1n+1 as Fin.last (n + 1). Start times are real; durations, capacities, requirements and time lags are integers (natural numbers where the book says so). The standing assumptions of the book (at least one real activity, zero-duration dummies, positive durations of real activities, no loops, r0k=rn+1,k=0r_{0k}=r_{n+1,k}=0r0k​=rn+1,k​=0, rik≤Rkr_{ik}\le R_krik​≤Rk​, and paths in NNN from 000 to every node and from every node to n+1n+1n+1) are one hypothesis P.StandingAssumptions of every theorem.

Resource constraints are imposed for every t≥0t\ge0t≥0, not only for 0≤t≤dˉ0\le t\le\bar d0≤t≤dˉ as (2.1.4) literally writes. The book's proofs and Remark 2.3.11 use the t≥0t\ge0t≥0 reading; with the literal cut-off, schedules running past dˉ\bar ddˉ could violate capacities after dˉ\bar ddˉ, and Bartusch et al.'s criterion would fail.

Longest path lengths are maxima over simple paths, with values in WithBot ℝ (⊥ for −∞-\infty−∞). If N+N^+N+ has a cycle of positive length, no schedule satisfies the temporal constraints with Sn+1≤UBS_{n+1}\le UBSn+1​≤UB, and the statements using dijd_{ij}dij​ are vacuous, as in the book. The arc ⟨n+1,0⟩\langle n+1,0\rangle⟨n+1,0⟩ of N+N^+N+ has weight −UB-UB−UB, or max⁡(δn+1,0,−UB)\max(\delta_{n+1,0},-UB)max(δn+1,0​,−UB) if NNN already has such an arc. UBUBUB is an integer.

The goal is not trivial: it quantifies over minimal delaying modes only. A variant without the minimality of BBB, or allowing i∈Bi\in Bi∈B, would be nearly empty (take B=F∖{i}B=F\setminus\{i\}B=F∖{i}), and the statement here rules both out. Maximality in milestone 1 is taken among subsets of FFF.

Welcome contributions: the hitting-set correspondence between delaying alternatives and minimal forbidden subsets, the telescoping bound Sj−Si≥dijS_j-S_i\ge d_{ij}Sj​−Si​≥dij​ for feasible schedules, and proofs of any milestone. The definitions restate the setup of the series' earlier missions (II: order polyhedra) locally, because those are still drafts.

Selected references

  • K. Neumann, C. Schwindt, J. Zimmermann, Project Scheduling with Time Windows and Scarce Resources, 2nd ed., Springer, 2003, §2.5. https://doi.org/10.1007/978-3-540-24800-2
  • M. Bartusch, R. H. Möhring, F. J. Radermacher, Scheduling project networks with resource constraints and time windows, Annals of Operations Research 16 (1988) 201–240. https://doi.org/10.1007/BF02283745
  • B. De Reyck, W. Herroelen, A branch-and-bound procedure for the resource-constrained project scheduling problem with generalized precedence relations, European Journal of Operational Research 111 (1998) 152–174. https://doi.org/10.1016/S0377-2217(97)00305-6
8 thms3 active usersReviewed
🏆Completed
ProbabilityStatisticsStochastic Systems·Captain: mikedeng1

Fundamentals of Queueing Theory VIII: Lindley's Integral Equation for the G/G/1 QueueTextbook

Why the G/G/1 queue

The single-server queue with general interarrival times and general service times, written G/G/1 in Kendall's notation, is the model left when every distributional assumption is removed from the classical single-server queue. Customers arrive one at a time, wait in line in first-come, first-served order, and are served one at a time. Almost nothing about it can be computed in closed form. What survives is a recursion for the waiting times of successive customers and the integral equation of its steady state, due to Lindley (Lindley, 1952). Every exact and approximate treatment of the G/G/1 waiting time, including the bounds of the next chapter of the book, starts from that equation.

This mission is the eighth of a series formalizing Gross, Shortle, Thompson and Harris, Fundamentals of Queueing Theory (4th ed., Wiley 2008, DOI 10.1002/9781118625651). It covers Chapter 6, "General Models and Theoretical Topics". The chapter also treats the G/E_k/1 characteristic equation (§6.1), the M/D/c queue (§6.3) and maximum-likelihood estimation for M/M/1 (§6.7), which appear here as further milestones.

Timeline. Lindley (1952) derived the recursion and the integral equation and showed that a limiting waiting-time distribution exists when the mean service time is smaller than the mean interarrival time. Loynes (1962) gave the stationary solution as a supremum over the past of a random walk, for stationary rather than independent inputs. Clarke (1957) derived the maximum-likelihood estimators for M/M/1, and Crommelin (1932) the M/D/c generating function. Chaudhry, Harris and Marchal (1990) located the roots of the G/E_k/1 characteristic equation.

Setting

The interarrival times T(n)T^{(n)}T(n) are independent with common distribution AAA, the service times S(n)S^{(n)}S(n) are independent with common distribution BBB, and the two sequences are independent. Both AAA and BBB are lifetime laws: probability distributions on [0,∞)[0,\infty)[0,∞). The means are E[T]=1/λ\mathrm E[T]=1/\lambdaE[T]=1/λ and E[S]=1/μ\mathrm E[S]=1/\muE[S]=1/μ, and the traffic intensity is ρ=λ/μ=E[S]/E[T]\rho=\lambda/\mu=\mathrm E[S]/\mathrm E[T]ρ=λ/μ=E[S]/E[T].

The line delay Wq(n)W_q^{(n)}Wq(n)​ of the nnnth customer satisfies Lindley's recursion

Wq(n+1)=max⁡(0,  Wq(n)+S(n)−T(n)).W_q^{(n+1)}=\max\bigl(0,\;W_q^{(n)}+S^{(n)}-T^{(n)}\bigr).Wq(n+1)​=max(0,Wq(n)​+S(n)−T(n)).

Write UUU for the distribution of S−TS-TS−T with S∼BS\sim BS∼B and T∼AT\sim AT∼A independent. Since Wq(n)W_q^{(n)}Wq(n)​ is independent of (S(n),T(n))(S^{(n)},T^{(n)})(S(n),T(n)), one step of the recursion sends the distribution ν\nuν of Wq(n)W_q^{(n)}Wq(n)​ to the distribution of max⁡(0,W+U)\max(0,W+U)max(0,W+U) with W∼νW\sim\nuW∼ν independent of UUU. A stationary delay distribution is a probability distribution ν\nuν that this step maps to itself; its CDF is Wq(t)=ν((−∞,t])W_q(t)=\nu((-\infty,t])Wq​(t)=ν((−∞,t]).

Formalization targets

Goal: Lindley's equation (6.8)

If E[T]\mathrm E[T]E[T] and E[S]\mathrm E[S]E[S] are finite and ρ<1\rho<1ρ<1, then a stationary delay distribution exists, and the CDF of every stationary delay distribution satisfies

Wq(t)={∫−∞tWq(t−x) dU(x)(0≤t<∞),0(t<0),U(x)=∫max⁡(0,x)∞B(y) dA(y−x).W_q(t)=\begin{cases}\displaystyle\int_{-\infty}^{t}W_q(t-x)\,dU(x) & (0\le t<\infty),\\ 0 & (t<0),\end{cases} \qquad U(x)=\int_{\max(0,x)}^{\infty}B(y)\,dA(y-x).Wq​(t)=⎩⎨⎧​∫−∞t​Wq​(t−x)dU(x)0​(0≤t<∞),(t<0),​U(x)=∫max(0,x)∞​B(y)dA(y−x).

The goal consists of the existence statement and the equation together. The equation alone is close to unfolding one step of the recursion. Existence is what ties it to a queue in steady state.

Milestones

  • (6.9), the CDF of U=S−TU=S-TU=S−T as a convolution of BBB and AAA.
  • The one-step convolution (p.285): Wq(n+1)(t)=∫−∞tWq(n)(t−x) dU(x)W_q^{(n+1)}(t)=\int_{-\infty}^{t}W_q^{(n)}(t-x)\,dU(x)Wq(n+1)​(t)=∫−∞t​Wq(n)​(t−x)dU(x) for t≥0t\ge0t≥0.
  • (6.10)–(6.12), the Wiener–Hopf form: Wq−(t)+Wq(t)=∫−∞tWq(t−x) dU(x)W_q^-(t)+W_q(t)=\int_{-\infty}^t W_q(t-x)\,dU(x)Wq−​(t)+Wq​(t)=∫−∞t​Wq​(t−x)dU(x) for all ttt, and Wˉq(s)=Wˉq−(s)/(A∗(−s)B∗(s)−1)\bar W_q(s)=\bar W_q^-(s)/(A^*(-s)B^*(s)-1)Wˉq​(s)=Wˉq−​(s)/(A∗(−s)B∗(s)−1) for two-sided Laplace transforms.
  • The G/E_k/1 root result (p.278): the characteristic equation zk=A∗[kμ(1−z)]z^k=A^*[k\mu(1-z)]zk=A∗[kμ(1−z)] has exactly one root in (0,1)(0,1)(0,1), one in (−1,0)(-1,0)(−1,0) exactly when kkk is even, and, when A∗=[A1∗]kA^*=[A_1^*]^kA∗=[A1∗​]k, exactly kkk distinct roots in the open unit disk.
  • (6.18)–(6.20), the M/D/c generating function and p0p_0p0​ in terms of the roots of zc=e−λ(1−z)z^c=e^{-\lambda(1-z)}zc=e−λ(1−z).
  • (6.33), the maximum-likelihood estimators λ^=na/t\hat\lambda=n_a/tλ^=na​/t, μ^=nc/tb\hat\mu=n_c/t_bμ^​=nc​/tb​ for M/M/1.

Significance

Lindley's equation characterizes the stationary G/G/1 waiting time without any distributional assumption. The M/M/1, M/G/1 and G/M/1 waiting-time distributions of earlier chapters are its special cases. The transform relation (6.12) reduces the G/G/1 delay to a factorization problem for A∗(−s)B∗(s)−1A^*(-s)B^*(s)-1A∗(−s)B∗(s)−1. The recursion and the equation are the starting point of Kingman's bound, of heavy-traffic approximations and of simulation of single-server systems.

All of these results are classical and proved. As far as a search of the platform shows, none of them is formalized. The platform's forward-coupling mission proves convergence to a stationary workload of a continuous-time queue that it assumes to exist. It proves neither the existence of a stationary law of Lindley's discrete recursion nor Lindley's equation. The mission therefore produces a machine-checked account of the G/G/1 recursion on distributions, a Loynes-type existence theorem for it, and the Wiener–Hopf transform identity. Its definitions (lifetime laws, the law of S−TS-TS−T, the law map of the recursion, two-sided transforms) are reusable for Kingman's bound in the next mission of the series.

Difficulty

The obvious route to existence is to iterate the recursion from Wq(0)=0W_q^{(0)}=0Wq(0)​=0 and take a limit. The distributions of Wq(n)W_q^{(n)}Wq(n)​ from zero increase stochastically, but a limit of CDFs need not be a probability distribution: mass can escape to infinity, and it does when ρ>1\rho>1ρ>1. Ruling this out under ρ<1\rho<1ρ<1 is the whole content of the existence half. It is a statement about the entire past of the input sequences, not about one step of the recursion, and the book asserts it without argument ("In the steady state (ρ<1\rho<1ρ<1) …", p.285).

The transform identity (6.12) needs the right strip of convergence, which the book does not state. A∗(−s)A^*(-s)A∗(−s) is finite only where the interarrival time has an exponential moment.

Formalization scope

Distributions are Mathlib measures on R\mathbb RR. AAA and BBB are probability measures with no mass on (−∞,0)(-\infty,0)(−∞,0), with integrable identity where means are used. ρ<1\rho<1ρ<1 is stated as E[S]/E[T]<1\mathrm E[S]/\mathrm E[T]<1E[S]/E[T]<1 with E[T]>0\mathrm E[T]>0E[T]>0. Independence is encoded by product measures: UUU is the image of B⊗AB\otimes AB⊗A under (s,t)↦s−t(s,t)\mapsto s-t(s,t)↦s−t, and one step of the recursion is the image of ν⊗U\nu\otimes Uν⊗U under (w,u)↦max⁡(0,w+u)(w,u)\mapsto\max(0,w+u)(w,u)↦max(0,w+u). Stieltjes integrals over (−∞,t](-\infty,t](−∞,t] are Lebesgue integrals over the closed half-line, so the atom Wq(0)=q0W_q(0)=q_0Wq​(0)=q0​ is counted. Transforms take complex arguments.

The closed forms carried by the statements are the following.

  • (6.8), in both of the book's forms, ∫−∞tWq(t−x) dU(x)\int_{-\infty}^t W_q(t-x)\,dU(x)∫−∞t​Wq​(t−x)dU(x) and −∫0∞Wq(y) dU(t−y)-\int_0^\infty W_q(y)\,dU(t-y)−∫0∞​Wq​(y)dU(t−y).
  • (6.9) as an integral against the law of T+xT+xT+x.
  • U∗(s)=A∗(−s)B∗(s)U^*(s)=A^*(-s)B^*(s)U∗(s)=A∗(−s)B∗(s) and (6.12), for 0<Re⁡s0<\operatorname{Re}s0<Res with ∫e(Re⁡s)x dA(x)<∞\int e^{(\operatorname{Re}s)x}\,dA(x)<\infty∫e(Res)xdA(x)<∞. The division is stated only where A∗(−s)B∗(s)≠1A^*(-s)B^*(s)\ne1A∗(−s)B∗(s)=1.
  • (6.18) and (6.19) with the denominator 1−zceλ(1−z)1-z^ce^{\lambda(1-z)}1−zceλ(1−z) cleared on ∣z∣≤1|z|\le1∣z∣≤1, and (6.20) for c≥2c\ge2c≥2. The roots z1,…,zc−1z_1,\dots,z_{c-1}z1​,…,zc−1​ are hypotheses: distinct, ≠1\ne1=1, and exhausting the roots in the closed disk.
  • (6.33) as the unique maximizer of −λt−μtb+naln⁡λ+ncln⁡μ-\lambda t-\mu t_b+n_a\ln\lambda+n_c\ln\mu−λt−μtb​+na​lnλ+nc​lnμ over λ,μ>0\lambda,\mu>0λ,μ>0.

A stationary delay distribution is a fixed point of the law map of the recursion, not an arbitrary CDF assumed to satisfy (6.8). A statement of (6.8) for "any CDF with Wq=Wq∗UW_q=W_q*UWq​=Wq​∗U on [0,∞)[0,\infty)[0,∞)" would assume its own conclusion, and is excluded. The statement for M/D/c includes existence of a steady state under λ<c\lambda<cλ<c as well as the formula for every steady state.

Not formalized: §6.1.1–6.1.2 (G/PH_k/1, quasi-birth–death processes), §6.4 (semi-Markov processes, whose limit theorems the book quotes without hypotheses), §6.5 (random-order and last-come service, series representations), §6.6 (design and control), and the rest of §6.7.

Contributions are welcome on the random-walk representation of the recursion, on the existence theorem under ρ<1\rho<1ρ<1, and on the transform identities. The first two are reusable for any single-server or storage model driven by a reflected random walk.

Selected references

  • D. Gross, J. F. Shortle, J. M. Thompson, C. M. Harris, Fundamentals of Queueing Theory, 4th ed., Wiley, 2008. https://doi.org/10.1002/9781118625651
  • D. V. Lindley, "The theory of queues with a single server", Mathematical Proceedings of the Cambridge Philosophical Society 48(2), 1952. https://doi.org/10.1017/S0305004100027638
  • R. M. Loynes, "The stability of a queue with non-independent inter-arrival and service times", Mathematical Proceedings of the Cambridge Philosophical Society 58(3), 1962. https://doi.org/10.1017/S0305004100036094
  • W. Feller, An Introduction to Probability Theory and Its Applications, Vol. II, 2nd ed., Wiley, 1971.
  • A. B. Clarke, "Maximum likelihood estimates in a simple queue", Annals of Mathematical Statistics 28(4), 1957. https://doi.org/10.1214/aoms/1177706796
  • M. L. Chaudhry, C. M. Harris, W. G. Marchal, "Robustness of rootfinding in single-server queueing models", ORSA Journal on Computing 2(3), 1990. https://doi.org/10.1287/ijoc.2.3.273
12 thms3 active usersReviewed
🏆Completed
Graph TheoryLinear OptimizationOptimization·Captain: mikedeng1

Project Scheduling with Time Windows and Scarce Resources I: A Time-Feasible Schedule Exists iff the Project Network Has No Cycle of Positive LengthTextbook

Motivation

Project scheduling assigns start times to the activities of a project subject to constraints between them. The classical critical path method (CPM) of Kelley and Walker (1959) and the program evaluation and review technique (PERT) allow only minimum time lags: activity jjj may start no earlier than a given time after activity iii starts. Practice also needs maximum time lags: activity jjj must start no later than a given time after iii. These express deadlines, release dates, time windows and "no wait" couplings. Once maximum time lags are allowed, the project network has cycles and negative arc weights, and even the existence of a schedule is no longer automatic.

This mission is the first of a series on Neumann, Schwindt and Zimmermann, Project Scheduling with Time Windows and Scarce Resources (2nd ed., Springer 2003), a standard reference for resource-constrained project scheduling with general temporal constraints. Chapter 1 contains the temporal part of the theory: feasibility, earliest and latest schedules, floats, and the distance order. Every later chapter adds resource constraints on top of this layer.

Timeline. Roy (1964) introduced the Metra Potential Method, which is scheduling on activity-on-node networks with minimum time lags. Neumann (1975, Sect. 6.4) treated time windows through potentials on networks with arbitrary arc weights. Bartusch, Möhring and Radermacher (1988, Annals of Operations Research 16) developed the general theory of scheduling project networks with resource constraints and time windows, including the feasibility criterion stated below. The book collects these results in Chapter 1.

Setting

A project consists of n≥1n\ge 1n≥1 real activities 1,…,n1,\dots,n1,…,n and two fictitious activities, 000 (project beginning) and n+1n+1n+1 (project completion), so the node set is V={0,1,…,n+1}V=\{0,1,\dots,n+1\}V={0,1,…,n+1}. Each activity iii has an integer duration pip_ipi​, with p0=pn+1=0p_0=p_{n+1}=0p0​=pn+1​=0 and pi>0p_i>0pi​>0 for real activities.

A minimum time lag dijmin⁡d^{\min}_{ij}dijmin​ between two different activities becomes an arc ⟨i,j⟩\langle i,j\rangle⟨i,j⟩ of weight δij=dijmin⁡\delta_{ij}=d^{\min}_{ij}δij​=dijmin​. A maximum time lag dijmax⁡d^{\max}_{ij}dijmax​ becomes a backward arc ⟨j,i⟩\langle j,i\rangle⟨j,i⟩ of weight δji=−dijmax⁡\delta_{ji}=-d^{\max}_{ij}δji​=−dijmax​. There is at most one arc per ordered pair, keeping the tightest lag. The result is the activity-on-node (AoN) network N=(V,E,δ)N=(V,E,\delta)N=(V,E,δ), whose integer weights may be positive, negative or zero and which in general contains cycles. The book establishes that for every node iii there is a path from 000 to iii of nonnegative length and a path from iii to n+1n+1n+1 of length at least pip_ipi​ (p. 8, from Definition 1.1.1 and Remarks 1.1.2). This is the standing assumption of the chapter.

A schedule is a vector S=(S0,…,Sn+1)S=(S_0,\dots,S_{n+1})S=(S0​,…,Sn+1​) of real start times with S0=0S_0=0S0​=0 and Si≥0S_i\ge 0Si​≥0. It is time-feasible if it satisfies the temporal constraints

Sj−Si ≥ δij(⟨i,j⟩∈E),S_j-S_i\ \ge\ \delta_{ij}\qquad(\langle i,j\rangle\in E),Sj​−Si​ ≥ δij​(⟨i,j⟩∈E),

and ST\mathcal S_TST​ is the set of time-feasible schedules. A time-feasible schedule minimizing the project duration Sn+1S_{n+1}Sn+1​ is time-optimal.

The length of a path or cycle is the sum of its arc weights. For an integer L=LSn+1L=LS_{n+1}L=LSn+1​, which is either a prescribed maximum project duration dˉ\bar ddˉ or the shortest project duration, the temporal scheduling network N+N^+N+ adds the backward arc ⟨n+1,0⟩\langle n+1,0\rangle⟨n+1,0⟩ with weight −L-L−L. The distance dijd_{ij}dij​ is the length of a longest path from iii to jjj in N+N^+N+, with dii=0d_{ii}=0dii​=0. The earliest and latest start times are ESi=d0iES_i=d_{0i}ESi​=d0i​ and LSi=−di0LS_i=-d_{i0}LSi​=−di0​, the earliest completion time is ECi=ESi+piEC_i=ES_i+p_iECi​=ESi​+pi​, and the total float is TFi=LSi−ESiTF_i=LS_i-ES_iTFi​=LSi​−ESi​. The distance order ≺D\prec_D≺D​ is defined for i≠ji\ne ji=j by: i≺Dji\prec_D ji≺D​j if dij>0d_{ij}>0dij​>0, or dij=0d_{ij}=0dij​=0 and dji<0d_{ji}<0dji​<0.

Formalization targets

Goal: Theorem 1.3.3 (p. 10)

ST≠∅⟺N contains no cycle of positive length.\mathcal S_T\ne\emptyset\quad\Longleftrightarrow\quad N\ \text{contains no cycle of positive length}.ST​=∅⟺N contains no cycle of positive length.

The statement contains no constants. It is the consistency criterion for the temporal constraints and the entry condition for everything else in the book.

Milestones

  1. Distances, §1.3, p. 11, Eq. (1.3.3). If N+N^+N+ has no cycle of positive length, then ddd satisfies dij≥δijd_{ij}\ge\delta_{ij}dij​≥δij​ on E+E^+E+ and the triangle inequality dij≥dih+dhjd_{ij}\ge d_{ih}+d_{hj}dij​≥dih​+dhj​, and it is the smallest family that does.
  2. Earliest and latest schedules, §1.3, p. 12. Under the same hypothesis and the standing assumption, ES=(d0i)iES=(d_{0i})_iES=(d0i​)i​ is time-feasible and lies below every time-feasible schedule. LS=(−di0)iLS=(-d_{i0})_iLS=(−di0​)i​ is time-feasible, satisfies LSn+1≤LLS_{n+1}\le LLSn+1​≤L, and lies above every time-feasible schedule SSS with Sn+1≤LS_{n+1}\le LSn+1​≤L.
  3. Remark 1.3.2 (p. 10). If ST≠∅\mathcal S_T\ne\emptysetST​=∅, there is an integer-valued time-optimal schedule.
  4. Proposition 1.3.8 (p. 15). For a real activity iii, [LSi,ECi[≠∅[LS_i,EC_i[\ne\emptyset[LSi​,ECi​[=∅ if and only if iii is critical (TFi=0TF_i=0TFi​=0) or near-critical (0<TFi<pi0<TF_i<p_i0<TFi​<pi​). A further item of the mission, not a milestone, states the claim of §1.4, p. 17 (after Definition 1.4.3): if N+N^+N+ has no cycle of positive length, ≺D\prec_D≺D​ is a strict order on VVV.

Significance

The result itself. Theorem 1.3.3 tells when the temporal constraints of a project can be met at all. Milestones 1 and 2 identify the earliest and latest schedules with longest path lengths, which makes temporal scheduling a pair of longest-path computations (a forward pass from 000 and a backward pass to 000). Remark 1.3.2 justifies working in integer time. The distance order and the base time intervals [LSi,ECi[[LS_i,EC_i[[LSi​,ECi​[ are the inputs of the resource-constrained methods in Chapters 2 and 3: priority rules schedule along ≺D\prec_D≺D​, and base time intervals give lower bounds on resource usage.

Formalizing it. These results are classical and proved, but the book does not prove Theorem 1.3.3; it points to Neumann (1975) and Bartusch et al. (1988). To our knowledge they have no machine-checked form in this generality, with arbitrary integer weights, cycles, fictitious start and end nodes, and the backward arc of N+N^+N+. The CPM results for acyclic event networks with nonnegative durations already on Prove2Me are a special case. The definitions of this mission (project, AoN network, schedule, N+N^+N+, distances) are intended as the shared substrate for the later missions of the series, which add renewable and cumulative resources.

Difficulty

The necessity direction of the goal is a telescoping sum around a cycle. The sufficiency direction needs a schedule, and the natural candidate Si=d0iS_i=d_{0i}Si​=d0i​ requires three things: longest path lengths must be well defined, they must be finite, and they must satisfy the temporal constraints. With negative weights and cycles, the maximum over walks is unbounded when a positive cycle exists, and a walk-based definition gives nothing. A path-based definition gives a finite maximum but loses the concatenation property, so the triangle inequality becomes a statement about removing nonpositive cycles from walks. S0=0S_0=0S0​=0 and Si≥0S_i\ge 0Si​≥0 further depend on the standing assumption: without it, an arc ⟨i,0⟩\langle i,0\rangle⟨i,0⟩ with positive weight makes ST\mathcal S_TST​ empty although no cycle is positive. The same combinatorics of walks, paths and cycles is behind milestones 1 and 2 and the distance-order item.

Formalization scope

Nodes are Fin (n + 2): 000 is the project beginning and Fin.last (n + 1) the project completion. The field one_le_n records n≥1n\ge 1n≥1. Durations are natural numbers with p0=pn+1=0p_0=p_{n+1}=0p0​=pn+1​=0 and pi>0p_i>0pi​>0 for real activities. Arc weights are arbitrary integers on a loop-free Finset of ordered pairs, so parallel arcs cannot occur. Start times are real; integrality appears only as Remark 1.3.2.

Walks are functions Fin (m + 1) → Fin (n + 2). A path is an injective walk, and a cycle is a closed walk with at least one arc and distinct nodes apart from the repeated endpoint. Distances are maxima over the finitely many paths, valued in WithBot ℤ with ⊥=−∞\bot=-\infty⊥=−∞ for unreachable pairs. No supremum over an unbounded set is taken. ESiES_iESi​ and LSiLS_iLSi​ convert these to integers with junk value 000 for −∞-\infty−∞, and every milestone that uses them carries the hypotheses under which the distances are finite. The backward arc of N+N^+N+ has weight −L-L−L for an integer parameter LLL. If NNN already contains an arc ⟨n+1,0⟩\langle n+1,0\rangle⟨n+1,0⟩, the two arcs merge into one carrying the larger weight, as in the book's convention for parallel lags. The standing assumption of p. 8 is a named predicate and a hypothesis of the goal and of milestones 2 and 4.

A trivializing formalization is ruled out: weights are signed integers and cycles are allowed, so the no-positive-cycle condition is not vacuous, and the standing assumption is satisfiable by projects with maximum time lags.

A complete development needs cycle removal from closed walks, the Bellman-type characterization of longest paths without positive cycles, and total unimodularity or a direct integrality argument for Remark 1.3.2. The walk, path and distance layer is reusable for any difference-constraint system. Proofs of milestones and lemmas on walk decomposition are welcome.

Selected references

  • K. Neumann, C. Schwindt, J. Zimmermann, Project Scheduling with Time Windows and Scarce Resources, 2nd ed., Springer, 2003. https://doi.org/10.1007/978-3-540-24800-2
  • M. Bartusch, R. H. Möhring, F. J. Radermacher, "Scheduling project networks with resource constraints and time windows", Annals of Operations Research 16 (1988), 201–240. https://doi.org/10.1007/BF02283745
  • K. Neumann, Operations Research Verfahren, Band III, Hanser, 1975, Sect. 6.4.
  • B. Roy, Les problèmes d'ordonnancement: applications et méthodes, Dunod, 1964.
  • J. E. Kelley, M. R. Walker, "Critical-path planning and scheduling", Proceedings of the Eastern Joint Computer Conference, 1959, 160–173. https://doi.org/10.1145/1460299.1460318
  • R. K. Ahuja, T. L. Magnanti, J. B. Orlin, Network Flows, Prentice Hall, 1993, Sect. 5.4 and 5.6.
8 thms3 active usersReviewed
🏆Completed
Convex OptimizationLinear OptimizationOptimization·Captain: mikedeng1

Understanding and Using Linear Programming V: The Primal–Dual Central Path and the Self-Dual EmbeddingTextbook

Motivation

Interior point methods solve linear programs in a number of iterations polynomial in the input size, and in practice they compete with the simplex method on large instances. Their modern form goes back to Karmarkar's projective algorithm (Karmarkar 1984); the primal–dual path-following variant analysed in textbooks follows a curve, the central path, defined by a perturbed system of optimality conditions. Two facts make the method well defined. First, the central path exists and is unique whenever the primal and dual programs have strictly feasible points. Second, an arbitrary linear program, possibly infeasible or unbounded, can be embedded in an auxiliary program that has an explicit starting point on its own central path and whose suitable optimal solutions either solve the original program or certify that it has no optimum.

Chapter 7, §7.2 of Matoušek and Gärtner, Understanding and Using Linear Programming (Springer 2007), presents both facts in elementary form, following Terlaky (2001). This mission formalizes its three numbered lemmas.

Timeline. The homogeneous system bearing their names is due to Goldman and Tucker (1956), in the study of the structure of optimal solution sets; the self-dual embedding for interior point methods was introduced by Ye, Todd and Mizuno (1994), and the book's presentation follows the skew-symmetric form in Roos, Terlaky and Vial (2005).

Setting

Let AAA be a real m×nm\times nm×n matrix, b∈Rmb\in\mathbb{R}^mb∈Rm, c∈Rnc\in\mathbb{R}^nc∈Rn. The linear program in equational form (7.2) is

maximize cTx subject to Ax=b, x≥0,\text{maximize } c^{T}x \text{ subject to } Ax=b,\ x\ge 0,maximize cTx subject to Ax=b, x≥0,

where AAA has rank mmm; its dual (7.5) is: minimize bTyb^{T}ybTy subject to ATy≥cA^{T}y\ge cATy≥c, y∈Rmy\in\mathbb{R}^my∈Rm. The notation x>0x>0x>0 means that all coordinates of xxx are strictly positive. For μ>0\mu>0μ>0 the barrier function is fμ(x)=cTx+μ∑j=1nln⁡xjf_\mu(x)=c^{T}x+\mu\sum_{j=1}^n\ln x_jfμ​(x)=cTx+μ∑j=1n​lnxj​, defined for x>0x>0x>0. The central-path system (7.4), in unknowns x,s∈Rnx,s\in\mathbb{R}^nx,s∈Rn and y∈Rmy\in\mathbb{R}^my∈Rm, is

Ax=b,ATy−s=c,(s1x1,…,snxn)=μ1,x,s≥0.Ax=b,\qquad A^{T}y-s=c,\qquad (s_1x_1,\dots,s_nx_n)=\mu\mathbf 1,\qquad x,s\ge 0 .Ax=b,ATy−s=c,(s1​x1​,…,sn​xn​)=μ1,x,s≥0.

For the embedding, the book switches to the inequality form (7.7): maximize cTxc^{T}xcTx subject to Ax≤bAx\le bAx≤b, x≥0x\ge 0x≥0, with dual: minimize bTyb^{T}ybTy subject to ATy≥cA^{T}y\ge cATy≥c, y≥0y\ge 0y≥0. The Goldman–Tucker system (GTS) is

Ax−τb≤0,−ATy+τc≤0,bTy−cTx≤0,x,y≥0, τ≥0,Ax-\tau b\le 0,\qquad -A^{T}y+\tau c\le 0,\qquad b^{T}y-c^{T}x\le 0,\qquad x,y\ge 0,\ \tau\ge 0,Ax−τb≤0,−ATy+τc≤0,bTy−cTx≤0,x,y≥0, τ≥0,

and ρ=ρ(x,y)=cTx−bTy\rho=\rho(x,y)=c^{T}x-b^{T}yρ=ρ(x,y)=cTx−bTy is the slack of its last inequality. With u=(y,x,τ)∈Rku=(y,x,\tau)\in\mathbb{R}^ku=(y,x,τ)∈Rk, k=n+m+1k=n+m+1k=n+m+1, (GTS) reads M0u≤0M_0u\le 0M0​u≤0, u≥0u\ge 0u≥0 for the skew-symmetric matrix

M0=(0A−b−AT0cbT−cT0).M_0=\begin{pmatrix}0&A&-b\\-A^{T}&0&c\\b^{T}&-c^{T}&0\end{pmatrix}.M0​=​0−ATbT​A0−cT​−bc0​​.

Put r=1+M01r=\mathbf 1+M_0\mathbf 1r=1+M0​1, M=(M0−rrT0)M=\begin{pmatrix}M_0&-r\\r^{T}&0\end{pmatrix}M=(M0​rT​−r0​) and q=(0,…,0,k+1)∈Rk+1q=(0,\dots,0,k+1)\in\mathbb{R}^{k+1}q=(0,…,0,k+1)∈Rk+1. The self-dual program (SD) in v=(u,ϑ)v=(u,\vartheta)v=(u,ϑ) is: maximize −qTv-q^{T}v−qTv subject to Mv≤qMv\le qMv≤q, v≥0v\ge 0v≥0. Its slacks are z=q−Mvz=q-Mvz=q−Mv, and a feasible vvv is strictly complementary if vj>0v_j>0vj​>0 or zj>0z_j>0zj​>0 for every j=1,…,k+1j=1,\dots,k+1j=1,…,k+1.

Formalization targets

Goal: Lemma 7.2.1 (p. 121)

If (7.2) has a feasible x~>0\tilde x>0x~>0 and (7.5) has a feasible y~\tilde yy~​ with s~=ATy~−c>0\tilde s=A^{T}\tilde y-c>0s~=ATy~​−c>0, then for every μ>0\mu>0μ>0

∃! (x∗,y∗,s∗) solving (7.4),x∗=arg⁡max⁡{fμ(x):Ax=b, x>0} (uniquely).\exists!\,(x^*,y^*,s^*)\ \text{solving (7.4)},\qquad x^*=\arg\max\{f_\mu(x): Ax=b,\ x>0\}\ \text{(uniquely)}.∃!(x∗,y∗,s∗) solving (7.4),x∗=argmax{fμ​(x):Ax=b, x>0} (uniquely).

Milestones

  1. Claim in the proof of Lemma 7.2.1 (p. 121): under the lemma's assumptions and for fixed μ>0\mu>0μ>0, the set Q={x:Ax=b, x>0, fμ(x)≥fμ(x~)}Q=\{x: Ax=b,\ x>0,\ f_\mu(x)\ge f_\mu(\tilde x)\}Q={x:Ax=b, x>0, fμ​(x)≥fμ​(x~)} is bounded.
  2. Lemma 7.2.2 (p. 126): no solution of (GTS) has τ≠0\tau\ne 0τ=0 and ρ≠0\rho\ne 0ρ=0; exactly one of "a solution with τ>0\tau>0τ>0" and "a solution with ρ>0\rho>0ρ>0" exists; in the first case 1τx\frac1\tau xτ1​x and 1τy\frac1\tau yτ1​y are optimal for (7.7) and its dual; in the second, (7.7) is infeasible or unbounded.
  3. Lemma 7.2.3 (p. 128): (SD) is feasible and bounded, every optimal solution has ϑ=0\vartheta=0ϑ=0 and its uuu-part solves (GTS), and every strictly complementary optimal solution gives a solution of (GTS) with τ>0\tau>0τ>0 or ρ>0\rho>0ρ>0.

Significance

Lemma 7.2.1 is what makes "the central path" a well-defined object: without existence, a path-following method has nothing to follow, and without uniqueness the point x∗(μ)x^*(\mu)x∗(μ) that the algorithm approximates is not determined. It also identifies the barrier maximizer with the solution of the Lagrange system (7.4), which is the system the Newton steps of the algorithm linearize. Lemmas 7.2.2 and 7.2.3 remove the need for an interior starting point and for knowing in advance that the program has an optimum: every linear program reduces to computing a strictly complementary optimal solution of a program with a known interior point on its central path.

The three lemmas are classical and proved in the book (7.2.2 as a sketch). Formalizing them adds a machine-checked account of the central path in equational form, and of the Goldman–Tucker and self-dual constructions with explicit block matrices, reusable by any later formalization of interior point complexity bounds. As far as a search of the platform shows, no formal statement of the Goldman–Tucker system or of the self-dual embedding exists there; the nearest item, Lemma 9.5 of Introduction to Linear Optimization XII (Bertsimas–Tsitsiklis), characterizes the central path by KKT conditions and is not linked to a formal statement.

Difficulty

For Lemma 7.2.1, uniqueness of a maximizer follows from strict concavity, but existence does not: the feasible region {Ax=b, x>0}\{Ax=b,\ x>0\}{Ax=b, x>0} is open relative to its affine hull and typically unbounded, and fμf_\mufμ​ need not attain its supremum on such a set. The interior dual point is what rules out escape to infinity; the primal interior point is what rules out escape to the boundary. Identifying the maximizer with the unique solution of (7.4) further needs the Lagrange multiplier rule on an open set and the full row rank of AAA to determine yyy from sss.

For Lemma 7.2.2 the exclusivity and the optimality statements are weak-duality arguments, but existence of a solution with ρ>0\rho>0ρ>0 when (7.7) is infeasible or unbounded requires a Farkas-type alternative for both the primal and the dual, and the case split in the book's sketch ("the dual case is analogous") must be carried out. Lemma 7.2.3 requires bookkeeping with the block structure of MMM and the skew-symmetry of M0M_0M0​.

Formalization scope

Vectors are functions Fin n → ℝ and Fin m → ℝ; the book's indices 1,…,n1,\dots,n1,…,n become 0,…,n−10,\dots,n-10,…,n−1. Matrices are Matrix (Fin m) (Fin n) ℝ, inequalities between vectors are componentwise, x>0x>0x>0 is ∀ j, 0 < x j. The rank condition of (7.2) is the hypothesis A.rank = m. Optimal solutions and unboundedness are expressed against every feasible point, never through a real supremum. The vector u=(y,x,τ)u=(y,x,\tau)u=(y,x,τ) is indexed by Fin m ⊕ Fin n ⊕ Unit and v=(u,ϑ)v=(u,\vartheta)v=(u,ϑ) by (Fin m ⊕ Fin n ⊕ Unit) ⊕ Unit; M0M_0M0​, rrr, MMM and qqq are defined entrywise on these index types, with the last entry of qqq equal to k+1=n+m+2k+1=n+m+2k+1=n+m+2. "Exactly one" in Lemma 7.2.2 is Xor.

Lean's Real.log returns 000 for nonpositive arguments, so a statement comparing fμf_\mufμ​ over {Ax=b}\{Ax=b\}{Ax=b} or {Ax=b, x≥0}\{Ax=b,\ x\ge 0\}{Ax=b, x≥0} would be a different, and generally false or trivial, claim; every comparison of barrier values is restricted to points with all coordinates strictly positive. The unique solution of (7.4) is stated as existence plus equality of every solution with it, not as existence of some solution.

Useful infrastructure: strict concavity of sums of logarithms, the Lagrange multiplier rule for affine constraints (Mathlib's IsLocalExtrOn.exists_multipliers_of_hasStrictFDerivAt or a direct orthogonality argument), compactness of closed bounded sets in Fin n → ℝ, and Farkas' lemma in the forms of Proposition 6.4.1 of the book. Proofs of the milestones, alternative arguments, and general lemmas about skew-symmetric linear programs are all welcome.

Selected references

  • J. Matoušek, B. Gärtner, Understanding and Using Linear Programming, Springer Universitext, 2007, §7.2. https://doi.org/10.1007/978-3-540-30717-4
  • T. Terlaky, An easy way to teach interior-point methods, European Journal of Operational Research 130(1), 2001, 1–19.
  • C. Roos, T. Terlaky, J.-P. Vial, Interior Point Methods for Linear Optimization, 2nd ed., Springer, 2005. https://doi.org/10.1007/b100325
  • A. J. Goldman, A. W. Tucker, Theory of linear programming, in Linear Inequalities and Related Systems, Annals of Mathematics Studies 38, Princeton University Press, 1956, 53–97.
  • Y. Ye, M. J. Todd, S. Mizuno, An O(nL)O(\sqrt{n}L)O(n​L)-iteration homogeneous and self-dual linear programming algorithm, Mathematics of Operations Research 19(1), 1994, 53–67. https://doi.org/10.1287/moor.19.1.53
  • N. Karmarkar, A new polynomial-time algorithm for linear programming, Combinatorica 4, 1984, 373–395. https://doi.org/10.1007/BF02579150
  • F. A. Potra, S. J. Wright, Interior-point methods, Journal of Computational and Applied Mathematics 124, 2000, 281–302.
6 thms3 active usersReviewed
🏆Completed
CombinatoricsGraph TheoryLinear Optimization+1·Captain: mikedeng1

Understanding and Using Linear Programming I: Integral Bipartite Matchings, Total Unimodularity and König's TheoremTextbook

Motivation

Many combinatorial optimization problems are integer programs: linear objectives and linear constraints, with the extra requirement that the variables be integers. Dropping that requirement gives the LP relaxation, which is solvable efficiently but in general only bounds the integer optimum. For a small but important class of problems the relaxation loses nothing: its optimum is attained at an integral point, so linear programming solves the combinatorial problem exactly. Bipartite matching is the standard example, and the job-assignment problem that opens Chapter 3 of Matoušek and Gärtner's Understanding and Using Linear Programming (Springer 2007) is a maximum-weight perfect matching problem in a bipartite graph.

The same phenomenon, combined with linear programming duality, produces combinatorial min–max theorems. The oldest of them is König's theorem (1931) on matchings and vertex covers in bipartite graphs; Hall's marriage theorem (1935) follows from it. This mission formalizes the book's treatment of both strands: the integrality of the bipartite matching LP (Section 3.2), total unimodularity and König's theorem (Section 8.2), and, on the same objects for general graphs, the LP-rounding 2-approximation for vertex cover (Section 3.3).

Setting

Let G=(V,E)G = (V, E)G=(V,E) be a finite simple graph. A bipartition of GGG is a pair of disjoint sets X,YX, YX,Y with X∪Y=VX \cup Y = VX∪Y=V such that every edge joins a vertex of XXX to a vertex of YYY; GGG is bipartite if it has one. A matching is a set M⊆EM \subseteq EM⊆E in which each vertex is incident to at most one edge; a vertex cover is a set C⊆VC \subseteq VC⊆V containing at least one end-vertex of every edge. A matching is maximum if no matching has more edges; a vertex cover is minimum if no vertex cover has fewer vertices.

The incidence matrix of GGG has a row for each vertex and a column for each edge, with entry 111 when the vertex lies on the edge and 000 otherwise. A real matrix is totally unimodular if every square submatrix, obtained by deleting some rows and some columns, has determinant 000, 111 or −1-1−1.

Given real edge weights wew_ewe​, the integer program (3.1) maximizes ∑ewexe\sum_{e} w_e x_e∑e​we​xe​ subject to ∑e∋vxe=1\sum_{e \ni v} x_e = 1∑e∋v​xe​=1 for every vertex vvv and xe∈{0,1}x_e \in \{0,1\}xe​∈{0,1}; its 0/1 solutions are the perfect matchings. Its LP relaxation replaces xe∈{0,1}x_e \in \{0,1\}xe​∈{0,1} by 0≤xe≤10 \le x_e \le 10≤xe​≤1. The vertex-cover relaxation (3.3) minimizes ∑vxv\sum_v x_v∑v​xv​ subject to xu+xv≥1x_u + x_v \ge 1xu​+xv​≥1 for every edge {u,v}\{u,v\}{u,v} and 0≤xv≤10 \le x_v \le 10≤xv​≤1.

Formalization targets

Goal: König's theorem (Theorem 8.2.2)

For every finite bipartite graph GGG,

max⁡{∣M∣:M a matching of G}  =  min⁡{∣C∣:C a vertex cover of G}.\max\{|M| : M \text{ a matching of } G\} \;=\; \min\{|C| : C \text{ a vertex cover of } G\}.max{∣M∣:M a matching of G}=min{∣C∣:C a vertex cover of G}.

Total unimodularity (Lemmas 8.2.3–8.2.5)

A TU  ⇒  (A∣ei) TU;A TU, b∈Zm, max⁡{cTx:Ax≤b, x≥0} attained  ⇒  attained at some x∗∈Zn;A \text{ TU} \;\Rightarrow\; (A \mid e_i) \text{ TU}; \qquad A \text{ TU},\ b \in \mathbb{Z}^m,\ \max\{c^Tx : Ax \le b,\ x \ge 0\} \text{ attained} \;\Rightarrow\; \text{attained at some } x^* \in \mathbb{Z}^n;A TU⇒(A∣ei​) TU;A TU, b∈Zm, max{cTx:Ax≤b, x≥0} attained⇒attained at some x∗∈Zn;

and the incidence matrix of a bipartite graph is totally unimodular.

Integrality of the perfect-matching relaxation (Theorem 3.2.1)

If the LP relaxation of (3.1) for a bipartite graph with real weights is feasible, it has an optimal solution with all xe∈{0,1}x_e \in \{0,1\}xe​∈{0,1}, which is also optimal for (3.1).

Consequences on the same objects

Hall's theorem (Theorem 8.2.1): if ∣N(T)∣≥∣T∣|N(T)| \ge |T|∣N(T)∣≥∣T∣ for every T⊆XT \subseteq XT⊆X, where N(T)⊆YN(T) \subseteq YN(T)⊆Y is the set of neighbours of TTT, then some matching covers every vertex of XXX. And for an arbitrary graph, with x∗x^*x∗ optimal for (3.3), SLP={v:xv∗≥12}S_{LP} = \{v : x^*_v \ge \tfrac12\}SLP​={v:xv∗​≥21​} and SOPTS_{OPT}SOPT​ a minimum vertex cover (§3.3, p. 38):

SLP is a vertex cover and ∣SLP∣≤2 ∣SOPT∣.S_{LP} \text{ is a vertex cover and } |S_{LP}| \le 2\,|S_{OPT}| .SLP​ is a vertex cover and ∣SLP​∣≤2∣SOPT​∣.

Significance

König's theorem says that for bipartite graphs the two natural certificates, a matching (a lower bound on any vertex cover) and a vertex cover (an upper bound on any matching), always meet. It makes maximum matchings and minimum vertex covers computable by linear programming, whereas minimum vertex cover in general graphs is NP-hard; Section 3.3's rounding bound quantifies what the LP still gives in that general case. Lemma 8.2.4 is the general tool behind this and behind the max-flow min-cut theorem that the book mentions on p. 148: any integer program with a totally unimodular constraint matrix and integral right-hand side can be solved as a linear program.

All results here are classical and proved. The formalization work is to connect them: Mathlib already has the definition of total unimodularity (Matrix.IsTotallyUnimodular), closure under appending unit-like rows, and Hall's theorem in the form of Finset.all_card_le_biUnion_card_iff_exists_injective. To the best of the drafting survey, neither König's theorem nor the total unimodularity of bipartite incidence matrices nor the integrality lemma 8.2.4 is in Mathlib, and no König statement was found among the platform's missions. The mission produces these in a form that later chapters on network flows and combinatorial duality can import.

Difficulty

The inequality "maximum matching ≤\le≤ minimum vertex cover" is immediate, since each edge of a matching needs its own cover vertex. The difficulty is the reverse inequality, and it is exactly where bipartiteness is needed: the triangle has maximum matching 111 and minimum vertex cover 222. Along the book's route, the obstacle is that LP duality equates the optima of the two relaxations, which are real numbers; one must show that both relaxations already have integral optimal solutions, which is the content of total unimodularity and Lemma 8.2.4. Theorem 3.2.1 is a separate integrality statement with equality constraints and weights of arbitrary sign; it is not a consequence of the Birkhoff–von Neumann theorem unless the graph is complete bipartite with equal sides.

Formalization scope

Graphs are SimpleGraph V on a Fintype vertex type with decidable equality; edges are elements of Sym2 V, and a matching is a Finset (Sym2 V) of edges. Bipartiteness is the existence of finite sets X,YX, YX,Y forming a bipartition; the empty graph and the empty vertex type are allowed and the statements remain the book's there. Vertex covers are Mathlib's SimpleGraph.IsVertexCover. Matrices are real; LP vectors are Fin n → ℝ (0-based indices) or indexed by the edge set or the vertices. An "optimal solution" is always a feasible point that is at least as good as every feasible point: no supremum or infimum over a possibly empty or unbounded set is used, and König's theorem asserts that both a maximum matching and a minimum vertex cover exist and have equal size. Lemma 8.2.4 takes b∈Zmb \in \mathbb{Z}^mb∈Zm and allows real ccc; its conclusion is an integral optimal solution, not merely an integral feasible one. Theorem 3.2.1 is the perfect-matching version with equality constraints, not the "≤1\le 1≤1" matching version discussed in the book's remarks.

A formalization in which König's theorem compares a supremum and an infimum of possibly empty sets, or in which "optimal" is not tied to feasibility, would be trivializing and is ruled out by these conventions.

Useful infrastructure, reusable beyond this mission: Laplace expansion arguments for totally unimodular matrices, the equivalence of the inequality form with the equational form, existence of optimal basic feasible solutions, and LP duality in inequality form (the platform's LinearOptimization.lp_strong_duality states duality for a general-form LP). Combinatorial proofs of König and Hall are equally welcome; only the statements are fixed.

Selected references

  • J. Matoušek, B. Gärtner, Understanding and Using Linear Programming, Springer Universitext, 2007, §3.2–3.3 and §8.2. https://doi.org/10.1007/978-3-540-30717-4
  • D. Kőnig, "Gráfok és mátrixok", Matematikai és Fizikai Lapok 38 (1931), 116–119.
  • P. Hall, "On representatives of subsets", Journal of the London Mathematical Society 10 (1935), 26–30. https://doi.org/10.1112/jlms/s1-10.37.26
  • A. J. Hoffman, J. B. Kruskal, "Integral boundary points of convex polyhedra", in Linear Inequalities and Related Systems, Annals of Mathematics Studies 38, Princeton, 1956, 223–246. https://doi.org/10.1515/9781400881987-014
  • A. Schrijver, Theory of Linear and Integer Programming, Wiley, 1986, Chapter 19.
14 thms3 active usersReviewed
🏆Completed
Markov ChainProbabilityStochastic Systems·Captain: mikedeng1

Fundamentals of Queueing Theory VII: The Geometric Arrival-Point Law of the G/M/1 QueueTextbook

Motivation

Most queueing models with a closed-form answer assume Poisson arrivals. In practice the times between arrivals are often far from exponential: scheduled appointments, batch releases from an upstream process, or arrivals timed by a machine cycle. The G/M/1 queue keeps the service side exponential and makes no assumption about the arrival stream beyond independent, identically distributed interarrival times. It is the standard counterpart of the M/G/1 queue, and its solution is the one used in teaching and in practice whenever the input is not Poisson (Gross, Shortle, Thompson & Harris, Fundamentals of Queueing Theory, 4th ed., Wiley 2008, §5.3.1, DOI 10.1002/9781118625651).

The answer has an unusually clean form. The number of customers that an arriving customer finds in the system is geometric, exactly as in the M/M/1 queue, with the traffic intensity ρ\rhoρ replaced by a number r0r_0r0​ that depends on the whole interarrival distribution through a single scalar equation. This mission is the seventh of a series formalizing the book chapter by chapter; it covers the G/M/1 half of §5.3 (printed pp.259–263).

Setting

Customers arrive at a single server. The interarrival times are independent with common law AAA, a probability distribution on [0,∞)[0,\infty)[0,∞) with CDF A(t)A(t)A(t) and finite mean E[T]=1/λE[T] = 1/\lambdaE[T]=1/λ, λ>0\lambda > 0λ>0. Service times are independent exponential random variables with rate μ>0\mu > 0μ>0, and the discipline is first come, first served.

Let XnX_nXn​ be the number of customers in the system just before the nnnth arrival. Between two arrivals the server completes a Poisson number of services (truncated by the number present), so {Xn}\{X_n\}{Xn​} is a Markov chain on {0,1,2,… }\{0,1,2,\dots\}{0,1,2,…}. Its transition probabilities are built from

bk=∫0∞e−μt(μt)kk! dA(t)(k≥0),b_k = \int_0^\infty \frac{e^{-\mu t}(\mu t)^k}{k!}\,dA(t) \qquad (k \ge 0),bk​=∫0∞​k!e−μt(μt)k​dA(t)(k≥0),

the probability of exactly kkk completions during one interarrival time (Eq. (5.50)): pi0=1−∑k=0ibkp_{i0} = 1 - \sum_{k=0}^{i} b_kpi0​=1−∑k=0i​bk​, pij=bi+1−jp_{ij} = b_{i+1-j}pij​=bi+1−j​ for 1≤j≤i+11 \le j \le i+11≤j≤i+1, and pij=0p_{ij} = 0pij​=0 otherwise (Eq. (5.51)). A stationary arrival-point distribution is a probability vector q={qn}q = \{q_n\}q={qn​} with qP=qqP = qqP=q and qe=1qe = 1qe=1 (Eq. (5.52)); qnq_nqn​ is the long-run probability that an arrival finds nnn customers present.

The characteristic equation of the chain is

z=β(z),β(z)=∑n≥0bnzn,z = \beta(z), \qquad \beta(z) = \sum_{n \ge 0} b_n z^n ,z=β(z),β(z)=n≥0∑​bn​zn,

where β\betaβ is the probability generating function of {bn}\{b_n\}{bn​} (Eq. (5.55)). Equivalently z=A∗[μ(1−z)]z = A^*[\mu(1-z)]z=A∗[μ(1−z)] (Eq. (5.56)), where A∗(s)=∫0∞e−sx dA(x)A^*(s) = \int_0^\infty e^{-sx}\,dA(x)A∗(s)=∫0∞​e−sxdA(x) is the Laplace–Stieltjes transform of the interarrival law. The traffic intensity is ρ=λ/μ\rho = \lambda/\muρ=λ/μ.

Formalization targets

Goal: Eq. (5.60), the geometric arrival-point law

If ρ=λ/μ<1\rho = \lambda/\mu < 1ρ=λ/μ<1, there is a number r0r_0r0​ with 0<r0<10 < r_0 < 10<r0​<1 and r0=β(r0)r_0 = \beta(r_0)r0​=β(r0​), it is the only complex root of z=β(z)z = \beta(z)z=β(z) in the open unit disk, and

qn=(1−r0) r0 n(n≥0)q_n = (1 - r_0)\, r_0^{\,n} \qquad (n \ge 0)qn​=(1−r0​)r0n​(n≥0)

is a stationary arrival-point distribution and the only one. The root is part of the conclusion, not an assumption.

Milestones

  1. Eqs. (5.51)–(5.53): for a probability vector qqq, qP=qqP = qqP=q is equivalent to qi=∑k≥0qi+k−1bkq_i = \sum_{k\ge0} q_{i+k-1}b_kqi​=∑k≥0​qi+k−1​bk​ (i≥1i \ge 1i≥1) and q0=∑j≥0qj(1−∑k=0jbk)q_0 = \sum_{j\ge0} q_j\bigl(1 - \sum_{k=0}^{j} b_k\bigr)q0​=∑j≥0​qj​(1−∑k=0j​bk​).
  2. p.261: 0<b0<10 < b_0 < 10<b0​<1, bn>0b_n > 0bn​>0 for all nnn, β(1)=1\beta(1) = 1β(1)=1, and β′(1)=∑nnbn=μ/λ\beta'(1) = \sum_n n b_n = \mu/\lambdaβ′(1)=∑n​nbn​=μ/λ.
  3. Eq. (5.56): β(z)=A∗[μ(1−z)]\beta(z) = A^*[\mu(1-z)]β(z)=A∗[μ(1−z)] for ∣z∣≤1|z| \le 1∣z∣≤1.
  4. Eq. (5.58), Figure 5.2: z=β(z)z = \beta(z)z=β(z) has at most one root in (0,1)(0,1)(0,1), and one exists if and only if λ/μ<1\lambda/\mu < 1λ/μ<1.
  5. p.262: when λ/μ<1\lambda/\mu < 1λ/μ<1, z=β(z)z = \beta(z)z=β(z) has exactly one root with ∣z∣<1|z| < 1∣z∣<1.
  6. Eq. (5.59): successive substitution z(k+1)=β(z(k))z^{(k+1)} = \beta(z^{(k)})z(k+1)=β(z(k)) from any 0<z(0)<10 < z^{(0)} < 10<z(0)<1 converges to r0r_0r0​.
  7. Eq. (5.61): L(A)=r0/(1−r0)L^{(A)} = r_0/(1-r_0)L(A)=r0​/(1−r0​) and Lq(A)=r02/(1−r0)L_q^{(A)} = r_0^2/(1-r_0)Lq(A)​=r02​/(1−r0​).
  8. Eq. (5.62): Wq(t)=1−r0e−μ(1−r0)tW_q(t) = 1 - r_0 e^{-\mu(1-r_0)t}Wq​(t)=1−r0​e−μ(1−r0​)t and W(t)=1−e−μ(1−r0)tW(t) = 1 - e^{-\mu(1-r_0)t}W(t)=1−e−μ(1−r0​)t for t≥0t \ge 0t≥0.
  9. Eq. (5.63): Wq=r0/(μ(1−r0))W_q = r_0/(\mu(1-r_0))Wq​=r0​/(μ(1−r0​)) and W=1/(μ(1−r0))W = 1/(\mu(1-r_0))W=1/(μ(1−r0​)).

Significance

The result. Equation (5.60) reduces the analysis of a queue with arbitrary renewal input to one scalar root. Every arrival-point performance measure of the M/M/1 queue then carries over with ρ\rhoρ replaced by r0r_0r0​: the mean number found by an arrival, the mean queue found by an arrival, and the full distributions of line delay and system time seen by arrivals (Eqs. (5.61)–(5.63)). The same root drives the multiserver G/M/c analysis later in §5.3 and the relation between arrival-point and time-average probabilities in §6.3. The result also illustrates a point the book stresses: qnq_nqn​ is the distribution seen by arrivals, and it equals the time-average distribution pnp_npn​ only when the input is Poisson.

Formalizing it. The mathematics is classical (the embedded-chain method goes back to Kendall, 1953) and fully proved in the textbook literature; nothing here is open. To our knowledge none of it has a machine-checked proof: the platform had no G/M/1, embedded-chain, or Rouché-type statement when this mission was drafted. The work is to formalize the known argument, which touches analytic facts about power series with nonnegative coefficients, a mixture-of-Poisson computation, a counting of roots in the unit disk, and the uniqueness of the stationary law of an irreducible countable chain.

Difficulty

Locating a real root in (0,1)(0,1)(0,1) is a one-variable question. The hard step is excluding every other complex root inside the unit disk: a real-variable argument says nothing about complex roots, and the book's route relies on Rouché's theorem, which Mathlib does not have. A second point is uniqueness of the stationary vector: showing that the geometric vector solves qP=qqP = qqP=q does not show that no other probability vector does, and the goal asserts both. Computing ∑nnbn=μ/λ\sum_n n b_n = \mu/\lambda∑n​nbn​=μ/λ requires interchanging a sum with the integral against AAA, which is where the finite mean of the interarrival law enters.

Formalization scope

The interarrival law is a measure A : Measure ℝ with IsProbabilityMeasure A, A (Set.Iio 0) = 0, integrable identity, and ∫ x ∂A = 1/λ (the structure IsInterarrivalLaw). Every theorem also assumes λ>0\lambda > 0λ>0 and μ>0\mu > 0μ>0. The integrals defining bkb_kbk​ and A∗A^*A∗ are over [0,∞)[0,\infty)[0,∞), closed at 000. The generating function β\betaβ takes complex arguments; real roots are written with the real-to-complex coercion. A stationary vector is a function q : ℕ → ℝ with qn≥0q_n \ge 0qn​≥0, HasSum q 1, and HasSum (fun i => q i * p i j) (q j) for every jjj.

The explicit closed forms the statements carry are: the transition matrix (5.51); the equations (5.53); β(z)=A∗[μ(1−z)]\beta(z) = A^*[\mu(1-z)]β(z)=A∗[μ(1−z)] (5.56); β′(1)=μ/λ\beta'(1) = \mu/\lambdaβ′(1)=μ/λ; qn=(1−r0)r0nq_n = (1-r_0)r_0^nqn​=(1−r0​)r0n​ (5.60); r0/(1−r0)r_0/(1-r_0)r0​/(1−r0​) and r02/(1−r0)r_0^2/(1-r_0)r02​/(1−r0​) (5.61); 1−r0e−μ(1−r0)t1 - r_0e^{-\mu(1-r_0)t}1−r0​e−μ(1−r0​)t and 1−e−μ(1−r0)t1 - e^{-\mu(1-r_0)t}1−e−μ(1−r0​)t (5.62); r0/(μ(1−r0))r_0/(\mu(1-r_0))r0​/(μ(1−r0​)) and 1/(μ(1−r0))1/(\mu(1-r_0))1/(μ(1−r0​)) (5.63). The waiting-time CDFs are defined as in §2.2.5 of the book: Wq(t)=q0+∑n≥1qnPr⁡{n completions in≤t}W_q(t) = q_0 + \sum_{n\ge1} q_n \Pr\{n \text{ completions in} \le t\}Wq​(t)=q0​+∑n≥1​qn​Pr{n completions in≤t} with the Erlang type-nnn CDF, and W(t)W(t)W(t) likewise with n+1n+1n+1 completions. The means in (5.63) are ∫0∞[1−Wq(t)] dt\int_0^\infty [1 - W_q(t)]\,dt∫0∞​[1−Wq​(t)]dt and ∫0∞[1−W(t)] dt\int_0^\infty [1 - W(t)]\,dt∫0∞​[1−W(t)]dt.

A trivializing formalization would take "r0∈(0,1)r_0 \in (0,1)r0​∈(0,1) solves z=β(z)z = \beta(z)z=β(z)" as a hypothesis of the goal, which turns (5.60) into a geometric-series check; here existence, location and uniqueness of the root, and uniqueness of the stationary vector, are all conclusions.

Out of scope for this mission: the M/G/c and M/G/∞ results of §5.2 and the multiserver G/M/c analysis of §5.3.2. Reusable pieces include a Rouché-type or fixed-point counting lemma for power series with nonnegative coefficients summing to one, and the uniqueness of stationary laws for irreducible chains on N\mathbb NN. Contributions of either kind are welcome.

Selected references

  • D. Gross, J. F. Shortle, J. M. Thompson, C. M. Harris, Fundamentals of Queueing Theory, 4th ed., Wiley, 2008, §5.3.1, pp.259–263. https://doi.org/10.1002/9781118625651
  • D. G. Kendall, "Stochastic processes occurring in the theory of queues and their analysis by the method of the imbedded Markov chain", Annals of Mathematical Statistics 24(3), 1953, 338–354. https://doi.org/10.1214/aoms/1177728975
12 thms3 active usersReviewed
🏆Completed
Markov ChainProbabilityStochastic Systems·Captain: mikedeng1

Fundamentals of Queueing Theory VI: The Pollaczek–Khintchine Transform for the M/G/1 QueueTextbook

Motivation

The M/G/1 queue is the single-server queue with Poisson arrivals and an arbitrary service-time distribution. It is the first queueing model beyond the birth–death family in which exact formulas survive. It is also the model a practitioner reaches for when service times are measured and visibly not exponential: repair times, transmission times of variable-length packets, machining times. Its central result is the Pollaczek–Khintchine formula, first obtained by Pollaczek (1930) and Khintchine (1932). It expresses the stationary queue in terms of the service distribution, and it shows that the mean wait grows linearly in the squared coefficient of variation of service. That makes variability, and not only load, a measurable driver of congestion.

The textbook treatment followed here is Gross, Shortle, Thompson and Harris, Fundamentals of Queueing Theory, 4th ed. (Wiley 2008), §5.1. It derives the result through Kendall's (1953) imbedded Markov chain of system sizes at departure epochs. It then obtains the transforms of the waiting times and the busy-period functional equation of Takács (1962).

Setting

Customers arrive in a Poisson stream of rate λ>0\lambda > 0λ>0. Service times SSS are independent with distribution BBB, a probability distribution on [0,∞)[0,\infty)[0,∞) with mean E[S]\mathrm E[S]E[S], and the discipline is first-come first-served. The traffic intensity is ρ=λ E[S]\rho = \lambda\,\mathrm E[S]ρ=λE[S].

Let XnX_nXn​ be the number of customers the nnnth departing customer leaves behind. The number of arrivals during one service time equals iii with probability

ki=∫0∞e−λt(λt)ii! dB(t),k_i = \int_0^\infty \frac{e^{-\lambda t}(\lambda t)^i}{i!}\,dB(t),ki​=∫0∞​i!e−λt(λt)i​dB(t),

and (Xn)(X_n)(Xn​) is a Markov chain on {0,1,2,… }\{0,1,2,\dots\}{0,1,2,…} whose transition matrix PPP has first row (k0,k1,k2,… )(k_0,k_1,k_2,\dots)(k0​,k1​,k2​,…) and, for i≥1i \ge 1i≥1, entries pij=kj−i+1p_{ij} = k_{j-i+1}pij​=kj−i+1​ for j≥i−1j \ge i-1j≥i−1 and 000 otherwise. A stationary distribution is a probability vector π\piπ with πP=π\pi P = \piπP=π. Its generating function is Π(z)=∑iπizi\Pi(z) = \sum_i \pi_i z^iΠ(z)=∑i​πi​zi, and that of the arrivals per service is K(z)=∑ikiziK(z) = \sum_i k_i z^iK(z)=∑i​ki​zi, for complex ∣z∣≤1|z| \le 1∣z∣≤1. The Laplace–Stieltjes transform of a distribution FFF on [0,∞)[0,\infty)[0,∞) is F∗(s)=∫0∞e−st dF(t)F^*(s) = \int_0^\infty e^{-st}\,dF(t)F∗(s)=∫0∞​e−stdF(t). In the Lean development these are arrivalProb, transitionMatrix, IsStationaryDist, pgf, utilization and lst in the namespace QueueingFundamentals.MG1.

Formalization targets

Goal: the Pollaczek–Khintchine transform formula (5.15)–(5.16)

If E[S]<∞\mathrm E[S] < \inftyE[S]<∞ and ρ<1\rho < 1ρ<1, the chain has a stationary distribution, and every stationary distribution satisfies π0=1−ρ\pi_0 = 1-\rhoπ0​=1−ρ and

Π(z)=(1−ρ)(1−z)K(z)K(z)−z,∣z∣≤1, z≠1,\Pi(z) = \frac{(1-\rho)(1-z)K(z)}{K(z)-z}, \qquad |z| \le 1,\ z \ne 1,Π(z)=K(z)−z(1−ρ)(1−z)K(z)​,∣z∣≤1, z=1,

with K(z)≠zK(z) \ne zK(z)=z at each such zzz. It leaves the service distribution completely general.

Milestones

  1. The stationary equations (5.12): πi=π0ki+∑j=1i+1πjki−j+1\pi_i = \pi_0 k_i + \sum_{j=1}^{i+1}\pi_j k_{i-j+1}πi​=π0​ki​+∑j=1i+1​πj​ki−j+1​.
  2. The transform (5.14), Π(z)=π0(1−z)K(z)/(K(z)−z)\Pi(z) = \pi_0(1-z)K(z)/(K(z)-z)Π(z)=π0​(1−z)K(z)/(K(z)−z), with π0\pi_0π0​ free and no condition on ρ\rhoρ.
  3. Ergodicity (§5.1.4): a unique stationary distribution exists if and only if ρ<1\rho < 1ρ<1.
  4. The departure-point mean (5.7): L(D)=ρ+(ρ2+λ2σB2)/(2(1−ρ))L^{(D)} = \rho + (\rho^2+\lambda^2\sigma_B^2)/(2(1-\rho))L(D)=ρ+(ρ2+λ2σB2​)/(2(1−ρ)).
  5. K(z)=B∗[λ(1−z)]K(z) = B^*[\lambda(1-z)]K(z)=B∗[λ(1−z)] (5.32).
  6. The system-wait transform (5.29), (5.33): Π(z)=W∗[λ(1−z)]\Pi(z) = W^*[\lambda(1-z)]Π(z)=W∗[λ(1−z)] and W∗(s)=(1−ρ)sB∗(s)/(s−λ[1−B∗(s)])W^*(s) = (1-\rho)sB^*(s)/(s-\lambda[1-B^*(s)])W∗(s)=(1−ρ)sB∗(s)/(s−λ[1−B∗(s)]).
  7. The line-wait transform (5.34): Wq∗(s)=(1−ρ)s/(s−λ[1−B∗(s)])W_q^*(s) = (1-\rho)s/(s-\lambda[1-B^*(s)])Wq∗​(s)=(1−ρ)s/(s−λ[1−B∗(s)]).
  8. The busy-period equation (5.37): G∗(s)=B∗[s+λ−λG∗(s)]G^*(s) = B^*[s+\lambda-\lambda G^*(s)]G∗(s)=B∗[s+λ−λG∗(s)].
  9. The mean busy period: E[X]=1/(μ−λ)\mathrm E[X] = 1/(\mu-\lambda)E[X]=1/(μ−λ) with μ=1/E[S]\mu = 1/\mathrm E[S]μ=1/E[S].

Significance

The transform formula determines the whole stationary departure-point distribution from the service distribution. Its derivatives at z=1z = 1z=1 give every moment of the system size, including the mean-value formula (5.7). Combined with the transform identity (5.32), it gives the waiting-time transforms (5.33)–(5.34). Those in turn give the classical geometric-series representation of the line-wait distribution through the residual service time. The busy-period equation is the starting point for busy-period moments and for the M/G/1 analysis of priority and vacation models later in the book.

All results here are classical and proved in the literature. As far as a search of the platform shows (2026-09-28), none is machine-checked: there is no M/G/1 queue, imbedded departure-point chain, Laplace–Stieltjes transform of a service distribution, or busy-period equation on Prove2Me. Mathlib has Poisson distributions and measure convolution but no generating-function theory for countable Markov chains, no Laplace–Stieltjes transform, and no identity theorem in the form these statements need. The mission produces a checked statement of the Pollaczek–Khintchine formulas that later queueing developments (vacations, priorities, M/G/1-type chains) can build on.

Difficulty

Turning the stationary equations into (5.14) is formal power-series algebra. The difficulties lie elsewhere. First, the formula must hold for complex zzz on the closed disk, which needs the non-vanishing of K(z)−zK(z)-zK(z)−z away from z=1z = 1z=1. That fact fails for ρ>1\rho > 1ρ>1, where KKK has a fixed point inside the disk. Second, (5.15) evaluates π0\pi_0π0​ from Π(1)=1\Pi(1) = 1Π(1)=1 by a limit at the point where the formula is 0/00/00/0, and this uses K′(1)=ρK'(1) = \rhoK′(1)=ρ, an interchange of sum and integral. Third, the existence half of the goal requires positive recurrence of a chain with unbounded jumps. The book obtains it from Foster's criterion, which is not in Mathlib. Fourth, the waiting-time and busy-period transforms are stated for all real s>0s > 0s>0, while the generating-function route reaches only s=λ(1−z)∈(0,2λ]s = \lambda(1-z) \in (0, 2\lambda]s=λ(1−z)∈(0,2λ]. Extending the identity requires either analyticity arguments or a direct derivation. A formal proof of (5.14) alone does not touch any of these.

Formalization scope

The service distribution is a Measure ℝ with IsProbabilityMeasure B and B (Set.Iio 0) = 0; no density is assumed. The arrival rate is lam : ℝ with 0 < lam. Stationarity is IsStationaryDist P π: nonnegative entries, HasSum π 1, and HasSum (fun i => π i * P i j) (π j) for every j. That is global balance on ℕ, as the book writes it. Generating functions take a complex argument with ‖z‖ ≤ 1; transforms take a complex argument, and the waiting-time and busy-period statements use real s. The mean and variance of B are Bochner integrals, and every statement that uses them assumes Integrable. The mean busy period assumes 0 < E[S], so that μ=1/E[S]\mu = 1/\mathrm E[S]μ=1/E[S] is the book's service rate.

Closed forms carried by the statements: π0=1−ρ\pi_0 = 1-\rhoπ0​=1−ρ (5.15); (1−ρ)(1−z)K(z)/(K(z)−z)(1-\rho)(1-z)K(z)/(K(z)-z)(1−ρ)(1−z)K(z)/(K(z)−z) (5.16); π0(1−z)K(z)/(K(z)−z)\pi_0(1-z)K(z)/(K(z)-z)π0​(1−z)K(z)/(K(z)−z) (5.14); ρ+(ρ2+λ2σB2)/(2(1−ρ))\rho + (\rho^2+\lambda^2\sigma_B^2)/(2(1-\rho))ρ+(ρ2+λ2σB2​)/(2(1−ρ)) (5.7); B∗[λ(1−z)]B^*[\lambda(1-z)]B∗[λ(1−z)] (5.32); (1−ρ)sB∗(s)/(s−λ[1−B∗(s)])(1-\rho)sB^*(s)/(s-\lambda[1-B^*(s)])(1−ρ)sB∗(s)/(s−λ[1−B∗(s)]) (5.33); (1−ρ)s/(s−λ[1−B∗(s)])(1-\rho)s/(s-\lambda[1-B^*(s)])(1−ρ)s/(s−λ[1−B∗(s)]) (5.34); B∗[s+λ−λG∗(s)]B^*[s+\lambda-\lambda G^*(s)]B∗[s+λ−λG∗(s)] (5.37); 1/(μ−λ)1/(\mu-\lambda)1/(μ−λ) for the mean busy period.

The waiting-time distribution WWW enters through the book's FCFS relation πn=1n!∫(λt)ne−λt dW(t)\pi_n = \frac1{n!}\int(\lambda t)^n e^{-\lambda t}\,dW(t)πn​=n!1​∫(λt)ne−λtdW(t), and WqW_qWq​ through W=Wq∗BW = W_q * BW=Wq​∗B; both are hypotheses, as in the book. The busy-period distribution GGG enters through the equation (5.36) in CDF form, with nnn-fold convolutions built from Mathlib's Measure.conv.

The goal is not the algebraic consequence of (5.12) for an arbitrary sequence: π\piπ must be a probability vector, π0\pi_0π0​ is determined as 1−ρ1-\rho1−ρ, and the existence of a stationary distribution is part of the conclusion, so the statement cannot hold vacuously. The departure-point/time-average equality (§5.1.3, via PASTA) is not formalized.

Useful infrastructure, reusable beyond this mission: generating functions of stationary distributions on ℕ, Poisson mixtures, Laplace–Stieltjes transforms of measures on [0,∞)[0,\infty)[0,∞), and a Foster-type drift criterion for countable chains. Contributions proving any milestone, or those tools, are welcome.

Selected references

  • D. Gross, J. F. Shortle, J. M. Thompson, C. M. Harris, Fundamentals of Queueing Theory, 4th ed., Wiley, 2008, §5.1. https://doi.org/10.1002/9781118625651
  • D. G. Kendall, Stochastic processes occurring in the theory of queues and their analysis by the method of the imbedded Markov chain, Annals of Mathematical Statistics 24 (1953) 338–354. https://doi.org/10.1214/aoms/1177728975
  • F. G. Foster, On the stochastic matrices associated with certain queuing processes, Annals of Mathematical Statistics 24 (1953) 355–360. https://doi.org/10.1214/aoms/1177728976
  • L. Takács, Introduction to the Theory of Queues, Oxford University Press, 1962.
  • F. Pollaczek, Über eine Aufgabe der Wahrscheinlichkeitstheorie, Mathematische Zeitschrift 32 (1930) 64–100. https://doi.org/10.1007/BF01194620
12 thms3 active usersReviewed
🏆Completed
Markov ChainProbabilityStochastic Systems·Captain: mikedeng1

Fundamentals of Queueing Theory V: Closed Jackson Networks and the Mean-Value RecursionTextbook

Motivation

Networks of queues model systems in which a job visits several service stations in turn: jobs in a computer system alternating between CPU and disks, machines cycling between operation and repair, parts routed through a job shop. In a closed network no job enters or leaves; a fixed population of NNN customers circulates among kkk nodes. Closed networks are the standard model of multiprogrammed computer systems and of machine-repair and finite-source systems, and they are the setting of chapter 4 of Gross, Shortle, Thompson and Harris, Fundamentals of Queueing Theory (4th ed., Wiley 2008, doi:10.1002/9781118625651).

The chapter's results form a short line of computational ideas:

  • Jackson (1957, 1963) showed that open networks of exponential servers with Markovian routing have a product-form steady state; Gordon and Newell (1967) gave the closed-network version, (4.15)–(4.18) of the book.
  • Buzen (1973) gave a convolution recursion for the normalizing constant G(N)G(N)G(N) and for marginal distributions, (4.19)–(4.22).
  • Reiser and Lavenberg (1980) introduced mean-value analysis (MVA), which computes mean queue lengths, waiting times and throughputs population by population without ever forming G(N)G(N)G(N), (4.23)–(4.25); the book presents it following Bruell and Balbo (1980).
  • The book closes the section with a recursion for the full marginal distributions, (4.26), which it proves from the product form (pp.207–209).

This mission formalizes that line, ending at (4.26).

Setting

A closed Jackson network has nodes i=1,…,ki = 1, \dots, ki=1,…,k, each with a single server whose service times are exponential with rate μi>0\mu_i > 0μi​>0. A customer finishing service at node iii moves to node jjj with probability rijr_{ij}rij​; the routing matrix R=(rij)R = (r_{ij})R=(rij​) has nonnegative entries and rows summing to one, and it is irreducible: every node can be reached from every other. The state is nˉ=(n1,…,nk)\bar n = (n_1, \dots, n_k)nˉ=(n1​,…,nk​), the number of customers at each node, with n1+⋯+nk=Nn_1 + \cdots + n_k = Nn1​+⋯+nk​=N; this state space is finite.

The steady-state distribution pnˉp_{\bar n}pnˉ​ is the probability vector on the state space that solves the flow-balance equations (4.14),

∑j=1k∑i=1i≠jkμirij pnˉ;i+j−=∑i=1kμi(1−rii) pnˉ,\sum_{j=1}^{k}\sum_{\substack{i=1\\ i\ne j}}^{k} \mu_i r_{ij}\, p_{\bar n;i^+j^-} = \sum_{i=1}^{k}\mu_i(1-r_{ii})\,p_{\bar n},j=1∑k​i=1i=j​∑k​μi​rij​pnˉ;i+j−​=i=1∑k​μi​(1−rii​)pnˉ​,

where nˉ;i+j−\bar n;i^+j^-nˉ;i+j− has one more customer at iii and one fewer at jjj, and terms with a negative subscript or with μi\mu_iμi​ at an empty node vanish. The traffic equations (4.16) are μiρi=∑jμjrjiρj\mu_i\rho_i = \sum_j \mu_j r_{ji}\rho_jμi​ρi​=∑j​μj​rji​ρj​; they determine ρ=(ρ1,…,ρk)\rho = (\rho_1, \dots, \rho_k)ρ=(ρ1​,…,ρk​) up to a positive factor. The normalizing constant is

G(N)=∑n1+⋯+nk=Nρ1n1⋯ρknk,G(N) = \sum_{n_1+\cdots+n_k=N}\rho_1^{n_1}\cdots\rho_k^{n_k},G(N)=n1​+⋯+nk​=N∑​ρ1n1​​⋯ρknk​​,

and more generally, with fi(n)=ρi n/ai(n)f_i(n) = \rho_i^{\,n}/a_i(n)fi​(n)=ρin​/ai​(n) for cic_ici​-server nodes ((4.13)), G(N)=∑∏ifi(ni)G(N) = \sum \prod_i f_i(n_i)G(N)=∑∏i​fi​(ni​) and Buzen's function gm(n)=∑n1+⋯+nm=n∏i≤mfi(ni)g_m(n) = \sum_{n_1+\cdots+n_m=n}\prod_{i\le m} f_i(n_i)gm​(n)=∑n1​+⋯+nm​=n​∏i≤m​fi​(ni​).

For each population NNN write pi(n,N)=Pr⁡{Ni=n}p_i(n, N) = \Pr\{N_i = n\}pi​(n,N)=Pr{Ni​=n} for the marginal distribution at node iii, Pˉi(n;N)=Pr⁡{Ni≥n}\bar P_i(n; N) = \Pr\{N_i \ge n\}Pˉi​(n;N)=Pr{Ni​≥n}, Li(N)L_i(N)Li​(N) for the mean number at node iii, and

λi(N)=Pr⁡{server busy at node i}⋅μi\lambda_i(N) = \Pr\{\text{server busy at node } i\}\cdot\mu_iλi​(N)=Pr{server busy at node i}⋅μi​

for the throughput of node iii.

Formalization targets

Goal: the marginal recursion (4.26)

For every node iii,

pi(0,0)=1,pi(n,N)=λi(N)μi pi(n−1,N−1)(n,N≥1).p_i(0,0) = 1, \qquad p_i(n, N) = \frac{\lambda_i(N)}{\mu_i}\,p_i(n-1, N-1) \quad (n, N \ge 1).pi​(0,0)=1,pi​(n,N)=μi​λi​(N)​pi​(n−1,N−1)(n,N≥1).

It involves only the steady-state distributions and quantities computed from them; it holds for every irreducible routing matrix and every choice of rates.

Milestones

  1. Product form (4.14)–(4.16). For any positive solution ρ\rhoρ of (4.16), a probability distribution solves (4.14) if and only if pnˉ=G(N)−1ρ1n1⋯ρknkp_{\bar n} = G(N)^{-1}\rho_1^{n_1}\cdots\rho_k^{n_k}pnˉ​=G(N)−1ρ1n1​​⋯ρknk​​.
  2. Buzen's algorithm (4.19)–(4.21). G(N)=gk(N)G(N) = g_k(N)G(N)=gk​(N), gm(n)=∑i=0nfm(i) gm−1(n−i)g_m(n) = \sum_{i=0}^{n} f_m(i)\,g_{m-1}(n-i)gm​(n)=∑i=0n​fm​(i)gm−1​(n−i), g1=f1g_1 = f_1g1​=f1​, gm(0)=1g_m(0) = 1gm​(0)=1.
  3. Marginal at the last node (4.22). pk(n)=fk(n) gk−1(N−n)/G(N)p_k(n) = f_k(n)\,g_{k-1}(N-n)/G(N)pk​(n)=fk​(n)gk−1​(N−n)/G(N) for 0≤n≤N0 \le n \le N0≤n≤N.
  4. Complementary marginal (p.208). Pˉi(ni;N)=ρi niG(N−ni)/G(N)\bar P_i(n_i; N) = \rho_i^{\,n_i}G(N-n_i)/G(N)Pˉi​(ni​;N)=ρini​​G(N−ni​)/G(N).
  5. Mean-value analysis (4.23)–(4.25). Li(0)=0L_i(0) = 0Li​(0)=0; Li(N)=λi(N)Wi(N)L_i(N) = \lambda_i(N)W_i(N)Li​(N)=λi​(N)Wi​(N) with Wi(N)=(1+Li(N−1))/μiW_i(N) = (1 + L_i(N-1))/\mu_iWi​(N)=(1+Li​(N−1))/μi​; and for vvv solving vi=∑jvjrjiv_i = \sum_j v_j r_{ji}vi​=∑j​vj​rji​ with vl=1v_l = 1vl​=1, λl(N)=N/∑iviWi(N)\lambda_l(N) = N/\sum_i v_iW_i(N)λl​(N)=N/∑i​vi​Wi​(N) and λi(N)=λl(N)vi\lambda_i(N) = \lambda_l(N)v_iλi​(N)=λl​(N)vi​.

Significance

The product form reduces a (N+k−1N)\binom{N+k-1}{N}(NN+k−1​)-state Markov chain to the constants G(0),…,G(N)G(0), \dots, G(N)G(0),…,G(N), and Buzen's recursion computes them in O(kN2)O(kN^2)O(kN2) operations. Mean-value analysis goes further and avoids G(N)G(N)G(N), whose magnitude can overflow or underflow for large populations; it is the method used in capacity planning of computer systems. The recursion (4.26) extends MVA from means to full marginal distributions, so a single pass over NNN yields every nodal distribution.

All of these results are classical and proved in the literature; the book proves (4.26) itself. What the mission adds is a machine-checked development of them from the global balance equations: the product form with its uniqueness, the convolution identities, the marginal formulas, and the correctness of the MVA iteration as stated by the book, all over one shared definition layer. A search of the platform on 2026-09-28 found no formal statement of Buzen's algorithm or of MVA. The platform has Kelly's closed migration process theorem (KellyStochasticNetworks.closed_migration_equilibrium), which shows that the unnormalized product form satisfies the equilibrium equations under Kelly's conventions; the normalization, uniqueness and everything downstream of the product form are new here.

Difficulty

The combinatorial identities (Buzen's recursion, the tail marginal) are reindexings of finite sums over compositions of NNN; in Lean the work is in bijections between the state spaces {n1+⋯+nk=N}\{n_1+\cdots+n_k = N\}{n1​+⋯+nk​=N} for different kkk and NNN. The substantive step is uniqueness in the product-form theorem: the global balance equations have a one-dimensional solution space only because the chain on the NNN-customer states is irreducible on the population level, which is a property of the network chain and not of the routing matrix alone. The goal and MVA also need a positive solution of the traffic equations, which is not among the hypotheses and has to come from irreducibility of RRR. The book's own intuitive derivation of MVA via the arrival theorem is not the route the statements require; they are stated in terms of the steady-state distributions alone.

Formalization scope

Nodes are Fin k (book node iii is index i−1i-1i−1); states are n : Fin k → ℕ with ∑ i, n i = N, collected in a Finset, and all sums are finite. A distribution is a real function on Nk\mathbb N^kNk that is nonnegative, vanishes off the NNN-customer states and sums to one there. The balance equations are (4.14) verbatim with the book's boundary convention (p.188), not detailed balance. All results except Buzen's algorithm and (4.22) are for single-server nodes, as in the book; (4.13)'s multiserver factor ai(n)a_i(n)ai​(n) enters only (4.19)–(4.22).

Closed forms instantiated in the statements: the product form G(N)−1∏iρiniG(N)^{-1}\prod_i\rho_i^{n_i}G(N)−1∏i​ρini​​ ((4.15)); G(N)G(N)G(N) as the explicit sum (4.18)/(4.19); ai(n)a_i(n)ai​(n) from (4.13); gmg_mgm​ from (4.20); pk(n)=fk(n)gk−1(N−n)/G(N)p_k(n) = f_k(n)g_{k-1}(N-n)/G(N)pk​(n)=fk​(n)gk−1​(N−n)/G(N) ((4.22)); Pˉi(n;N)=ρinG(N−n)/G(N)\bar P_i(n;N) = \rho_i^nG(N-n)/G(N)Pˉi​(n;N)=ρin​G(N−n)/G(N) (p.208); Wi(N)=(1+Li(N−1))/μiW_i(N) = (1+L_i(N-1))/\mu_iWi​(N)=(1+Li​(N−1))/μi​ ((4.23)); λl(N)=N/∑iviWi(N)\lambda_l(N) = N/\sum_i v_iW_i(N)λl​(N)=N/∑i​vi​Wi​(N) (MVA step (iii)(b)).

Two trivializing formalizations are ruled out: λi(N)\lambda_i(N)λi​(N) in (4.26) and (4.24) is the throughput computed from the steady-state distribution, not a free constant (which would make (4.26) a definition); and gmg_mgm​ is defined by the sum (4.20), so the recursion (4.21) is a theorem rather than rfl. The product-form statement is an equivalence, so it asserts both that the product form is a steady state and that it is the only one.

Needed infrastructure: bijections between compositions of NNN into kkk and k−1k-1k−1 parts, uniqueness of stationary distributions of irreducible finite continuous-time chains (stated directly via the balance equations), and existence of positive solutions of v=vRv = vRv=vR for irreducible stochastic RRR. The last two are reusable beyond this mission. Contributions welcome: proofs of the milestones in any order, and helper lemmas on these three points.

Not formalized: open Jackson networks (4.11) and Burke's theorem (4.5)–(4.6), multiclass networks (§4.2.1), the multiserver recursion (4.27) and cyclic queues (§4.4).

Selected references

  • D. Gross, J. F. Shortle, J. M. Thompson, C. M. Harris, Fundamentals of Queueing Theory, 4th ed., Wiley, 2008, §4.3, pp.195–209. https://doi.org/10.1002/9781118625651
  • J. R. Jackson, "Jobshop-like queueing systems", Management Science 10(1), 1963. https://doi.org/10.1287/mnsc.10.1.131
  • W. J. Gordon, G. F. Newell, "Closed queuing systems with exponential servers", Operations Research 15(2), 1967. https://doi.org/10.1287/opre.15.2.254
  • J. P. Buzen, "Computational algorithms for closed queueing networks with exponential servers", Communications of the ACM 16(9), 1973. https://doi.org/10.1145/362342.362345
  • M. Reiser, S. S. Lavenberg, "Mean-value analysis of closed multichain queuing networks", Journal of the ACM 27(2), 1980. https://doi.org/10.1145/322186.322195
  • S. C. Bruell, G. Balbo, Computational Algorithms for Closed Queueing Networks, North-Holland, 1980.
7 thms3 active usersReviewed
🏆Completed
Markov ChainProbabilityStochastic Systems·Captain: mikedeng1

Fundamentals of Queueing Theory IV: The Stationary Distribution of the M/M/1 Retrial QueueTextbook

Motivation

In many service systems a customer who finds every server busy does not join a queue. A caller who hears a busy signal hangs up and redials later; a request rejected by a saturated server is resent after a timeout; an aircraft that cannot land circles and tries again. These retrial queues are the subject of a substantial literature in telephone traffic engineering, computer networks and call-centre design, surveyed in the monograph of Falin and Templeton (1997) and the bibliography of Artalejo (1999). Their analysis is harder than that of ordinary queues: the blocked customers form an orbit whose size is part of the state, so even the simplest model is a two-dimensional Markov chain, and explicit stationary distributions are rare.

This mission is the fourth of a series formalizing Gross, Shortle, Thompson and Harris, Fundamentals of Queueing Theory (4th ed., Wiley 2008). Its goal is the explicit stationary distribution of the single-server retrial queue, Eq. (3.57) of §3.5.1, one of the few retrial models solvable in closed form. Chapter 3 of the book treats Markovian queues that are not birth–death processes: bulk arrivals, bulk service, Erlang phases, priority disciplines and retrials. The milestones also collect three capstone formulas from the chapter's other sections: the bulk-input queue, the partial-batch bulk-service queue, and Cobham's formula for nonpreemptive priorities (Cobham, 1954).

Setting

In the M/M/1M/M/1M/M/1 retrial queue customers arrive according to a Poisson process with rate λ\lambdaλ and are served one at a time by a single server, with exponential service times of mean 1/μ1/\mu1/μ. An arrival that finds the server busy enters the orbit and stays there for an exponential time with mean 1/γ1/\gamma1/γ, after which it tries again; each customer in orbit retries independently. No customer leaves because of impatience. With Ns(t)∈{0,1}N_s(t) \in \{0,1\}Ns​(t)∈{0,1} the number in service and No(t)N_o(t)No​(t) the number in orbit, the pair is a continuous-time Markov chain on states {i,n}\{i, n\}{i,n}, i∈{0,1}i \in \{0,1\}i∈{0,1}, n∈{0,1,2,… }n \in \{0,1,2,\dots\}n∈{0,1,2,…}. Writing pi,np_{i,n}pi,n​ for the steady-state probability of {i,n}\{i,n\}{i,n}, the rate-balance equations are

(λ+nγ)p0,n=μp1,n,n≥0,(3.47)(λ+μ)p1,n=λp0,n+(n+1)γp0,n+1+λp1,n−1,n≥1,(3.48)(λ+μ)p1,0=λp0,0+γp0,1.(3.49)\begin{aligned} (\lambda + n\gamma)p_{0,n} &= \mu p_{1,n}, && n \ge 0, && (3.47)\\ (\lambda+\mu)p_{1,n} &= \lambda p_{0,n} + (n+1)\gamma p_{0,n+1} + \lambda p_{1,n-1}, && n \ge 1, && (3.48)\\ (\lambda+\mu)p_{1,0} &= \lambda p_{0,0} + \gamma p_{0,1}. && && (3.49) \end{aligned}(λ+nγ)p0,n​(λ+μ)p1,n​(λ+μ)p1,0​​=μp1,n​,=λp0,n​+(n+1)γp0,n+1​+λp1,n−1​,=λp0,0​+γp0,1​.​​n≥0,n≥1,​​(3.47)(3.48)(3.49)​

Following the book's convention (§1.9, and the footnote on p.118), a steady-state solution is a nonnegative solution of these equations whose total mass ∑n(p0,n+p1,n)\sum_n (p_{0,n} + p_{1,n})∑n​(p0,n​+p1,n​) equals 111. The traffic intensity is ρ=λ/μ\rho = \lambda/\muρ=λ/μ, and the partial generating functions are P0(z)=∑nznp0,nP_0(z) = \sum_n z^n p_{0,n}P0​(z)=∑n​znp0,n​ and P1(z)=∑nznp1,nP_1(z) = \sum_n z^n p_{1,n}P1​(z)=∑n​znp1,n​.

The other models of the mission use the same convention. In the bulk-input queue M[X]/M/1M^{[X]}/M/1M[X]/M/1, batches arrive at rate λ\lambdaλ with batch-size probabilities cn=Pr⁡{X=n}c_n = \Pr\{X = n\}cn​=Pr{X=n}, n≥1n \ge 1n≥1, and batch-size generating function C(z)=∑ncnznC(z) = \sum_n c_n z^nC(z)=∑n​cn​zn. In the partial-batch bulk-service queue M/M[K]/1M/M^{[K]}/1M/M[K]/1, single arrivals come at rate λ\lambdaλ and the server serves up to KKK customers together in an exponential time of mean 1/μ1/\mu1/μ. In the nonpreemptive priority queue there are rrr classes with rates λk\lambda_kλk​ and μk\mu_kμk​, loads ρk=λk/μk\rho_k = \lambda_k/\mu_kρk​=λk​/μk​ and cumulative loads σk=ρ1+⋯+ρk\sigma_k = \rho_1 + \cdots + \rho_kσk​=ρ1​+⋯+ρk​.

Formalization targets

Goal: the stationary distribution (3.57)

For λ,μ,γ>0\lambda, \mu, \gamma > 0λ,μ,γ>0 and ρ<1\rho < 1ρ<1, the numbers

p0,n=(1−ρ)(λ/γ)+1ρnn! γn∏i=0n−1(λ+iγ),p1,n=(1−ρ)(λ/γ)+1ρn+1n! γn∏i=1n(λ+iγ)p_{0,n} = (1-\rho)^{(\lambda/\gamma)+1}\frac{\rho^n}{n!\,\gamma^n}\prod_{i=0}^{n-1}(\lambda+i\gamma), \qquad p_{1,n} = (1-\rho)^{(\lambda/\gamma)+1}\frac{\rho^{n+1}}{n!\,\gamma^n}\prod_{i=1}^{n}(\lambda+i\gamma)p0,n​=(1−ρ)(λ/γ)+1n!γnρn​i=0∏n−1​(λ+iγ),p1,n​=(1−ρ)(λ/γ)+1n!γnρn+1​i=1∏n​(λ+iγ)

form a steady-state solution of (3.47)–(3.49), and every steady-state solution equals them.

Milestones on the retrial queue

The generating functions satisfy (3.50)–(3.52) on (−1,1)(-1,1)(−1,1), including the separable equation

P0′(z)=λργ(1−ρz)P0(z),P_0'(z) = \frac{\lambda\rho}{\gamma(1-\rho z)}P_0(z),P0′​(z)=γ(1−ρz)λρ​P0​(z),

their closed form is (3.55),

P0(z)=(1−ρz)(1−ρ1−ρz)(λ/γ)+1,P1(z)=ρ(1−ρ1−ρz)(λ/γ)+1,P_0(z) = (1-\rho z)\left(\frac{1-\rho}{1-\rho z}\right)^{(\lambda/\gamma)+1}, \qquad P_1(z) = \rho\left(\frac{1-\rho}{1-\rho z}\right)^{(\lambda/\gamma)+1},P0​(z)=(1−ρz)(1−ρz1−ρ​)(λ/γ)+1,P1​(z)=ρ(1−ρz1−ρ​)(λ/γ)+1,

and the mean orbit size is (3.58), Lo=ρ21−ρ⋅μ+γγL_o = \frac{\rho^2}{1-\rho}\cdot\frac{\mu+\gamma}{\gamma}Lo​=1−ρρ2​⋅γμ+γ​.

Milestones from the rest of Chapter 3

The bulk-input generating function (3.3), p0=1−ρp_0 = 1 - \rhop0​=1−ρ with ρ=λE[X]/μ\rho = \lambda\mathrm E[X]/\muρ=λE[X]/μ, and the mean (3.4); the unique root r0∈(0,1)r_0 \in (0,1)r0​∈(0,1) of μrK+1−(λ+μ)r+λ=0\mu r^{K+1} - (\lambda+\mu)r + \lambda = 0μrK+1−(λ+μ)r+λ=0 and the geometric law pn=(1−r0)r0np_n = (1-r_0)r_0^npn​=(1−r0​)r0n​ (3.9); and Cobham's formula (3.41)/(3.43), the unique solution of the linear system (3.40).

Significance

The closed form (3.57) makes every performance measure of the M/M/1M/M/1M/M/1 retrial queue explicit. The server is busy a fraction ρ\rhoρ of the time, exactly as without retrials. The mean orbit size (3.58) is the M/M/1M/M/1M/M/1 mean queue length multiplied by (μ+γ)/γ(\mu+\gamma)/\gamma(μ+γ)/γ, and the mean time in orbit (3.59) follows from Little's law. These formulas quantify the cost of retrials against an ordinary queue and are the reference case against which approximations for multi-server retrial systems are checked.

The results are classical and proved in the book, partly through exercises (Problems 3.39–3.41). None of them is formalized in any proof assistant, as far as the platform's catalogue shows: there is no retrial, bulk or priority queue on Prove2Me. The mission produces machine-checked statements and, once solved, proofs of the chapter's main closed forms. It also produces a small reusable layer: generating functions of probability sequences on the closed unit disc, and the "probability solution of the balance equations" pattern for chains with countable state spaces.

Difficulty

The derivation in the book is formal. It differentiates power series term by term, divides by 1−z1 - z1−z, integrates ln⁡P0\ln P_0lnP0​, and fixes the constant by setting z=1z = 1z=1, without justifying any of these steps. A formal proof has to show that the series converge and are differentiable on (−1,1)(-1,1)(−1,1), that the differential equation determines P0P_0P0​ up to a constant, and that the values at z=1z = 1z=1 are the limits of the values inside the disc (Abel's theorem). The uniqueness half of the goal is the hardest part. The book never proves it; it follows from the ODE argument only once every step is shown to hold for an arbitrary probability solution. Verifying that (3.57) solves (3.47)–(3.49) is only the easy half. The same pattern recurs in the bulk-input queue, where z=1z = 1z=1 is a removable singularity of (3.3). In the bulk-service queue the root r0r_0r0​ is only characterized as the unique root in (0,1)(0,1)(0,1), so existence and uniqueness of the root are part of the claim.

Formalization scope

A steady-state solution is a pair p0 p1 : ℕ → ℝ (resp. one sequence p : ℕ → ℝ) that is pointwise nonnegative, has total mass 111 as a HasSum, and solves the book's balance equations exactly as printed, global balance and not detailed balance. Every "the steady-state solution is X" is stated with both halves: X is a steady-state solution, and every steady-state solution equals X. Stating only that (3.57) solves (3.47)–(3.49), without normalization or uniqueness, would be a trivializing formalization. So would taking r0r_0r0​ as a given root in (3.9), or taking the Wq(i)W_q^{(i)}Wq(i)​ in (3.41) as numbers assumed to satisfy it. None of these is used. The closed forms instantiated are (3.52), (3.55), (3.57), (3.58), (3.3), (3.4), (3.9), (3.41) and (3.43), each written out in full, with the real power (1−ρ)(λ/γ)+1(1-\rho)^{(\lambda/\gamma)+1}(1−ρ)(λ/γ)+1 as Real.rpow.

The conventions are as follows. The retrial generating functions take real arguments, on (−1,1)(-1,1)(−1,1) for the differential equations and on [−1,1][-1,1][−1,1] for the closed form. The bulk-input generating function takes complex arguments with ∣z∣≤1|z| \le 1∣z∣≤1, z≠1z \ne 1z=1, because (3.3) is 0/00/00/0 at z=1z = 1z=1. The condition ρ<1\rho < 1ρ<1 is a hypothesis of every retrial statement. For the bulk-service queue the book's unnamed condition is stated as λ<Kμ\lambda < K\muλ<Kμ. For bulk input, E[X]<∞\mathrm E[X] < \inftyE[X]<∞ is assumed throughout, and the mean (3.4) is asserted under the further condition E[X2]<∞\mathrm E[X^2] < \inftyE[X2]<∞, which it requires. For Cobham's formula only the algebraic content is formalized; the mean-value argument that yields (3.40) and (3.42) is not.

Needed infrastructure: power series of summable nonnegative sequences on the closed unit disc (convergence, term-by-term differentiation, Abel continuity), the binomial series (1−x)−a=∑na(a+1)⋯(a+n−1)n!xn(1 - x)^{-a} = \sum_n \frac{a(a+1)\cdots(a+n-1)}{n!}x^n(1−x)−a=∑n​n!a(a+1)⋯(a+n−1)​xn for real aaa, and uniqueness of invariant probability vectors for irreducible chains. All of this is reusable beyond the mission. Proofs of any milestone, of the easy half of the goal, or of the needed series facts are welcome contributions.

Selected references

  • D. Gross, J. F. Shortle, J. M. Thompson, C. M. Harris, Fundamentals of Queueing Theory, 4th ed., Wiley, 2008, §§3.1, 3.2.0.1, 3.4.2, 3.5.1. https://doi.org/10.1002/9781118625651
  • G. I. Falin, J. G. C. Templeton, Retrial Queues, Chapman & Hall, 1997. https://doi.org/10.1007/978-1-4899-2977-8
  • J. R. Artalejo, Accessible bibliography on retrial queues, Mathematical and Computer Modelling 30 (1999) 1–6. https://doi.org/10.1016/S0895-7177(99)00128-4
  • A. Cobham, Priority assignment in waiting line problems, Journal of the Operations Research Society of America 2 (1954) 70–76. https://doi.org/10.1287/opre.2.1.70
10 thms3 active usersReviewed
🏆Completed
AnalysisMarkov ChainProbability+1·Captain: mikedeng1

Fundamentals of Queueing Theory III: The Transient M/M/1 Queue via Modified Bessel FunctionsTextbook

Motivation

Steady-state formulas describe a queue that has been running forever. Many practical questions are about a queue that has not: a call centre just after opening, a server just after a reset, a system under a burst of load. For these, the relevant quantity is the transient distribution pn(t)=Pr⁡{N(t)=n}p_n(t) = \Pr\{N(t) = n\}pn​(t)=Pr{N(t)=n} of the number N(t)N(t)N(t) in the system at a finite time ttt. It is also what determines how fast the steady state is approached, and it is needed for the busy period: the length of time a server stays busy once a customer arrives at an idle server.

For the single-server Markovian queue M/M/1 the transient distribution has an explicit closed form in modified Bessel functions. Its history is short and well documented. Ledermann and Reuter (1954) obtained it by spectral analysis of the birth–death process. Bailey (1954) found it by generating functions and Laplace transforms, and Champernowne (1956) by combinatorial methods. Bailey's route is the standard textbook derivation, and it is the one Gross, Shortle, Thompson and Harris outline in §2.11 of Fundamentals of Queueing Theory (4th ed., 2008). Abate and Whitt (1989) showed that computing with the resulting series is numerically delicate, which is one reason for having the formula pinned down exactly.

This mission formalizes §§2.11–2.12 of that book: the transient laws of M/M/1/1, M/M/1 and M/M/∞, and the M/M/1 busy period.

Setting

Customers arrive in a Poisson stream of rate λ>0\lambda > 0λ>0. Each service takes an exponential time of rate μ>0\mu > 0μ>0, and ρ=λ/μ\rho = \lambda/\muρ=λ/μ. The number in the system is a continuous-time Markov chain on {0,1,2,… }\{0, 1, 2, \dots\}{0,1,2,…}, and its state probabilities pn(t)p_n(t)pn​(t) satisfy the forward (differential–difference) equations. For M/M/1 started with N(0)=iN(0) = iN(0)=i they are, for t≥0t \ge 0t≥0,

pn′(t)=−(λ+μ)pn(t)+λpn−1(t)+μpn+1(t) (n>0),p0′(t)=−λp0(t)+μp1(t),(2.72)p_n'(t) = -(\lambda+\mu)p_n(t) + \lambda p_{n-1}(t) + \mu p_{n+1}(t)\ (n > 0), \qquad p_0'(t) = -\lambda p_0(t) + \mu p_1(t), \tag{2.72}pn′​(t)=−(λ+μ)pn​(t)+λpn−1​(t)+μpn+1​(t) (n>0),p0′​(t)=−λp0​(t)+μp1​(t),(2.72)

with pn(0)=1p_n(0) = 1pn​(0)=1 if n=in = in=i and 000 otherwise. The other systems are variants:

  • M/M/1/1, no waiting room: two states and equations (2.70).
  • M/M/∞, ample service: the death rate in state nnn is nμn\munμ, giving (2.76).
  • The busy-period system: (2.72) with 000 made absorbing (λ0=0\lambda_0 = 0λ0​=0) and N(0)=1N(0) = 1N(0)=1. Its p0(t)p_0(t)p0​(t) is the distribution function of the busy period TbpT_{bp}Tbp​.

A family (pn)(p_n)(pn​) solves a system on [0,∞)[0,\infty)[0,∞) when each pnp_npn​ has, at every t≥0t \ge 0t≥0, the prescribed derivative (a right derivative at t=0t = 0t=0). It is a probability solution when pn(t)≥0p_n(t) \ge 0pn​(t)≥0 and ∑npn(t)=1\sum_n p_n(t) = 1∑n​pn​(t)=1 for every t≥0t \ge 0t≥0. The modified Bessel function of the first kind is

In(y)=∑k=0∞(y/2)n+2kk! (n+k)!,I−n=In,I_n(y) = \sum_{k=0}^{\infty} \frac{(y/2)^{n+2k}}{k!\,(n+k)!}, \qquad I_{-n} = I_n,In​(y)=k=0∑∞​k!(n+k)!(y/2)n+2k​,I−n​=In​,

and the Laplace transform of fff is fˉ(s)=∫0∞e−stf(t) dt\bar f(s) = \int_0^\infty e^{-st} f(t)\,dtfˉ​(s)=∫0∞​e−stf(t)dt for Re⁡s>0\operatorname{Re} s > 0Res>0.

Formalization targets

Goal: the transient M/M/1 law, (2.75)

With y=2tλμy = 2t\sqrt{\lambda\mu}y=2tλμ​,

pn(t)=e−(λ+μ)t[ρ(n−i)/2In−i(y)+ρ(n−i−1)/2In+i+1(y)+(1−ρ)ρn∑j=n+i+2∞ρ−j/2Ij(y)].p_n(t) = e^{-(\lambda+\mu)t}\Big[\rho^{(n-i)/2} I_{n-i}(y) + \rho^{(n-i-1)/2} I_{n+i+1}(y) + (1-\rho)\rho^n \sum_{j=n+i+2}^{\infty} \rho^{-j/2} I_j(y)\Big].pn​(t)=e−(λ+μ)t[ρ(n−i)/2In−i​(y)+ρ(n−i−1)/2In+i+1​(y)+(1−ρ)ρnj=n+i+2∑∞​ρ−j/2Ij​(y)].

The goal asserts five things for every λ,μ>0\lambda, \mu > 0λ,μ>0 and every iii, with no restriction on ρ\rhoρ:

  1. the series converges;
  2. these functions solve (2.72);
  3. they meet the initial condition;
  4. they form a probability distribution at every ttt;
  5. they are the only probability solution.

Milestones

  1. (2.71): the M/M/1/1 solution p1(t)=λλ+μ(1−e−(λ+μ)t)+p1(0)e−(λ+μ)tp_1(t) = \frac{\lambda}{\lambda+\mu}(1-e^{-(\lambda+\mu)t}) + p_1(0)e^{-(\lambda+\mu)t}p1​(t)=λ+μλ​(1−e−(λ+μ)t)+p1​(0)e−(λ+μ)t, and the matching formula for p0p_0p0​.
  2. (2.74) and Rouché's theorem: for Re⁡s>0\operatorname{Re} s > 0Res>0, the quadratic (λ+μ+s)z−μ−λz2(\lambda+\mu+s)z - \mu - \lambda z^2(λ+μ+s)z−μ−λz2 has exactly one zero in ∣z∣<1|z| < 1∣z∣<1, namely z1=(λ+μ+s−(λ+μ+s)2−4λμ)/(2λ)z_1 = (\lambda+\mu+s-\sqrt{(\lambda+\mu+s)^2-4\lambda\mu})/(2\lambda)z1​=(λ+μ+s−(λ+μ+s)2−4λμ​)/(2λ).
  3. The transform of p0p_0p0​: pˉ0(s)=z1i+1/(μ(1−z1))\bar p_0(s) = z_1^{i+1}/(\mu(1-z_1))pˉ​0​(s)=z1i+1​/(μ(1−z1​)).
  4. The limit of (2.75): pn(t)→(1−ρ)ρnp_n(t) \to (1-\rho)\rho^npn​(t)→(1−ρ)ρn if ρ<1\rho < 1ρ<1, and pn(t)→0p_n(t) \to 0pn​(t)→0 if ρ≥1\rho \ge 1ρ≥1.
  5. (2.77), M/M/∞: started empty, pn(t)=a(t)ne−a(t)/n!p_n(t) = a(t)^n e^{-a(t)}/n!pn​(t)=a(t)ne−a(t)/n! with a(t)=(1−e−μt)λ/μa(t) = (1-e^{-\mu t})\lambda/\mua(t)=(1−e−μt)λ/μ. The statement says that this family solves (2.76), is the unique probability solution, and has generating function exp⁡((z−1)a(t))\exp((z-1)a(t))exp((z−1)a(t)).
  6. The busy-period transform: pˉ0(s)=2μ/(s[λ+μ+s+(λ+μ+s)2−4λμ])\bar p_0(s) = 2\mu/(s[\lambda+\mu+s+\sqrt{(\lambda+\mu+s)^2-4\lambda\mu}])pˉ​0​(s)=2μ/(s[λ+μ+s+(λ+μ+s)2−4λμ​]).
  7. The busy-period density: p0′(t)=μ/λ e−(λ+μ)tI1(2λμ t)/tp_0'(t) = \sqrt{\mu/\lambda}\,e^{-(\lambda+\mu)t} I_1(2\sqrt{\lambda\mu}\,t)/tp0′​(t)=μ/λ​e−(λ+μ)tI1​(2λμ​t)/t.
  8. (2.79): for λ<μ\lambda < \muλ<μ, E[Tbp]=1/(μ−λ)E[T_{bp}] = 1/(\mu-\lambda)E[Tbp​]=1/(μ−λ) and E[Tbc]=1/λ+1/(μ−λ)E[T_{bc}] = 1/\lambda + 1/(\mu-\lambda)E[Tbc​]=1/λ+1/(μ−λ).

Significance

The formula (2.75) is the exact finite-time law of the most basic queue. It gives the rate at which M/M/1 approaches equilibrium, and it gives the distribution of the queue under overload (ρ≥1\rho \ge 1ρ≥1), where no steady state exists. It is the reference against which numerical transient methods, such as the uniformization of Chapter 8 of the same book, are checked. The busy-period density and its mean (2.79) enter server-utilisation and vacation models, and the Laplace-transform method used here recurs in the M/G/1 analysis of Chapter 5.

All of these results are classical and proved in the literature. None of them is machine-checked, as far as the platform's catalogue and Mathlib show. The chain from a countable system of linear ODEs, through generating functions and a root-location argument, to a Bessel series is a standard pattern in applied probability, and a formal version of it is what this mission adds. The formal statements also make explicit what the book leaves implicit: the sense in which the equations hold at t=0t = 0t=0, and the class in which the solution is unique.

Difficulty

The forward equations (2.72) form an infinite linear system. The obvious approach is to treat it like a finite system of ODEs, whose solution is a matrix exponential, and read off (2.75). That fails for two reasons. The generator is an infinite matrix, so its exponential needs a functional-analytic setting. And uniqueness is not automatic for infinite systems: it needs a class, such as probability solutions, and an argument that works in that class.

The Bessel form is a second, independent difficulty. The transform pˉ0(s)\bar p_0(s)pˉ​0​(s) is fixed by a root-location argument in the complex plane. Inverting the transform, or verifying (2.75) directly, requires manipulating the three-term Bessel recurrence and exchanging infinite sums. The tail sum ∑jρ−j/2Ij\sum_{j} \rho^{-j/2} I_j∑j​ρ−j/2Ij​ has to be controlled uniformly enough to be differentiated term by term. For ρ≥1\rho \ge 1ρ≥1 the factor (1−ρ)(1-\rho)(1−ρ) is non-positive, so the nonnegativity of pn(t)p_n(t)pn​(t) is not visible from the formula.

Formalization scope

Conventions committed to:

  • Parameters. Rates are real with λ,μ>0\lambda, \mu > 0λ,μ>0, and ρ=λ/μ\rho = \lambda/\muρ=λ/μ. States are ℕ (Fin 2 for M/M/1/1).
  • Solutions. "Solves on [0,∞)[0,\infty)[0,∞)" is HasDerivWithinAt on Set.Ici 0 at every t≥0t \ge 0t≥0. Uniqueness is asserted among solutions that are probability distributions at every time.
  • Special functions. Half-integer powers of ρ\rhoρ are real powers, and I−m=ImI_{-m} = I_mI−m​=Im​ is part of the definition. Laplace transforms are complex Bochner integrals over (0,∞)(0,\infty)(0,∞), and each statement also asserts the integrability it needs. Square roots with positive real part are hypotheses r2=(λ+μ+s)2−4λμr^2 = (\lambda+\mu+s)^2 - 4\lambda\mur2=(λ+μ+s)2−4λμ, Re⁡r>0\operatorname{Re} r > 0Rer>0.

The closed forms stated exactly as in the book are:

  • (2.71);
  • z1z_1z1​ and z2z_2z2​ of (2.74);
  • pˉ0(s)=z1i+1/(μ(1−z1))\bar p_0(s) = z_1^{i+1}/(\mu(1-z_1))pˉ​0​(s)=z1i+1​/(μ(1−z1​));
  • (2.75), with the Bessel series of p.101;
  • the M/M/∞ law and (2.77);
  • the busy-period transform and density of p.102;
  • (2.79).

The book derives (2.79) by a steady-state ratio argument valid for M/G/1. Here it is stated for M/M/1, as the mean of the explicit density.

A statement of (2.75) that only asserts the right-hand side is well defined, or checks only n=0n = 0n=0, is ruled out: the goal requires the ODE system, the initial condition, the probability property and uniqueness. For the same reason, the M/M/∞ law is tied to the system (2.76) and does not reduce to a Taylor expansion.

Needed infrastructure that Mathlib lacks:

  • modified Bessel functions of integer order;
  • Laplace transforms;
  • a Rouché-type zero count or a direct root-location lemma;
  • uniqueness for countable linear ODE systems with bounded or linearly growing rates.

The Bessel and Laplace definitions, and the uniqueness lemma for birth–death forward equations, are reusable beyond this mission. Contributions of those as separate lemmas are welcome.

Selected references

  • D. Gross, J. F. Shortle, J. M. Thompson, C. M. Harris, Fundamentals of Queueing Theory, 4th ed., Wiley, 2008, §§2.11–2.12, pp.97–103. https://doi.org/10.1002/9781118625651
  • N. T. J. Bailey, "A continuous time treatment of a simple queue using generating functions", J. Royal Statistical Society B 16 (1954) 288–291. https://doi.org/10.1111/j.2517-6161.1954.tb00172.x
  • W. Ledermann, G. E. H. Reuter, "Spectral theory for the differential equations of simple birth and death processes", Phil. Trans. Royal Society A 246 (1954) 321–369. https://doi.org/10.1098/rsta.1954.0001
  • D. G. Champernowne, "An elementary method of solution of the queueing problem with a single server and constant parameters", J. Royal Statistical Society B 18 (1956) 125–128. https://doi.org/10.1111/j.2517-6161.1956.tb00217.x
  • J. Abate, W. Whitt, "Calculating time-dependent performance measures for the M/M/1 queue", IEEE Trans. Communications 37 (1989) 1102–1104. https://doi.org/10.1109/26.41165
15 thms3 active usersReviewed
🏆Completed
Markov ChainProbabilityStochastic Systems·Captain: mikedeng1

Fundamentals of Queueing Theory II: Erlang's Formulas and the Halfin–Whitt Square-Root Staffing LawTextbook

Why birth–death queues and Erlang's formulas

Every call center, hospital ward, cloud server pool and telephone exchange that is sized by formula is sized by one of a handful of explicit expressions from Markovian queueing theory. The two oldest are A. K. Erlang's: the Erlang-B (loss) formula of 1917, which gives the fraction of calls lost when ccc trunks carry an offered load of rrr erlangs, and the Erlang-C formula, which gives the probability that a customer of a ccc-server queue must wait. Both are still the default dimensioning rules of telecommunications and call-center workforce management (Gans, Koole & Mandelbaum 2003).

This mission formalizes Chapter 2, §§2.1–2.10, of Gross, Shortle, Thompson and Harris, Fundamentals of Queueing Theory, 4th ed. (Wiley 2008), which derives these formulas from a single result about birth–death processes and closes with the modern answer to the staffing question.

Timeline. Erlang (1917) obtained the loss formula; Vaulot (1927), Pollaczek (1932), Palm (1938) and Kosten (1948) completed its proof for general service times. Halfin and Whitt (1981) showed that in the M/M/nM/M/nM/M/n queue the delay probability converges to a limit strictly between 000 and 111 exactly when the number of servers exceeds the offered load by an amount of order n\sqrt nn​. This is the quality-and-efficiency-driven (QED) regime on which square-root staffing rests.

Setting

A birth–death process is a continuous-time Markov chain on the states n∈{0,1,2,… }n \in \{0, 1, 2, \dots\}n∈{0,1,2,…} that moves from nnn to n+1n+1n+1 at rate λn≥0\lambda_n \ge 0λn​≥0 (a birth, or arrival) and, for n≥1n \ge 1n≥1, from nnn to n−1n-1n−1 at rate μn>0\mu_n > 0μn​>0 (a death, or departure). A steady-state solution is a probability sequence {pn}\{p_n\}{pn​} (pn≥0p_n \ge 0pn​≥0, ∑npn=1\sum_n p_n = 1∑n​pn​=1) solving the global balance equations (2.1):

(λn+μn)pn=λn−1pn−1+μn+1pn+1 (n≥1),λ0p0=μ1p1.(\lambda_n + \mu_n)p_n = \lambda_{n-1}p_{n-1} + \mu_{n+1}p_{n+1}\ (n \ge 1), \qquad \lambda_0 p_0 = \mu_1 p_1.(λn​+μn​)pn​=λn−1​pn−1​+μn+1​pn+1​ (n≥1),λ0​p0​=μ1​p1​.

The queues of the chapter are birth–death processes with particular rates. The M/M/1M/M/1M/M/1 queue has λn=λ\lambda_n = \lambdaλn​=λ, μn=μ\mu_n = \muμn​=μ and traffic intensity ρ=λ/μ\rho = \lambda/\muρ=λ/μ. The M/M/cM/M/cM/M/c queue has λn=λ\lambda_n = \lambdaλn​=λ, μn=min⁡(n,c)μ\mu_n = \min(n, c)\muμn​=min(n,c)μ (2.30), offered load r=λ/μr = \lambda/\mur=λ/μ and ρ=r/c\rho = r/cρ=r/c. The M/M/c/cM/M/c/cM/M/c/c loss system is the same with λn=0\lambda_n = 0λn​=0 for n≥cn \ge cn≥c. The M/M/∞M/M/\inftyM/M/∞ queue has μn=nμ\mu_n = n\muμn​=nμ.

The explicit functions are the Erlang-B formula

B(c,r)=rc/c!∑i=0cri/i!,B(c, r) = \frac{r^c/c!}{\sum_{i=0}^{c} r^i/i!},B(c,r)=∑i=0c​ri/i!rc/c!​,

the Erlang-C formula, defined for ρ=r/c<1\rho = r/c < 1ρ=r/c<1,

C(c,r)=rc/(c!(1−ρ))rc/(c!(1−ρ))+∑n=0c−1rn/n!,C(c, r) = \frac{r^c/(c!(1-\rho))}{r^c/(c!(1-\rho)) + \sum_{n=0}^{c-1} r^n/n!},C(c,r)=rc/(c!(1−ρ))+∑n=0c−1​rn/n!rc/(c!(1−ρ))​,

and, with ϕ\phiϕ, Φ\PhiΦ the standard normal density and distribution function,

α(β)=ϕ(β)ϕ(β)+βΦ(β).\alpha(\beta) = \frac{\phi(\beta)}{\phi(\beta) + \beta\Phi(\beta)}.α(β)=ϕ(β)+βΦ(β)ϕ(β)​.

Formalization targets

Goal: the Halfin–Whitt theorem (§2.4, p.75)

For offered loads 0<rn<n0 < r_n < n0<rn​<n,

lim⁡n→∞C(n,rn)=α∈(0,1)  ⟺  lim⁡n→∞n−rnn=β>0,α=α(β).\lim_{n\to\infty} C(n, r_n) = \alpha \in (0,1) \iff \lim_{n\to\infty} \frac{n - r_n}{\sqrt n} = \beta > 0, \qquad \alpha = \alpha(\beta).n→∞lim​C(n,rn​)=α∈(0,1)⟺n→∞lim​n​n−rn​​=β>0,α=α(β).

It is stated as three facts: α\alphaα maps (0,∞)(0, \infty)(0,∞) into (0,1)(0, 1)(0,1); each α∈(0,1)\alpha \in (0, 1)α∈(0,1) has exactly one preimage β>0\beta > 0β>0; and for every β>0\beta > 0β>0 the two limits are equivalent.

Milestones

  1. (2.3)–(2.4): the steady-state solution of a general birth–death process, pn=p0∏i=1nλi−1/μip_n = p_0\prod_{i=1}^n \lambda_{i-1}/\mu_ipn​=p0​∏i=1n​λi−1​/μi​, and its existence if and only if 1+∑n≥1∏i=1nλi−1/μi<∞1 + \sum_{n\ge1}\prod_{i=1}^n \lambda_{i-1}/\mu_i < \infty1+∑n≥1​∏i=1n​λi−1​/μi​<∞.
  2. (2.9): M/M/1M/M/1M/M/1, pn=(1−ρ)ρnp_n = (1-\rho)\rho^npn​=(1−ρ)ρn, existing iff ρ<1\rho < 1ρ<1.
  3. (2.31)–(2.32): the M/M/cM/M/cM/M/c law, existing iff λ/(cμ)<1\lambda/(c\mu) < 1λ/(cμ)<1.
  4. (2.33): Lq=rcρ p0/(c!(1−ρ)2)L_q = r^c\rho\,p_0/(c!(1-\rho)^2)Lq​=rcρp0​/(c!(1−ρ)2).
  5. (2.37)–(2.38): 1−∑n<cpn=C(c,r)1 - \sum_{n<c} p_n = C(c, r)1−∑n<c​pn​=C(c,r).
  6. (2.52)–(2.53): the M/M/c/cM/M/c/cM/M/c/c law and pc=B(c,r)p_c = B(c, r)pc​=B(c,r).
  7. (2.54): B(c,r)=rB(c−1,r)/(c+rB(c−1,r))B(c, r) = rB(c-1, r)/(c + rB(c-1, r))B(c,r)=rB(c−1,r)/(c+rB(c−1,r)), B(0,r)=1B(0, r) = 1B(0,r)=1.
  8. (2.55): C(c,r)=cB(c,r)/(c−r+rB(c,r))C(c, r) = cB(c, r)/(c - r + rB(c, r))C(c,r)=cB(c,r)/(c−r+rB(c,r)).
  9. (2.57): M/M/∞M/M/\inftyM/M/∞, pn=rne−r/n!p_n = r^n e^{-r}/n!pn​=rne−r/n!.

Significance

The results. Items 1–9 are the working formulas of Markovian capacity planning: a stationary law for each basic model and the measures read off from it. (2.54) and (2.55) are how BBB and CCC are computed in practice, since the factorials of the closed forms overflow for c>170c > 170c>170. The Halfin–Whitt theorem is the reason the rule c≈r+βrc \approx r + \beta\sqrt rc≈r+βr​ holds a fixed service level, and it is the entry point to the QED heavy-traffic literature (diffusion limits of many-server queues, Garnett–Mandelbaum–Reiman, Gamarnik–Momčilović).

Formalizing them. All results are classical and proved in the literature. The book states the Halfin–Whitt theorem without proof, and (2.54)–(2.55) are left to exercises. The formalization would supply machine-checked versions of the Erlang identities and of the Halfin–Whitt limit theorem. No Lean development of either was found on the platform when this mission was drafted. A related Erlang-B statement from Kelly and Yudovina is on the platform, stated with detailed balance on a finite state space.

Difficulty

The stationary laws are induction plus geometric and exponential series, and the Erlang identities are finite algebra. The difficulty is concentrated in the goal. C(n,rn)C(n, r_n)C(n,rn​) is a ratio of a Poisson-type tail to a truncated exponential sum in which both nnn and rnr_nrn​ grow. The naive route, substituting Stirling's formula term by term, fails: the sums have Θ(n)\Theta(\sqrt n)Θ(n​) significant terms, each of relative size exp⁡(−k2/2n)\exp(-k^2/2n)exp(−k2/2n), and the error has to be controlled uniformly over them. The converse direction also requires showing that α(⋅)\alpha(\cdot)α(⋅) is strictly monotone. Without that, convergence of C(n,rn)C(n, r_n)C(n,rn​) does not force convergence of (n−rn)/n(n - r_n)/\sqrt n(n−rn​)/n​.

Formalization scope

Rates are real sequences indexed by N\mathbb NN, and a steady-state solution is a real sequence with HasSum p 1, nonnegative entries, and the balance equations (2.1) exactly as printed (global balance, not detailed balance). Every "the steady-state solution is X" is stated in both halves: X is a steady-state solution, and every steady-state solution equals X; the book's existence conditions (ρ<1\rho < 1ρ<1, λ/(cμ)<1\lambda/(c\mu) < 1λ/(cμ)<1, convergence of the series) are part of the statements. The M/M/c/cM/M/c/cM/M/c/c system is the N\mathbb NN-indexed process with λn=0\lambda_n = 0λn​=0 for n≥cn \ge cn≥c, as §2.5 sets it up; the statement records that states above ccc carry no mass.

The closed forms that are fixed in Lean: ∏i=1nλi−1/μi\prod_{i=1}^n \lambda_{i-1}/\mu_i∏i=1n​λi−1​/μi​ over Finset.Icc 1 n; B(c,r)B(c, r)B(c,r) and C(c,r)C(c, r)C(c,r) exactly as displayed above; ϕ\phiϕ = gaussianPDFReal 0 1, Φ\PhiΦ = the CDF of gaussianReal 0 1; Wq(0)=∑n=0c−1pnW_q(0) = \sum_{n=0}^{c-1} p_nWq​(0)=∑n=0c−1​pn​, as evaluated on p.69; Lq=∑n>c(n−c)pnL_q = \sum_{n > c}(n - c)p_nLq​=∑n>c​(n−c)pn​ as a convergent series.

C(c,r)C(c, r)C(c,r) is a total function in Lean, but its value for r≥cr \ge cr≥c carries no meaning. The goal assumes 0<rn<n0 < r_n < n0<rn​<n for n≥1n \ge 1n≥1, the book's standing condition ρ<1\rho < 1ρ<1. A statement about some other function with the same limiting behaviour, or with BBB and CCC left abstract, would not be this mission. Neither would one-directional or existence-only versions of the stationary laws.

Not included: the waiting-time distributions (2.28) and (2.39), which need an FCFS waiting-time model with arrival-point probabilities; the M/M/c/KM/M/c/KM/M/c/K measures (2.45)–(2.48); finite-source and state-dependent models (§§2.8–2.10). Useful contributions beyond the milestones are Poisson tail estimates at the n\sqrt nn​ scale and monotonicity of α(β)\alpha(\beta)α(β). Both are reusable in other many-server heavy-traffic statements.

Selected references

  • D. Gross, J. F. Shortle, J. M. Thompson, C. M. Harris, Fundamentals of Queueing Theory, 4th ed., Wiley, 2008. https://doi.org/10.1002/9781118625651
  • S. Halfin, W. Whitt, Heavy-traffic limits for queues with many exponential servers, Operations Research 29(3), 567–588, 1981. https://doi.org/10.1287/opre.29.3.567
  • N. Gans, G. Koole, A. Mandelbaum, Telephone call centers: tutorial, review, and research prospects, Manufacturing & Service Operations Management 5(2), 79–141, 2003. https://doi.org/10.1287/msom.5.2.79.16071
  • A. K. Erlang, Solution of some problems in the theory of probabilities of significance in automatic telephone exchanges, Elektroteknikeren 13, 1917 (English translation in The Life and Works of A. K. Erlang, 1948).
  • F. P. Kelly, E. Yudovina, Stochastic Networks, Cambridge University Press, 2014. https://doi.org/10.1017/CBO9781139565363
12 thms3 active usersReviewed
🏆Completed
Algorithmic Game TheoryLinear OptimizationMechanism Design+1·Captain: mikedeng1

An Introduction to the Theory of Mechanism Design VII: Crémer–McLean Full Surplus Extraction with Correlated TypesTextbook

Motivation

Bayesian mechanism design asks which collective decisions and payments a designer can implement when every agent holds private information, the type, drawn from a commonly known prior. Most of the classical theory (Myerson's optimal auction, the Myerson–Satterthwaite impossibility, the public goods results) assumes that types are independent. Once types are correlated, as they are when bidders' values share a common component, the theory changes in a way that is best read as a paradox: Crémer and McLean (Econometrica 1988) showed that a designer can then extract the entire surplus, leaving agents no information rents. This mission formalizes Chapter 6 of Börgers, An Introduction to the Theory of Mechanism Design (Oxford University Press 2015), which sets up Bayesian mechanism design in general, treats independent types, and proves the Crémer–McLean theorem, together with the one numbered result of Chapter 9, the impossibility theorem of Jehiel and Moldovanu for interdependent values.

Timeline of the results formalized here:

  • Rochet (1987) characterized implementable decision rules by cyclical monotonicity; Proposition 6.1 is its interim version for independent types.
  • Crémer and McLean (1988): under their condition on the prior, every direct mechanism can be made Bayesian incentive-compatible without changing its decision rule or interim payments (Proposition 6.4).
  • Krishna and Maenner (2001): revenue equivalence for convex type sets and convex utilities (Proposition 6.2).
  • Jehiel and Moldovanu (2001): with interdependent values, efficient decisions are generically not Bayesian implementable (Proposition 9.1).
  • Kosenok and Severinov (2008): an identifiability condition added to Crémer–McLean gives ex post budget balance as well (Proposition 6.6).

Setting

There are finitely many agents i∈Ii \in Ii∈I and a set AAA of alternatives. Agent iii has a type θi∈Θi\theta_i \in \Theta_iθi​∈Θi​ and utility ui(a,θi)−tiu_i(a,\theta_i) - t_iui​(a,θi​)−ti​ from alternative aaa and payment tit_iti​. Types θ=(θ1,…,θN)∈Θ=∏iΘi\theta = (\theta_1,\dots,\theta_N) \in \Theta = \prod_i \Theta_iθ=(θ1​,…,θN​)∈Θ=∏i​Θi​ are drawn from a common prior μ\muμ; μ(⋅∣θi)\mu(\cdot\mid\theta_i)μ(⋅∣θi​) is the conditional distribution of the others' types θ−i\theta_{-i}θ−i​ given θi\theta_iθi​. Types are independent if μ(⋅∣θi)\mu(\cdot\mid\theta_i)μ(⋅∣θi​) does not depend on θi\theta_iθi​.

A direct mechanism (q,t1,…,tN)(q, t_1,\dots,t_N)(q,t1​,…,tN​) is a decision rule q:Θ→Aq:\Theta\to Aq:Θ→A and payment rules ti:Θ→Rt_i:\Theta\to\mathbb Rti​:Θ→R. It is Bayesian incentive-compatible (BIC) if for every iii and all θi,θi′\theta_i,\theta_i'θi​,θi′​,

∫Θ−iui(q(θi,θ−i),θi)−ti(θi,θ−i) dμ(θ−i∣θi) ≥ ∫Θ−iui(q(θi′,θ−i),θi)−ti(θi′,θ−i) dμ(θ−i∣θi).\int_{\Theta_{-i}} u_i(q(\theta_i,\theta_{-i}),\theta_i) - t_i(\theta_i,\theta_{-i})\,d\mu(\theta_{-i}\mid\theta_i) \ \ge\ \int_{\Theta_{-i}} u_i(q(\theta_i',\theta_{-i}),\theta_i) - t_i(\theta_i',\theta_{-i})\,d\mu(\theta_{-i}\mid\theta_i).∫Θ−i​​ui​(q(θi​,θ−i​),θi​)−ti​(θi​,θ−i​)dμ(θ−i​∣θi​) ≥ ∫Θ−i​​ui​(q(θi′​,θ−i​),θi​)−ti​(θi′​,θ−i​)dμ(θ−i​∣θi​).

The interim decision rule Qi(θi)Q_i(\theta_i)Qi​(θi​) is the distribution of q(θi,θ−i)q(\theta_i,\theta_{-i})q(θi​,θ−i​) given θi\theta_iθi​, and the interim expected payment is Ti(θi)=∫ti(θi,θ−i) dμ(θ−i∣θi)T_i(\theta_i) = \int t_i(\theta_i,\theta_{-i})\,d\mu(\theta_{-i}\mid\theta_i)Ti​(θi​)=∫ti​(θi​,θ−i​)dμ(θ−i​∣θi​). A mechanism is ex post budget balanced if ∑iti(θ)=0\sum_i t_i(\theta) = 0∑i​ti​(θ)=0 for every θ\thetaθ, and ex ante budget balanced if ∫Θ∑iti dμ=0\int_\Theta \sum_i t_i\,d\mu = 0∫Θ​∑i​ti​dμ=0.

In §6.4 every Θi\Theta_iΘi​ is finite and μ(θ)>0\mu(\theta) > 0μ(θ)>0 for every θ\thetaθ. The prior satisfies the Crémer–McLean condition if for no agent iii and type θi\theta_iθi​ there are weights λ≥0\lambda \ge 0λ≥0 on Θi∖{θi}\Theta_i\setminus\{\theta_i\}Θi​∖{θi​} with

μ(θ−i∣θi)=∑θi′≠θiλ(θi′) μ(θ−i∣θi′)for all θ−i.\mu(\theta_{-i}\mid\theta_i) = \sum_{\theta_i'\ne\theta_i}\lambda(\theta_i')\,\mu(\theta_{-i}\mid\theta_i')\quad\text{for all }\theta_{-i}.μ(θ−i​∣θi​)=θi′​=θi​∑​λ(θi′​)μ(θ−i​∣θi′​)for all θ−i​.

Identifiability requires that for every full-support distribution ν≠μ\nu\ne\muν=μ some agent's type θi\theta_iθi​ has a belief ν(⋅∣θi)\nu(\cdot\mid\theta_i)ν(⋅∣θi​) that is not a nonnegative combination of the beliefs μ(⋅∣θi′)\mu(\cdot\mid\theta_i')μ(⋅∣θi′​).

Formalization targets

Goal: Crémer–McLean (Proposition 6.4)

If μ\muμ satisfies the Crémer–McLean condition, then for every direct mechanism (q,t)(q,t)(q,t) there is a BIC direct mechanism (q,t′)(q,t')(q,t′) with

∑θ−iti(θi,θ−i) μ(θ−i∣θi)=∑θ−iti′(θi,θ−i) μ(θ−i∣θi)for all i,θi.\sum_{\theta_{-i}} t_i(\theta_i,\theta_{-i})\,\mu(\theta_{-i}\mid\theta_i) = \sum_{\theta_{-i}} t_i'(\theta_i,\theta_{-i})\,\mu(\theta_{-i}\mid\theta_i)\quad\text{for all } i,\theta_i.θ−i​∑​ti​(θi​,θ−i​)μ(θ−i​∣θi​)=θ−i​∑​ti′​(θi​,θ−i​)μ(θ−i​∣θi​)for all i,θi​.

Milestones

  1. Proposition 6.1 (independent types): qqq is part of a BIC mechanism iff it is interim cyclically monotone, ∑κ=1k−1(∫Aui(a,θiκ+1) dQi(θiκ)−∫Aui(a,θiκ) dQi(θiκ))≤0\sum_{\kappa=1}^{k-1}\big(\int_A u_i(a,\theta_i^{\kappa+1})\,dQ_i(\theta_i^\kappa) - \int_A u_i(a,\theta_i^\kappa)\,dQ_i(\theta_i^\kappa)\big)\le 0∑κ=1k−1​(∫A​ui​(a,θiκ+1​)dQi​(θiκ​)−∫A​ui​(a,θiκ​)dQi​(θiκ​))≤0 for every cycle θik=θi1\theta_i^k = \theta_i^1θik​=θi1​.
  2. Proposition 6.2 (independent types, convex type sets, convex utilities): two BIC mechanisms with Qi′=QiQ_i' = Q_iQi′​=Qi​ have Ti′=Ti+τiT_i' = T_i + \tau_iTi′​=Ti​+τi​.
  3. Proposition 6.3 (independent types): every ex ante budget balanced mechanism has an equivalent ex post budget balanced one.
  4. Proposition 6.5 (Farkas's alternative), already proved on the platform as Polyhedral.farkas_lemma.
  5. Proposition 6.6 (Kosenok–Severinov): under Crémer–McLean and identifiability, every ex ante budget balanced mechanism has an equivalent BIC and ex post budget balanced one.
  6. Proposition 9.1 (Jehiel–Moldovanu): in the linear interdependent-values model, under a regularity condition on first best rules and the weight condition αaii/αbii≠∑jαaji/∑jαbji\alpha^i_{ai}/\alpha^i_{bi}\ne\sum_j\alpha^i_{aj}/\sum_j\alpha^i_{bj}αaii​/αbii​=∑j​αaji​/∑j​αbji​ for some i,a,bi,a,bi,a,b, no first best direct mechanism is BIC. It uses its own model (§9.3) and is not on the goal's proof path.

Significance

The Crémer–McLean theorem says that, with correlated finite types, incentive compatibility imposes essentially no constraint: every decision rule and every interim payment rule can be implemented. In a single-unit auction this gives full surplus extraction. The result is the benchmark against which the literature on risk aversion, limited liability, collusion and the genericity of priors (Robert 1991, Laffont–Martimort 2000, Heifetz–Neeman 2006) measures its departures, and Proposition 6.6 extends it to budget-balanced mechanisms, which is what bilateral trade and public goods applications need. Propositions 6.1–6.3 are the independent-types counterpart that the correlated case breaks: they show why revenue equivalence and the ex ante/ex post budget-balance equivalence hold there and fail here. Proposition 9.1 shows the opposite failure, for interdependent values, where efficient decisions cannot be implemented even without participation or budget constraints.

All results are proved in the literature (Kosenok–Severinov's proof is omitted in the book). Apart from Farkas's alternative (Proposition 6.5), which is proved on the platform, none of them is formalized in Mathlib or on the platform; the platform's other mechanism design results (dominant-strategy results in a valuation model, revenue equivalence for symmetric independent auctions) do not cover correlated types.

Difficulty

For the goal the difficulty is the uniformity of the construction: a single payment adjustment must make truth-telling optimal against every possible deviation of every type of every agent, while leaving each type's expected payment unchanged. The obvious scoring-rule adjustment, charging −ln⁡μ(θ−i∣θi′)-\ln\mu(\theta_{-i}\mid\theta_i')−lnμ(θ−i​∣θi′​), changes interim payments, and removing that change is where the Crémer–McLean condition enters. For Proposition 6.2 the envelope argument must handle convex type sets that are not open and utilities that are only convex, not differentiable. For Proposition 9.1 the obvious argument differentiates interim utility twice; the proposition does not assume that interim utility is twice differentiable, so that regularity has to be derived from the hypotheses on the interim probabilities.

Formalization scope

Three definition files carry the three models. Independent types (§6.2–6.3): type sets are arbitrary measurable spaces, the prior is the product of probability measures ρi\rho_iρi​, and interim quantities are integrals against it. Finite correlated types (§6.4): finite type sets, a prior μ:Θ→R\mu:\Theta\to\mathbb Rμ:Θ→R with μ(θ)>0\mu(\theta)>0μ(θ)>0 and ∑θμ(θ)=1\sum_\theta\mu(\theta)=1∑θ​μ(θ)=1, and conditional beliefs μ(θ−i∣θi)=μ(θ)/μ(θi)\mu(\theta_{-i}\mid\theta_i) = \mu(\theta)/\mu(\theta_i)μ(θ−i​∣θi​)=μ(θ)/μ(θi​); the alternative set AAA is arbitrary. Interdependent values (§9.3): finite AAA, signals in [0,1]A[0,1]^A[0,1]A with positive densities, independent across agents, linear utilities with nonzero weights αaij\alpha^j_{ai}αaij​.

Committed conventions and explicit formulas:

  • The Crémer–McLean condition uses nonnegative weights without a sum-to-one constraint, as Definition 6.7 prints it.
  • "Equivalent" in Propositions 6.4 and 6.6 means the same decision rule and the same interim expected payments at truthful reports, as Proposition 6.4 states; in Proposition 6.3 it is the report-by-report notion, which under independence is equality of the interim payment rules TiT_iTi​.
  • Proposition 6.2 adds continuity of ui(a,⋅)u_i(a,\cdot)ui​(a,⋅) on Θi\Theta_iΘi​, and Proposition 6.3 adds at least two agents; without them the printed statements are false.
  • Measurability, which the book omits throughout, is made explicit: decision rules are measurable, and the payment sections and utilities are integrable against the relevant interim distributions.
  • In Proposition 9.1 the partial derivatives are derivatives within the closed cube [0,1]K[0,1]^K[0,1]K, and the weight condition is the displayed ratio inequality.

A trivializing formalization of the goal, one that proves it only for mechanisms that are already incentive compatible, or with a payment rule that is not a function of the reported type profile, or with "equivalent" weakened to "some BIC mechanism exists", is ruled out: the statement quantifies over every direct mechanism and fixes both the decision rule and every interim payment.

Reusable infrastructure: finite conditional expectations under a full-support prior, the Crémer–McLean and identifiability conditions, and the interim model with product priors. Proofs of any milestone and of the goal, including via the Farkas reference, are welcome.

Selected references

  • T. Börgers, An Introduction to the Theory of Mechanism Design, Oxford University Press, 2015. https://doi.org/10.1093/acprof:oso/9780199734023.001.0001
  • J. Crémer and R. P. McLean, "Full extraction of the surplus in Bayesian and dominant strategy auctions", Econometrica 56(6), 1988. https://doi.org/10.2307/1913096
  • G. Kosenok and S. Severinov, "Individually rational, budget-balanced mechanisms and allocation of surplus", Journal of Economic Theory 140(1), 2008.
  • P. Jehiel and B. Moldovanu, "Efficient design with interdependent valuations", Econometrica 69(5), 2001. https://doi.org/10.1111/1468-0262.00237
  • V. Krishna and E. Maenner, "Convex potentials with an application to mechanism design", Econometrica 69(4), 2001. https://doi.org/10.1111/1468-0262.00225
  • J.-C. Rochet, "A necessary and sufficient condition for rationalizability in a quasi-linear context", Journal of Mathematical Economics 16(2), 1987. https://doi.org/10.1016/0304-4068(87)90007-3
10 thms3 active usersReviewed
🏆Completed
Algorithmic Game TheoryDynamic ProgrammingOptimization·Captain: mikedeng1

Bellman's Dynamic Programming VIII: Multi-Stage Games, Games of Survival and the Extended Min-Max TheoremTextbook

Why multi-stage games

Chapter X of Richard Bellman's Dynamic Programming (Princeton University Press, 1957; Princeton Landmarks edition 2010, DOI 10.2307/j.ctv1nxcw0f) carries the functional-equation method of the earlier chapters from one-player decision processes to two-player zero-sum games that are played over many stages. At each stage the two players make simultaneous choices, and those choices change the state in which the next stage is played. Games of survival are the standard example. They generalize the gambler's ruin: two players with finite resources play until one of them is ruined. The same framework is the discrete-time form of what Shapley (1953) called stochastic games. It underlies pursuit games, attrition models and the later theory of dynamic games in operations research and economics.

The chapter builds on von Neumann's min-max theorem (1928), which Bellman assumes without proof (§ 3, Eq. (3.4)). It proves three kinds of result: existence and uniqueness for the general multi-stage game equation (§§ 11–16), existence and uniqueness for games of survival with integer payoffs (§ 19), and an extended min-max theorem for ratios of bilinear forms (§ 23). Bellman proposes the ratio as a criterion for non-zero-sum games.

Setting

A matrix game is given by a real M×NM\times NM×N matrix A=(aij)A=(a_{ij})A=(aij​). Player AAA chooses row iii with probability pip_ipi​ and player BBB chooses column jjj with probability qjq_jqj​; ppp and qqq are distribution vectors (points of the standard simplex). The expected return to AAA is

EA(p,q)=∑i,jaij pi qj.E_A(p,q)=\sum_{i,j}a_{ij}\,p_i\,q_j .EA​(p,q)=i,j∑​aij​pi​qj​.

The game has value vvv when max⁡pmin⁡qEA=v=min⁡qmax⁡pEA\max_p\min_q E_A=v=\min_q\max_p E_Amaxp​minq​EA​=v=minq​maxp​EA​, all extrema attained. In the Lean development this is IsMaxMinMinMaxValue, and the bilinear form is bilin.

In the general multi-stage game (§ 12) the state is a pair of vectors P∈D⊆RnP\in D\subseteq\mathbb R^nP∈D⊆Rn, P′∈D′⊆Rn′P'\in D'\subseteq\mathbb R^{n'}P′∈D′⊆Rn′ with norms ∥P∥=∑i∣Pi∣\|P\|=\sum_i|P_i|∥P∥=∑i​∣Pi​∣. In state (P,P′)(P,P')(P,P′), AAA chooses uuu in a choice domain S(P,P′)S(P,P')S(P,P′) and BBB chooses vvv in S′(P,P′)S'(P,P')S′(P,P′). AAA receives R(u,v)R(u,v)R(u,v), and play continues from the state T(P,P′;u,v)T(P,P';u,v)T(P,P′;u,v), T′(P,P′;u,v)T'(P,P';u,v)T′(P,P′;u,v) with weight h(P,P′;u,v)h(P,P';u,v)h(P,P′;u,v). Mixed strategies GGG, G′G'G′ are probability measures on the choice domains. The multi-stage game equation is

f(P,P′)=max⁡Gmin⁡G′∬[R(u,v)+h(P,P′;u,v) f(T,T′)] dG(u) dG′(v)=min⁡G′max⁡G[ ⋯ ].f(P,P')=\max_G\min_{G'}\iint\big[R(u,v)+h(P,P';u,v)\,f(T,T')\big]\,dG(u)\,dG'(v)=\min_{G'}\max_G[\ \cdots\ ].f(P,P′)=Gmax​G′min​∬[R(u,v)+h(P,P′;u,v)f(T,T′)]dG(u)dG′(v)=G′min​Gmax​[ ⋯ ].

A game of survival with integer payoffs (§ 19) has one integer state xxx, the resources of AAA out of a fixed total ddd. AAA is ruined at x≤0x\le0x≤0 and wins at x≥dx\ge dx≥d. At each stage the players play the 2×22\times22×2 game with matrix (−1ac−b)\begin{pmatrix}-1&a\\c&-b\end{pmatrix}(−1c​a−b​), whose entries are the transfers to AAA.

Formalization targets

Goal: the extended min-max theorem (Chapter X, Theorem 7)

For real matrices A=(aij)A=(a_{ij})A=(aij​), B=(bij)B=(b_{ij})B=(bij​) with ∑i,jbijpiqj≥d>0\sum_{i,j}b_{ij}p_iq_j\ge d>0∑i,j​bij​pi​qj​≥d>0 for all distribution vectors p,qp,qp,q,

max⁡pmin⁡q∑i,jaijpiqj∑i,jbijpiqj=min⁡qmax⁡p∑i,jaijpiqj∑i,jbijpiqj.\max_p\min_q\frac{\sum_{i,j}a_{ij}p_iq_j}{\sum_{i,j}b_{ij}p_iq_j}=\min_q\max_p\frac{\sum_{i,j}a_{ij}p_iq_j}{\sum_{i,j}b_{ij}p_iq_j}.pmax​qmin​∑i,j​bij​pi​qj​∑i,j​aij​pi​qj​​=qmin​pmax​∑i,j​bij​pi​qj​∑i,j​aij​pi​qj​​.

It is the chapter's final result, and its statement involves only finite matrices.

Milestones

  1. Eq. (3.4), von Neumann's theorem VA=VBV_A=V_BVA​=VB​. It is already on the platform as AGT.zero_sum_minimax (existence of a saddle point) and enters as a reference.
  2. Lemma 1: the value of a one-stage game moves by at most the largest change of its kernel, ∣L(f)−L1(F)∣≤max⁡u,v [ ∣R−R1∣+∣h∣ ∣f(T,T′)−F(T,T′)∣ ]|L(f)-L_1(F)|\le\max_{u,v}\,[\,|R-R_1|+|h|\,|f(T,T')-F(T,T')|\,]∣L(f)−L1​(F)∣≤maxu,v​[∣R−R1​∣+∣h∣∣f(T,T′)−F(T,T′)∣].
  3. Theorem 1: under hypotheses (4a)–(4e), the multi-stage game equation has a unique solution among functions continuous on D×D′D\times D'D×D′ that vanish at the origin, and it is the uniform limit on bounded regions of the successive approximations.
  4. Theorem 3: the successive approximations converge from any admissible initial function.
  5. Theorem 4: stability, ∣f(P,P′)−F(P,P′)∣≤∑n≥0Δ(knc)|f(P,P')-F(P,P')|\le\sum_{n\ge0}\Delta(k^nc)∣f(P,P′)−F(P,P′)∣≤∑n≥0​Δ(knc).
  6. Theorem 5: the game of survival equation with boundary values 000 and 111 has a unique solution with values in [0,1][0,1][0,1].

Significance

Theorem 7 gives a value for ratio games, the games in which a player maximizes a return per unit of a resource consumed. Bellman uses it to give a rationale for the play of non-zero-sum games (§ 24) and to derive the approximate equation (22.4) for non-zero-sum games of survival. Chapter XI, Theorem 5 generalizes it to Markovian decision processes. Theorem 1 is the existence and uniqueness result that justifies replacing an infinite game by its functional equation, and Theorems 3 and 4 make that equation usable for computation and perturbation. Theorem 5 is an early uniqueness theorem for a discrete stochastic game with absorbing boundaries.

None of these statements is formalized on the platform. Von Neumann's theorem is (AGT.zero_sum_minimax, and Sion's theorem as FamousTheorems.sion_minimax_theorem). Theorem 7 is a classical result: with a positive denominator the ratio is quasiconcave in ppp and quasiconvex in qqq. This mission asks for a machine-checked proof of the book's statement, by Bellman's route or by any other. Theorems 1, 3 and 4 need a formal theory of games whose mixed strategies are probability measures on moving compact choice domains. No such theory is on the platform yet.

Difficulty

The ratio in Theorem 7 is not bilinear, and in general it is neither concave in ppp nor convex in qqq, so neither von Neumann's theorem nor a concave-convex min-max theorem applies to it directly. For Theorem 1, the contraction argument needs each one-stage game to have a value and the value to depend continuously on the state. That requires a min-max theorem for continuous games on compact sets, together with continuity of the value when the choice domains move. In Theorem 5, existence follows from monotone iteration. The uniqueness step is the hard part: the operator is not a contraction in the sup norm, and a second solution must be ruled out even at states where the optimal mixed strategies are degenerate.

Formalization scope

  • Distribution vectors are points of stdSimplex ℝ ι for nonempty finite types. "Max-min equals min-max" always asserts attainment (IsGreatest/IsLeast) and never uses sSup/sInf, which return 000 on empty or unbounded sets. Theorem 7 must not assume a saddle point; its only hypothesis is the bound on the denominator.
  • States are Fin n → ℝ with the ℓ1\ell^1ℓ1 norm of Eq. (12.1). Choice domains are nonempty compact sets (NonemptyCompacts), and mixed strategies are probability measures of full mass on them. "Vary continuously" in (4b), which the book does not define, is read as continuity in Mathlib's topology on nonempty compact sets (for a metric space, the topology of the Hausdorff distance).
  • The maxima w(c)w(c)w(c) of (4d) and Δ(c)\Delta(c)Δ(c) of Theorem 4 are expressed through majorants, which is equivalent to the book's condition. k≥0k\ge0k≥0 is stated explicitly. In Lemma 1 the kernels are assumed measurable and bounded on S×S′S\times S'S×S′ so that the integrals exist, and each game having a value is a hypothesis, as in Bellman's footnote 4.
  • Theorem 3's "converges" is stated as uniform convergence on bounded regions, the mode of Theorem 1, whose proof Theorem 3 repeats. Theorem 4's unspecified ccc is any c≥∥P∥+∥P′∥c\ge\|P\|+\|P'\|c≥∥P∥+∥P′∥. The book's p. 301 prints the first min-max of Theorem 4 with the subscripts GGG, G′G'G′ interchanged; the equation used is that of Theorem 1.
  • Theorem 5 adds the implicit d≥1d\ge1d≥1: for d≤0d\le0d≤0 the boundary conditions contradict each other. The state is an integer and f:Z→Rf:\mathbb Z\to\mathbb Rf:Z→R.
  • Taking the choice domains constant would trivialize Theorem 1: condition (4d) would then force R≡0R\equiv0R≡0 on them. The choice domains therefore depend on the state.
  • Not included: Theorem 2 (optimal strategies of the infinite game, which needs a model of the game's plays), Theorem 6 (non-zero-sum survival; the boundary conditions (21.3) do not cover all states, see the moderation notes).

Contributions of reusable infrastructure are welcome: the min-max theorem for continuous games on compact sets (Eq. (4.2)), continuity of the value in the choice domains, and value iteration for contracting game operators.

Selected references

  • R. Bellman, Dynamic Programming, Princeton University Press, 1957; Princeton Landmarks in Mathematics, 2010. DOI 10.2307/j.ctv1nxcw0f
  • J. von Neumann, Zur Theorie der Gesellschaftsspiele, Mathematische Annalen 100, 1928. DOI 10.1007/BF01448847
  • L. S. Shapley, Stochastic games, Proceedings of the National Academy of Sciences 39, 1953. DOI 10.1073/pnas.39.10.1095
  • M. Sion, On general minimax theorems, Pacific Journal of Mathematics 8, 1958. DOI 10.2140/pjm.1958.8.171
  • I. L. Glicksberg, A further generalization of the Kakutani fixed point theorem, with application to Nash equilibrium points, Proceedings of the AMS 3, 1952. DOI 10.1090/S0002-9939-1952-0046638-5
9 thms3 active usersReviewed
🏆Completed
ProbabilityStatisticsStochastic Systems·Captain: mikedeng1

Elements of Queueing Theory V: Strassen's Theorems and the Stochastic Ordering of QueuesTextbook

Strassen's Theorems and the Stochastic Ordering of Queues

Background

Chapters 1–3 of Baccelli and Brémaud's Elements of Queueing Theory compute exact quantities: Palm identities, stability criteria, PASTA, Pollaczek–Khinchin. Chapter 4 asks a different question. When you cannot compute a queue, can you at least say it is better than another one?

That requires an order on distributions. The chapter builds a family of them — integral orders — by choosing a class ℒ of test functions and declaring F ≤_ℒ G when ∫f dF ≤ ∫f dG for all f ∈ ℒ. Three matter: {i} the non-decreasing functions, giving the strong (stochastic) order; {cx} the convex functions, giving the convex order; and their intersection {icx}.

The goal

An integral order compares two distributions that need not live on the same probability space, and that is both its convenience and its difficulty. Strassen's theorems say each of these orders is secretly a statement about a coupling.

Theorem 4.2.2 (p.278), Strassen's ≤_cx theorem:

F ≤_cx G   ⟺   ∃ X ~ F, Y ~ G on one space with  E[Y | X] = X  a.s.
F ≤_icx G  ⟺   the same with  E[Y | X] ≥ X  a.s.

The convex order holds exactly when G is a martingale dilation of F — obtained by spreading each point out without moving its conditional mean. That is what makes the order usable: comparison results for queues become induction arguments on a coupling instead of analytic manipulations of convolutions of c.d.f.'s.

Its companion Theorem 4.2.1 is the ≤_st version, where the coupling is the simpler X ≤ Y a.s. In dimension one both are explicit — take X = F⁻¹(U), Y = G⁻¹(U) for a uniform U. In dimension n there is no such formula, and that is why these are Strassen's theorems. The book attributes both to Strassen (1965) and proves neither.

Why FIFO is optimal

§4.1 is a different kind of comparison: not between two queues, but between two service disciplines for the same queue. The order there is majorization ≺, which compares how spread out two vectors of the same total are.

The answer is that FIFO minimizes E⁰[f(V)] for every convex f (Property 4.1.3), and the proof is an interchange argument. Under any non-preemptive discipline that uses no information on the service times, customer k effectively receives service σ_{γ(k)} for some permutation γ; Lemma 4.1.3 shows the same queue is produced by FIFO fed with that reordered input, and that the reordering does not change the law of the input. Lemma 4.1.4 passes to the limit, which needs ρ < 1. Lemmas 4.1.1 and 4.1.2 then do the combinatorics: undoing one inversion of γ makes the waiting-time vector less spread out, so the identity permutation — FIFO — is extremal.

Feller's paradox, and what survives it

§4.4 compares time-stationary queues, and opens with a warning. T_n[P⁰] ≤_i T̃_n[P̃⁰] for every n does not imply T_n[P] ≤_i T̃_n[P̃]: Example 4.4.1, "Feller's paradox revisited", exhibits a Poisson process and a renewal process where the Palm order holds and the stationary one fails. The order does not pass from the Palm probability to the stationary one.

For ≤_cx it does. Lemma 4.4.1 is why: it expands E_P[f(N[0,x))] as a series of second differences of f against Palm expectations, and a convex f makes every coefficient non-negative. Lemma 4.4.2 handles the S-orders, built by dividing Palm integrals by the mean cycle length, and shows that the normalisation does not hide the comparison it normalises by.

Formalization scope

  • Orders. ≤_i, ≤_cx, ≤_icx on distributions on ℝⁿ are integral orders over the book's test classes (§4.2.1), with the page's qualification that only test functions with well-defined integrals count. Majorization ≺ is (4.1.2) with increasing reorderings of both vectors.
  • Strassen. Both theorems are stated as equivalences, with the coupling existential over the probability space. Theorem 4.2.2 carries both clauses — E[Y | X] = X for ≤_cx, E[Y | X] ≥ X for ≤_icx, as conditional expectations given σ(X) — and assumes both distributions integrable; Theorem 4.2.1 has no integrability hypothesis. A one-directional statement (the Jensen half) is not the theorem.
  • The queue of §4.1.3 is constructed: a GI/GI input (i.i.d. inter-arrival and service times, independent), a single work-conserving server started empty, and any non-preemptive discipline whose choices are measurable in the information the book's σ-field 𝒢_t carries (arrivals, service times of customers already started) plus external randomisation. FIFO is one such discipline. The interchange permutations γ_n and their limit γ are built from the schedule as on pp.268–270; Lemma 4.1.4 assumes ρ = E[σ₀]/E[τ₀] < 1.
  • Lemma 4.4.1 is stated with the exact second-difference series and assumes that series converges absolutely; the page states it for all f, which fails for heavy-tailed counts and sparse f.
  • The S-orders test against {I-ℒ} — primitives ∫_0^t f(u, x) du of test functions — and apply only to distributions whose first coordinate is a.s. positive with a finite mean.

What this mission provides

None of it exists. Mathlib has no stochastic order, no convex order, no increasing-convex order, no majorization, no Schur-convexity and no Strassen theorem; the platform returns zero hits for q=stochastic ordering. Everything in this chapter is new substrate — and §§4.1–4.2 need nothing from Palm calculus, so this mission can be read on its own.

14 thms3 active usersReviewed
🏆Completed
Convex OptimizationLinear OptimizationOptimization·Captain: mikedeng1

Linear Programming: Foundations and Extensions II: Farkas' Lemma and Strict Complementary SlacknessTextbook

Motivation

Every linear program comes with a second linear program, its dual, and most of what is known about linear programming is a statement about how the two interact. Weak duality gives certificates of optimality; strong duality says those certificates always exist; complementary slackness turns optimality into a system of equations. These three facts are the core of any first course in optimization and of every correctness argument for the simplex method.

Strict complementarity is the sharpest statement of the same kind. Complementary slackness says that in each pair (a primal variable and its dual slack, a dual variable and its primal slack) at least one member vanishes at optimality. Strict complementarity says that some optimal pair can be chosen so that exactly one member vanishes in each pair. The result is due to Goldman and Tucker (1956). It is what identifies the optimal face of a linear program and its partition of the variables into those that can be positive at an optimum and those that cannot, and it is a standing ingredient in the analysis of interior-point methods, which approach this strictly complementary optimum rather than a vertex.

This mission formalizes the chain from duality to strict complementarity as it is developed in Chapters 5 and 10 of Vanderbei, Linear Programming: Foundations and Extensions (4th ed., Springer 2014, doi:10.1007/978-1-4614-7630-6). It is the second mission of a series on that book.

Timeline:

  • 1902 — Farkas publishes the lemma on the solvability of linear inequality systems.
  • 1947–1951 — von Neumann, and Gale, Kuhn and Tucker, establish linear programming duality.
  • 1956 — Goldman and Tucker prove the existence of strictly complementary optimal solutions (in Linear Inequalities and Related Systems, Annals of Mathematics Studies 38).

Setting

Fix integers m,n≥0m, n \ge 0m,n≥0, a real m×nm \times nm×n matrix A=(aij)A = (a_{ij})A=(aij​), a vector b∈Rmb \in \mathbb{R}^mb∈Rm and a vector c∈Rnc \in \mathbb{R}^nc∈Rn. The primal problem is

maximize cTxsubject toAx+w=b,x≥0, w≥0,(10.9)\text{maximize } c^T x \quad\text{subject to}\quad Ax + w = b,\quad x \ge 0,\ w \ge 0, \qquad (10.9)maximize cTxsubject toAx+w=b,x≥0, w≥0,(10.9)

where w=b−Axw = b - Axw=b−Ax is the primal slack. The dual problem is

minimize bTysubject toATy−z=c,y≥0, z≥0,(10.10)\text{minimize } b^T y \quad\text{subject to}\quad A^T y - z = c,\quad y \ge 0,\ z \ge 0, \qquad (10.10)minimize bTysubject toATy−z=c,y≥0, z≥0,(10.10)

where z=ATy−cz = A^T y - cz=ATy−c is the dual slack. A vector xxx is primal feasible if x≥0x \ge 0x≥0 and w≥0w \ge 0w≥0; it is primal optimal if it is feasible and cTx′≤cTxc^T x' \le c^T xcTx′≤cTx for every feasible x′x'x′. Dual feasibility and dual optimality are defined in the same way, with minimization. Inequalities between vectors are componentwise, and ξ>0\xi > 0ξ>0 means that every component of ξ\xiξ is strictly positive.

A halfspace of Rn\mathbb{R}^nRn is a set {x:aTx≤β}\{x : a^T x \le \beta\}{x:aTx≤β} with a≠0a \ne 0a=0; a polyhedron is a set {x:Ax≤b}\{x : Ax \le b\}{x:Ax≤b} for some mmm, AAA and bbb.

In the Lean development these objects live in the namespace VanderbeiLP.StrictComp: primalSlack A b x, dualSlack A c y, PrimalFeasible, DualFeasible, PrimalOptimal, DualOptimal, IsHalfspace, IsPolyhedron.

Formalization targets

Goal: Strict Complementary Slackness (Theorem 10.7)

If the primal (10.9) has an optimal solution, then there exist a primal optimal x∗x^*x∗ and a dual optimal y∗y^*y∗, with slacks w∗=b−Ax∗w^* = b - Ax^*w∗=b−Ax∗ and z∗=ATy∗−cz^* = A^T y^* - cz∗=ATy∗−c, such that

x∗+z∗>0andy∗+w∗>0.x^* + z^* > 0 \qquad\text{and}\qquad y^* + w^* > 0.x∗+z∗>0andy∗+w∗>0.

The only hypothesis is primal optimality; the existence of a dual optimum is part of the conclusion.

Milestones

  1. Theorem 5.1 (Weak Duality). Primal feasible xxx and dual feasible yyy satisfy cTx≤bTyc^T x \le b^T ycTx≤bTy.
  2. Theorem 5.2 (Strong Duality). If the primal has an optimal x∗x^*x∗, the dual has an optimal y∗y^*y∗ with cTx∗=bTy∗c^T x^* = b^T y^*cTx∗=bTy∗.
  3. Theorem 5.3 (Complementary Slackness). Feasible xxx, yyy are both optimal if and only if xjzj=0x_j z_j = 0xj​zj​=0 for all jjj and wiyi=0w_i y_i = 0wi​yi​=0 for all iii.
  4. Lemma 10.5 (Farkas' Lemma). Ax≤bAx \le bAx≤b has no solution if and only if some yyy satisfies ATy=0A^T y = 0ATy=0, y≥0y \ge 0y≥0, bTy<0b^T y < 0bTy<0.
  5. Theorem 10.4 (Separation of polyhedra). Two disjoint nonempty polyhedra lie in two disjoint halfspaces.
  6. Theorem 10.6. If both problems are feasible, there are feasible xˉ\bar xxˉ, yˉ\bar yyˉ​ with xˉ+zˉ>0\bar x + \bar z > 0xˉ+zˉ>0 and yˉ+wˉ>0\bar y + \bar w > 0yˉ​+wˉ>0.

Theorem 10.6 is the feasible-solution version of the goal; Theorem 10.4 is a further consequence of Farkas' Lemma in the same chapter.

Significance

Strict complementarity determines the optimal partition: the set of indices jjj for which some optimal x∗x^*x∗ has xj∗>0x^*_j > 0xj∗​>0 is exactly the complement of the set for which some optimal dual slack zj∗z^*_jzj∗​ is positive. This partition describes the optimal faces of both problems, is the object that interior-point methods recover in the limit, and is the starting point of sensitivity analysis beyond a single optimal basis. Farkas' Lemma and the separation theorem are the linear-algebraic form of convex separation and are reused across optimization, game theory and polyhedral combinatorics.

All results of this mission are classical and proved in the literature. What the mission adds is a machine-checked version of them in one fixed linear-programming form, the inequality form with explicit slacks used throughout Vanderbei's book. Weak duality, strong duality and complementary slackness are already formalized on this platform for other forms (Bertsimas–Tsitsiklis's general form, a minimization, and a covering pair with the roles of primal and dual exchanged). Those statements are equivalent to the ones here only after a transformation (negating the objective, swapping primal and dual), so they are not the same theorems. The Farkas variant for inequality systems, the separation theorem for two polyhedra, and both strict complementarity theorems have no formal counterpart on the platform.

Difficulty

Weak duality and the converse direction of complementary slackness are short computations. The substance lies elsewhere. Strong duality and Farkas' Lemma require a genuine existence argument; the book obtains them from the simplex method, whose termination is itself a nontrivial fact, and any other route needs an independent theorem of the alternative.

For strict complementarity the obvious attempt fails. Complementary slackness gives, for each optimal pair, only that one member of each complementary pair vanishes; nothing in a single optimal basic solution forces the other member to be positive, and in degenerate problems every basic optimal pair can fail strictness. A strictly complementary pair is in general not a vertex of either optimal face, so it cannot be found by inspecting basic solutions. The goal also asks for more than Theorem 10.6: the positivity must be achieved within the optimal sets, which are faces cut out by an additional objective-level constraint, so the feasible-solution argument does not transfer verbatim.

Formalization scope

Vectors are Fin n → ℝ and Fin m → ℝ; the constraint matrix is Matrix (Fin m) (Fin n) ℝ; m and n are arbitrary natural numbers, including zero. The slacks are functions of the solution (primalSlack A b x = b - A *ᵥ x, dualSlack A c y = Aᵀ *ᵥ y - c), never free variables, so a "solution (x,w)(x, w)(x,w)" of the book is the vector xxx with the slack it determines. Optimality is attainment of the maximum (minimum) over the feasible set; no supremum, value function or extended reals are involved. The strict inequality ξ>0\xi > 0ξ>0 is written componentwise as ∀ j, 0 < x j + dualSlack A c y j and ∀ i, 0 < y i + primalSlack A b x i.

A halfspace carries a nonzero normal vector. Without that requirement the empty set would be a halfspace and the separation theorem would be trivial; the formal definition rules this out.

The book states every result in this mission with its hypotheses explicit, and none of them asserts the existence of an unspecified constant, so no explicit-constant instantiation was needed. The remark after Theorem 10.7 refers to "the complementary slackness theorem (Theorem 5.1)"; the complementary slackness theorem is Theorem 5.3, and the milestones follow the theorem numbering.

A complete development needs a theorem of the alternative for real linear inequality systems (Mathlib has Farkas-type results for cones and the geometric Hahn–Banach theorem, but no ready-made matrix version of Lemma 10.5) and elementary convex-combination arguments on feasible sets. The definitions of this mission are self-contained and reusable for any later chapter that works in Vanderbei's inequality form. Proofs of any milestone are welcome, as are proofs that avoid the simplex method.

Selected references

  • R. J. Vanderbei, Linear Programming: Foundations and Extensions, 4th ed., International Series in Operations Research & Management Science 196, Springer, 2014. doi:10.1007/978-1-4614-7630-6
  • J. Farkas, "Theorie der einfachen Ungleichungen", Journal für die reine und angewandte Mathematik 124 (1902), 1–27. doi:10.1515/crll.1902.124.1
  • A. J. Goldman and A. W. Tucker, "Theory of linear programming", in Linear Inequalities and Related Systems, Annals of Mathematics Studies 38, Princeton University Press, 1956, 53–97.
  • D. Gale, H. W. Kuhn and A. W. Tucker, "Linear programming and the theory of games", in Activity Analysis of Production and Allocation, Wiley, 1951, 317–329.
9 thms3 active usersReviewed
🏆Completed
Convex OptimizationLinear OptimizationOptimization·Captain: mikedeng1

Minimization Methods for Non-Differentiable Functions XI: Convexity and Subgradients of the Value Function in Decomposition with Respect to VariablesTextbook

Motivation

Large convex programs often have a block structure: a small set of "complicating" variables xxx couples otherwise separate subproblems in the remaining variables yyy. Decomposition with respect to variables fixes xxx, solves the subproblem in yyy, and treats the optimal subproblem value as a function of xxx alone. The outer problem in xxx is then small but nonsmooth, because the optimal value of a constrained program is generally not differentiable in its parameters. Shor's Chapter 4 (Shor 1985, Ch. 4) presents this reduction as a principal application of subgradient methods: once a subgradient of the outer function can be read off from the subproblem, the methods of Chapters 2–3 apply directly. The same construction underlies Benders decomposition (Benders 1962) and its convex generalization (Geoffrion 1972), and parametric decomposition schemes for linear programs.

The chapter's other numbered results serve the same programme from the dual side: the Lagrangian dual function of a program over a compact set gives a lower bound usable in branch and bound (Theorem 4.3), exact nonsmooth penalty functions turn a constrained convex program into one unconstrained nonsmooth minimization (Theorem 4.2; nonsmooth penalties were first studied systematically by I. I. Eremin, 1967), and a stochastic transportation model is shown to be a convex program before being solved through its dual (Lemma 4.3).

Setting

The variables split into x∈Elxx \in E^x_lx∈Elx​ and y∈Emyy \in E^y_my∈Emy​ (Euclidean spaces, inner product (⋅,⋅)(\cdot,\cdot)(⋅,⋅)). The problem is

min⁡x,yf0(x,y)s.t.fi(x,y)≤0,i=1,…,n,(4.1)–(4.2)\min_{x,y} f_0(x,y) \quad \text{s.t.} \quad f_i(x,y) \le 0,\quad i = 1,\dots,n, \qquad (4.1)\text{–}(4.2)x,ymin​f0​(x,y)s.t.fi​(x,y)≤0,i=1,…,n,(4.1)–(4.2)

with f0,f1,…,fnf_0, f_1, \dots, f_nf0​,f1​,…,fn​ convex functions of z=(x,y)z = (x,y)z=(x,y) (jointly convex), finite everywhere. For a fixed xˉ\bar xxˉ, the subproblem (4.3)–(4.4) is min⁡y∈D(xˉ)f0(xˉ,y)\min_{y \in D(\bar x)} f_0(\bar x, y)miny∈D(xˉ)​f0​(xˉ,y) with D(xˉ)={y:fi(xˉ,y)≤0}D(\bar x) = \{y : f_i(\bar x,y) \le 0\}D(xˉ)={y:fi​(xˉ,y)≤0}. Where it has a solution y(xˉ)y(\bar x)y(xˉ), the value function is

Φ(xˉ)=min⁡y∈D(xˉ)f0(xˉ,y).(4.5)\Phi(\bar x) = \min_{y \in D(\bar x)} f_0(\bar x,y). \qquad (4.5)Φ(xˉ)=y∈D(xˉ)min​f0​(xˉ,y).(4.5)

The Slater condition at xˉ\bar xxˉ asks for a yyy with fi(xˉ,y)<0f_i(\bar x,y) < 0fi​(xˉ,y)<0 for all iii. The Lagrange function is LU(x,y)=f0(x,y)+∑iUifi(x,y)L_U(x,y) = f_0(x,y) + \sum_i U_i f_i(x,y)LU​(x,y)=f0​(x,y)+∑i​Ui​fi​(x,y), and U≥0U \ge 0U≥0 are Kuhn–Tucker multipliers at xˉ\bar xxˉ when Φ(xˉ)=min⁡yLU(xˉ,y)\Phi(\bar x) = \min_y L_U(\bar x,y)Φ(xˉ)=miny​LU​(xˉ,y). A subgradient of a function of (x,y)(x,y)(x,y) is written through its projections (gx,gy)(g^x, g^y)(gx,gy) on the two blocks; a subgradient of Φ\PhiΦ at xˉ\bar xxˉ is a ggg with Φ(x)−Φ(xˉ)≥(x−xˉ,g)\Phi(x) - \Phi(\bar x) \ge (x - \bar x, g)Φ(x)−Φ(xˉ)≥(x−xˉ,g).

Formalization targets

Goal: Theorem 4.1 (p. 94)

If WWW is a convex set of xxx-values at which the subproblem has a solution, then Φ\PhiΦ is convex on WWW; and if xˉ∈W\bar x \in Wxˉ∈W satisfies the Slater condition, then for every optimal y(xˉ)y(\bar x)y(xˉ), multipliers UUU exist, LUL_ULU​ has a subgradient at (xˉ,y(xˉ))(\bar x, y(\bar x))(xˉ,y(xˉ)) with vanishing yyy-projection, and the xxx-projection of any such subgradient satisfies

gΦ(xˉ)=gLUx(xˉ,y(xˉ))∈∂Φ(xˉ).(4.6)g_\Phi(\bar x) = g^x_{L_U}(\bar x, y(\bar x)) \in \partial \Phi(\bar x). \qquad (4.6)gΦ​(xˉ)=gLU​x​(xˉ,y(xˉ))∈∂Φ(xˉ).(4.6)

Milestones for the goal

  1. Convexity of Φ\PhiΦ on WWW (Theorem 4.1, first assertion).
  2. Existence of Kuhn–Tucker multipliers for the subproblem under Slater (p. 95, display).
  3. Existence of a subgradient of LUL_ULU​ with vanishing yyy-projection (p. 95).
  4. Formula (4.6) for a given multiplier vector and such a subgradient (p. 95, final display).

Further results of the chapter

  1. Corollary (4.7): if each fα(x,⋅)f_\alpha(x,\cdot)fα​(x,⋅) is continuously differentiable in yyy, then gf0x+∑iUigfixg^x_{f_0} + \sum_i U_i g^x_{f_i}gf0​x​+∑i​Ui​gfi​x​, built from arbitrary subgradients of the fαf_\alphafα​, is a subgradient of Φ\PhiΦ.
  2. Lemma 4.3: the stochastic transportation problem (4.163)–(4.165) is a convex program.
  3. Theorem 4.2: with nonsmooth penalties pip_ipi​ whose slopes ci=lim⁡t→0+pi(t)/tc_i = \lim_{t\to0+} p_i(t)/tci​=limt→0+​pi​(t)/t exceed a Lagrange multiplier vector yˉ\bar yyˉ​, the minimizers of S=f0+∑pi∘fiS = f_0 + \sum p_i \circ f_iS=f0​+∑pi​∘fi​ are exactly the solutions of the constrained program; and if a minimizer of SSS solves the program, some multiplier vector satisfies yˉ≤c\bar y \le cyˉ​≤c.
  4. Theorem 4.3: for Φ(u)=min⁡x∈X[f0+∑uifi]\Phi(u) = \min_{x\in X}[f_0 + \sum u_i f_i]Φ(u)=minx∈X​[f0​+∑ui​fi​] over a compact XXX, Q=max⁡u≥0Φ(u)≤f∗Q = \max_{u\ge0}\Phi(u) \le f^*Q=maxu≥0​Φ(u)≤f∗.

Significance

The result itself. Theorem 4.1 is what makes decomposition with respect to variables an instance of convex nonsmooth minimization: the outer problem is convex, and one subproblem solve returns both Φ(xˉ)\Phi(\bar x)Φ(xˉ) and a subgradient. The algorithm on p. 96 — solve the subproblem at xkx_kxk​, form gΦ(xk)g_\Phi(x_k)gΦ​(xk​) by (4.6) or (4.7), take a subgradient step — is exactly this, and the step-size theory of Chapter 2 then gives convergence. The Corollary is the version used in practice for linear and quadratic subproblems, where multipliers and partial subgradients are computed directly. Theorem 4.2 justifies replacing constraints by nonsmooth penalties of finite slope, and Theorem 4.3 is the weak-duality bound behind Lagrangian relaxation in branch and bound.

Formalizing it. All results are classical and proved in the book; none is formalized as stated here. The platform already has related statements with different shapes: convexity of the perturbation value function in the constraint right-hand side (VectorSpaceOpt.perturbationValue_convex, Luenberger), Slater strong duality over the whole space (ConvexOptimization.slater_strong_duality), weak duality with an unconstrained domain (ConvexOptimization.weak_duality), weak Lagrangean duality for integer programs (LinearOptimization.integer_program_weak_lagrangean_duality), and the LP special case of convexity of the optimal cost (LinearOptimization.lp_optimal_cost_convex_in_rhs). This mission adds the partial-minimization form in which one block of variables is minimized out under joint convexity, its subgradient calculus, exact nonsmooth penalties, and weak duality over a compact domain.

Difficulty

Convexity of Φ\PhiΦ is elementary once the minimum is attained. The subgradient formula is where the obvious argument fails: an arbitrary subgradient of LUL_ULU​ at (xˉ,y(xˉ))(\bar x, y(\bar x))(xˉ,y(xˉ)) does not project to a subgradient of Φ\PhiΦ, because its yyy-projection contributes a term (y(x)−y(xˉ),gy)(y(x) - y(\bar x), g^y)(y(x)−y(xˉ),gy) of unknown sign. The theorem needs a subgradient whose yyy-projection vanishes, and its existence is a separate fact about partial minimization of a jointly convex, everywhere-finite function. The multipliers come from the Kuhn–Tucker theorem for the subproblem, which requires the Slater condition. In the Corollary, the difficulty is to show that differentiability in yyy forces the yyy-projection of any combination gf0+∑Uigfig_{f_0} + \sum U_i g_{f_i}gf0​​+∑Ui​gfi​​ to vanish. In Theorem 4.2 the necessity part needs a subdifferential chain rule for pi∘fip_i \circ f_ipi​∘fi​.

Formalization scope

  • ElxE^x_lElx​, EmyE^y_mEmy​ and ENE_NEN​ are EuclideanSpace ℝ (Fin l), EuclideanSpace ℝ (Fin m), EuclideanSpace ℝ (Fin N); constraints are indexed by Fin n (or Fin m). All functions are real-valued and finite everywhere; joint convexity is convexity on the product Elx×EmyE^x_l \times E^y_mElx​×Emy​.
  • Φ\PhiΦ is a real infimum over D(x)D(x)D(x). It is the book's minimum wherever the minimum is attained, and every statement assumes attainment at each point of WWW. A statement about Φ\PhiΦ at points where the subproblem has no solution would be about Lean's default value 000, and is ruled out by these hypotheses.
  • "Convex on some convex subset WWW of EnE_nEn​" is read as convexity on every convex WWW on which Φ\PhiΦ is defined. A formalization quantifying over a single unspecified WWW (e.g. a singleton) would be trivially true.
  • Formula (4.6) is stated for subgradients of LUL_ULU​ whose yyy-projection is zero, as the book's proof uses it; the subgradient inequality for Φ\PhiΦ is stated on all of WWW.
  • Kuhn–Tucker multipliers relative to an optimal yˉ\bar yyˉ​: U≥0U \ge 0U≥0, Uifi(xˉ,yˉ)=0U_i f_i(\bar x,\bar y) = 0Ui​fi​(xˉ,yˉ​)=0, and yˉ\bar yyˉ​ minimizes LU(xˉ,⋅)L_U(\bar x,\cdot)LU​(xˉ,⋅) over all yyy.
  • Theorem 4.2: a Lagrange multiplier vector is yˉ≥0\bar y \ge 0yˉ​≥0 with f0+∑yˉifi≥f∗f_0 + \sum\bar y_i f_i \ge f^*f0​+∑yˉ​i​fi​≥f∗ everywhere, f∗f^*f∗ the finite optimal value; the necessity clause is read as "for some multiplier vector" (for all multiplier vectors it is false); the limits cic_ici​ are given as hypotheses.
  • Theorem 4.3: the minimum (4.187) attained for every u≥0u \ge 0u≥0 (the book's "min"; no continuity assumed); f∗f^*f∗ attained; QQQ attained as the book's "max" presupposes.
  • Lemma 4.3: demands with densities and finite mean, penalty coefficients rj≥0r_j \ge 0rj​≥0.

Useful infrastructure beyond this mission: partial minimization of jointly convex functions, subgradients on product spaces, and a Kuhn–Tucker saddle-point theorem for convex programs with inequality constraints. Proofs of any milestone, and reusable lemmas for these, are welcome.

Selected references

  • N. Z. Shor, Minimization Methods for Non-Differentiable Functions, Springer Series in Computational Mathematics 3, Springer, 1985, Ch. 4, pp. 93–96, 131–133, 146–148. https://doi.org/10.1007/978-3-642-82118-9
  • J. F. Benders, Partitioning procedures for solving mixed-variables programming problems, Numerische Mathematik 4, 1962, 238–252. https://doi.org/10.1007/BF01386316
  • A. M. Geoffrion, Generalized Benders decomposition, Journal of Optimization Theory and Applications 10, 1972, 237–260. https://doi.org/10.1007/BF00934810
  • I. I. Eremin, The penalty method in convex programming, Soviet Mathematics Doklady 8, 1967, 459–462 (Shor's reference [22]).
  • R. T. Rockafellar, Convex Analysis, Princeton University Press, 1970, §29 (partial minimization and perturbation functions). https://doi.org/10.1515/9781400873173
12 thms3 active usersReviewed
🏆Completed
Convex OptimizationNumerical AnalysisOptimization·Captain: mikedeng1

Minimization Methods for Non-Differentiable Functions X: The Space-Dilation Ellipsoid Method Localizes the Solution in Ellipsoids Shrinking by the Ratio q_nTextbook

Motivation

The ellipsoid method is the algorithm that settled the polynomial-time solvability of linear programming (Khachiyan, 1979) and that underlies the equivalence of separation and optimization in combinatorial optimization (Grötschel, Lovász and Schrijver, 1981). Its origin is in nonsmooth convex optimization. In 1976 Yudin and Nemirovskii proposed a modified method of centered sections that localizes an optimum inside a sequence of ellipsoids, and in 1977 N. Z. Shor observed independently that the same scheme is a subgradient method with space dilation along the gradient, the family of methods he had developed since 1969. Section 3.8 of Shor's monograph Minimization Methods for Non-Differentiable Functions (Springer 1985) presents the method in this second form and proves its basic localization property.

Timeline:

  • 1965. A. Yu. Levin proposes the method of centered sections (cuts through the center of gravity of a polyhedron); each cut removes at least a fixed fraction of the volume, but computing centers of gravity is impractical for n>3n > 3n>3.
  • 1969–1972. Shor introduces subgradient methods with space dilation along the gradient (SDG methods).
  • 1976. Yudin and Nemirovskii replace the polyhedron by a minimal ellipsoid containing a half-ellipsoid, obtaining a geometric volume decrease depending only on the dimension ([Yudin–Nemirovskii 1976]).
  • 1977. Shor shows that the same method is an SDG algorithm with coefficient β=(n−1)/(n+1)\beta = \sqrt{(n-1)/(n+1)}β=(n−1)/(n+1)​ ([Shor 1977]).
  • 1979. Khachiyan applies the method to linear inequalities with integer data, obtaining the first polynomial-time algorithm for linear programming ([Khachiyan 1979]).

Setting

Let EnE_nEn​ be nnn-dimensional Euclidean space with inner product (x,y)(x, y)(x,y), and n>1n > 1n>1. For a unit vector ξ\xiξ and a number α\alphaα, the operator of space dilation along ξ\xiξ with coefficient α\alphaα is Rα(ξ)=I+(α−1)ξξTR_\alpha(\xi) = I + (\alpha - 1)\xi\xi^TRα​(ξ)=I+(α−1)ξξT: it multiplies the component of a vector along ξ\xiξ by α\alphaα and leaves the orthogonal component unchanged.

Let g:En→Eng : E_n \to E_ng:En​→En​ be a vector field, not necessarily continuous. The problem is to find a point x∗x^*x∗ with

(g(x),x−x∗)≥0for all x∈En,(g(x), x - x^*) \ge 0 \quad \text{for all } x \in E_n,(g(x),x−x∗)≥0for all x∈En​,

where it is known that such an x∗x^*x∗ exists in the closed ball S(x0,R)S(x_0, R)S(x0​,R) of radius R>0R > 0R>0 about a given point x0x_0x0​. Put β=(n−1)/(n+1)\beta = \sqrt{(n-1)/(n+1)}β=(n−1)/(n+1)​ and r=n/n2−1r = n/\sqrt{n^2-1}r=n/n2−1​. The algorithm (3.57)–(3.60) starts from x0x_0x0​, B0=InB_0 = I_nB0​=In​, h0=R/(n+1)h_0 = R/(n+1)h0​=R/(n+1), and at iteration k+1k+1k+1 stops if g(xk)=0g(x_k) = 0g(xk​)=0, and otherwise sets

ξk=BkTg(xk)∥BkTg(xk)∥,xk+1=xk−hkBkξk,Bk+1=BkRβ(ξk),hk+1=rhk.\xi_k = \frac{B_k^T g(x_k)}{\|B_k^T g(x_k)\|}, \quad x_{k+1} = x_k - h_k B_k \xi_k, \quad B_{k+1} = B_k R_\beta(\xi_k), \quad h_{k+1} = r h_k .ξk​=∥BkT​g(xk​)∥BkT​g(xk​)​,xk+1​=xk​−hk​Bk​ξk​,Bk+1​=Bk​Rβ​(ξk​),hk+1​=rhk​.

With Ak=Bk−1A_k = B_k^{-1}Ak​=Bk−1​, the localizing ellipsoid is Φk={x:∥Ak(x−xk)∥≤(n+1)hk}\Phi_k = \{x : \|A_k(x - x_k)\| \le (n+1)h_k\}Φk​={x:∥Ak​(x−xk​)∥≤(n+1)hk​}, and the dimension-dependent ratio is

qn=n−1n+1(nn2−1)n<1.q_n = \sqrt{\frac{n-1}{n+1}}\left(\frac{n}{\sqrt{n^2-1}}\right)^n < 1 .qn​=n+1n−1​​(n2−1​n​)n<1.

Three problems produce such a field: minimizing a convex fff on a ball (the field (3.62), a subgradient inside the ball and the outward radial direction outside); the convex program min⁡f0\min f_0minf0​ s.t. fi≤0f_i \le 0fi​≤0 (the field (3.65), a subgradient of the objective at feasible points and of a most violated constraint otherwise); and a convex–concave saddle point problem (the field {gfx,−gfy}\{g_f^x, -g_f^y\}{gfx​,−gfy​}).

Formalization targets

Goal: Theorem 3.14 (p. 86)

For every kkk,

∥Ak(xk−x∗)∥≤hk(n+1),(3.61)\|A_k(x_k - x^*)\| \le h_k (n+1), \tag{3.61}∥Ak​(xk​−x∗)∥≤hk​(n+1),(3.61)

that is, x∗∈Φkx^* \in \Phi_kx∗∈Φk​. The statement holds for every field ggg satisfying the monotonicity condition at x∗x^*x∗; nothing about continuity or convexity is assumed.

Milestones

  1. Eq. (3.4): ∥Rα(ξ)x∥=∥x∥2+(α2−1)(x,ξ)2\|R_\alpha(\xi)x\| = \sqrt{\|x\|^2 + (\alpha^2-1)(x,\xi)^2}∥Rα​(ξ)x∥=∥x∥2+(α2−1)(x,ξ)2​ for unit ξ\xiξ.
  2. Volume of Φk\Phi_kΦk​ (p. 87): (n+1)hk=Rrk(n+1)h_k = R r^k(n+1)hk​=Rrk and v(Φk)=v0Rnrnk/det⁡Akv(\Phi_k) = v_0 R^n r^{nk}/\det A_kv(Φk​)=v0​Rnrnk/detAk​, v0v_0v0​ the volume of the unit ball.
  3. Volume ratio (p. 87–88): v(Φk+1)=qn v(Φk)v(\Phi_{k+1}) = q_n\, v(\Phi_k)v(Φk+1​)=qn​v(Φk​) with qn<1q_n < 1qn​<1, the volumes being positive and finite.
  4. Eq. (3.62): the ball field satisfies (g(x),x−x∗)≥0(g(x), x - x^*) \ge 0(g(x),x−x∗)≥0.
  5. Eq. (3.65): the convex-programming field satisfies (g(x),x−x∗)≥0(g(x), x - x^*) \ge 0(g(x),x−x∗)≥0.
  6. Saddle point field (p. 90): (g(z),z−z∗)≥0(g(z), z - z^*) \ge 0(g(z),z−z∗)≥0.

Significance

The result. Theorem 3.14 with the volume identity says that after kkk steps the solution is confined to an ellipsoid of volume qnkq_n^kqnk​ times that of the initial ball, for any field of the above kind. Milestones 4–6 turn this into localization guarantees for constrained convex minimization, general convex programming and convex–concave saddle points, with a rate that depends only on the dimension. The same localization underlies the complexity bounds of the ellipsoid method for linear programming and the polynomial equivalence of separation and optimization.

Formalizing it. The results are classical and proved in the book. No machine-checked proof of the space-dilation form of the method is known to exist. The platform already contains a proved version of the Bertsimas–Tsitsiklis form (LinearOptimization.ellipsoid_update_halfspace_subset, LinearOptimization.ellipsoid_update_volume_lt: a half-ellipsoid E(z,D)∩{aTx≥aTz}E(z, D) \cap \{a^Tx \ge a^Tz\}E(z,D)∩{aTx≥aTz} is covered by an updated ellipsoid whose volume is smaller by a factor below e−1/(2(n+1))e^{-1/(2(n+1))}e−1/(2(n+1))), and Khachiyan's feasibility algorithm (SmaleNinth.khachiyan_ellipsoid_decides). Those statements are about a center/shape-matrix update and give a volume inequality; this mission is about the iterates of Shor's matrix recursion Bk+1=BkRβ(ξk)B_{k+1} = B_k R_\beta(\xi_k)Bk+1​=Bk​Rβ​(ξk​) and the exact ratio qnq_nqn​. Relating the two parametrizations (Dk=(n+1)2hk2BkBkTD_k = (n+1)^2 h_k^2 B_k B_k^TDk​=(n+1)2hk2​Bk​BkT​) is a welcome side result.

Difficulty

The obvious approach, tracking the ellipsoid through the center and shape matrix and invoking a minimum-volume covering argument, is not what the algorithm computes: here the iterate is updated through the factor BkB_kBk​ and the stepsize hkh_khk​ is fixed in advance, independent of the field, so the induction must be carried out in the transformed coordinates zk=Ak(xk−x∗)z_k = A_k(x_k - x^*)zk​=Ak​(xk​−x∗) in which the ellipsoid is a ball. The difficulty is that the monotonicity condition gives only the sign of one inner product, (zk,ξk)≥0(z_k, \xi_k) \ge 0(zk​,ξk​)≥0, while the norm of zk+1z_{k+1}zk+1​ depends on both (zk,ξk)(z_k,\xi_k)(zk​,ξk​) and ∥zk∥\|z_k\|∥zk​∥; the constants β\betaβ and rrr are exactly those for which the resulting quadratic estimate closes. For the volume identity, the main technical step is computing det⁡Rβ(ξ)=β\det R_\beta(\xi) = \betadetRβ​(ξ)=β and the Lebesgue measure of a linear image of a ball in EuclideanSpace.

Formalization scope

  • EnE_nEn​ is EuclideanSpace ℝ (Fin n); matrices act through Matrix.toEuclideanLin; Bk∗B_k^*Bk∗​ is the transpose. Rα(ξ)R_\alpha(\xi)Rα​(ξ) is the matrix I+(α−1)ξξTI + (\alpha-1)\xi\xi^TI+(α−1)ξξT (the book's property 10); for unit ξ\xiξ this is the operator of the book's definition.
  • The algorithm is the definition ellipsoidMethod g R x₀ : ℕ → EllState n, with state (xk,Bk,hk)(x_k, B_k, h_k)(xk​,Bk​,hk​), B0=IB_0 = IB0​=I, h0=R/(n+1)h_0 = R/(n+1)h0​=R/(n+1). If g(xk)=0g(x_k) = 0g(xk​)=0 the state is repeated from then on (the book stops); the normalization in (3.57) is only performed when g(xk)≠0g(x_k) \ne 0g(xk​)=0. AkA_kAk​ is the matrix inverse of BkB_kBk​, which is nonsingular.
  • Theorem 3.14 is stated for n>1n > 1n>1, R>0R > 0R>0, x∗x^*x∗ with ∥x0−x∗∥≤R\|x_0 - x^*\| \le R∥x0​−x∗∥≤R and (g(x),x−x∗)≥0(g(x), x - x^*) \ge 0(g(x),x−x∗)≥0 for all xxx, for every kkk. The book's additional assumption that g(x)≠0g(x) \ne 0g(x)=0 for x≠x∗x \ne x^*x=x∗ is not used by its proof and is omitted. The iterates are those of the recursion; a statement about an arbitrary ellipsoid containing x∗x^*x∗, or about an arbitrary invertible matrix in place of BkB_kBk​, would not be this theorem and is ruled out by the definitions.
  • Volumes are Lebesgue measure in ℝ≥0∞. The ellipsoid is given for an arbitrary center (the book writes x∗x^*x∗ in one place and xkx_kxk​ in another; the volume is the same). The volume formula requires the first kkk iterations to have been performed (g(xj)≠0g(x_j) \ne 0g(xj​)=0, j<kj < kj<k); the ratio requires iteration k+1k+1k+1 to be performed.
  • The printed chain on p. 87 has misprints (exponents 222 and 111 on n/n2−1n/\sqrt{n^2-1}n/n2−1​ where nnn is meant, and (n−1)/(n+1)(n-1)/(n+1)(n−1)/(n+1) for (n−1)/(n+1)\sqrt{(n-1)/(n+1)}(n−1)/(n+1)​); the statement follows the value of qnq_nqn​ given on p. 88. The estimate for x∉S(x0,R)x \notin S(x_0,R)x∈/S(x0​,R) before (3.62) has a sign misprint; only the conclusion is stated.
  • Needed infrastructure: determinant of a rank-one perturbation of the identity (det⁡(I+c ξξT)=1+c∥ξ∥2\det(I + c\,\xi\xi^T) = 1 + c\|\xi\|^2det(I+cξξT)=1+c∥ξ∥2, available in Mathlib as the matrix determinant lemma), the measure of a linear image (MeasureTheory.Measure.addHaar_image_linearMap), and elementary real inequalities for qn<1q_n < 1qn​<1. The space-dilation lemmas are reusable in the other Shor missions on SDG methods and the rrr-algorithm.

Selected references

  • N. Z. Shor, Minimization Methods for Non-Differentiable Functions, Springer Series in Computational Mathematics 3, Springer, 1985, §3.8. https://doi.org/10.1007/978-3-642-82118-9
  • D. B. Yudin and A. S. Nemirovskii, Informational complexity and efficient methods for the solution of convex extremal problems, Ekonomika i Matematicheskie Metody 12 (1976), 357–369 (English translation: Matekon 13 (1977), 25–45).
  • N. Z. Shor, Cut-off method with space extension in convex programming problems, Cybernetics 13 (1977), 94–96.
  • L. G. Khachiyan, A polynomial algorithm in linear programming, Soviet Mathematics Doklady 20 (1979), 191–194.
  • R. G. Bland, D. Goldfarb and M. J. Todd, The ellipsoid method: a survey, Operations Research 29 (1981), 1039–1091. https://doi.org/10.1287/opre.29.6.1039
  • M. Grötschel, L. Lovász and A. Schrijver, Geometric Algorithms and Combinatorial Optimization, Springer, 1988. https://doi.org/10.1007/978-3-642-97881-4
8 thms3 active usersReviewed
🏆Completed
Convex OptimizationOptimization·Captain: mikedeng1

Minimization Methods for Non-Differentiable Functions III: Convergence of the Normalized Subgradient Method with Divergent-Series StepsizesTextbook

Motivation

Many optimization problems of operations research have objectives that are convex but not differentiable: Lagrangian duals of integer and combinatorial programs, maxima of finitely many affine or smooth functions, penalty functions for systems of inequalities, and the value functions produced by decomposition. For such functions the gradient method and steepest descent fail. Constant steps cannot work because the subgradients need not tend to zero at a nondifferentiable minimum, and exact line search along the negative gradient can converge to a point that is not a minimizer (the example on pp. 22–23 of the source).

The subgradient method replaces the gradient by an arbitrary subgradient and gives up monotone decrease of the objective. Its convergence theory is the foundation of nondifferentiable optimization and of Lagrangian relaxation in integer programming.

Timeline. N. Z. Shor proposed the method with normalized steps in 1962 (Kiev). Yu. M. Ermoliev proved convergence in finite dimensions with divergent-series stepsizes (Kibernetika, 1966), and B. T. Polyak proved it for constrained problems in Hilbert space (Doklady Akad. Nauk SSSR, 1967). Held, Wolfe and Crowder (Mathematical Programming, 1974) brought the method to large combinatorial problems through Lagrangian relaxation. This mission formalizes the exposition of Section 2.1–2.2 of Shor's monograph (Springer, 1985), which gives self-contained proofs of these results.

Setting

Let EnE_nEn​ be the nnn-dimensional Euclidean space with inner product (x,y)(x, y)(x,y) and norm ∥x∥\|x\|∥x∥. Let f:En→Rf : E_n \to \mathbb{R}f:En​→R be a convex function finite everywhere. A vector ggg is a subgradient of fff at x0x_0x0​ if

f(x)−f(x0)≥(g,x−x0)for all x∈En.f(x) - f(x_0) \ge (g, x - x_0) \quad \text{for all } x \in E_n.f(x)−f(x0​)≥(g,x−x0​)for all x∈En​.

Every convex fff has at least one subgradient at every point. Let M∗={x:f(x)≤f(y) ∀y}M^* = \{x : f(x) \le f(y) \ \forall y\}M∗={x:f(x)≤f(y) ∀y} be the set of minimum points and, when it is nonempty, f∗=min⁡ff^* = \min ff∗=minf.

A subgradient selection gfg_fgf​ assigns to each xxx some subgradient gf(x)g_f(x)gf​(x) of fff at xxx. No particular choice is made: every result holds for every selection. Given stepsizes h1,h2,⋯>0h_1, h_2, \dots > 0h1​,h2​,⋯>0 and a starting point x0x_0x0​, the normalized subgradient method is

xk+1=xk−hk+1 gf(xk)∥gf(xk)∥,k=0,1,…(2.4)x_{k+1} = x_k - h_{k+1}\, \frac{g_f(x_k)}{\|g_f(x_k)\|}, \qquad k = 0, 1, \dots \tag{2.4}xk+1​=xk​−hk+1​∥gf​(xk​)∥gf​(xk​)​,k=0,1,…(2.4)

If gf(xk)=0g_f(x_k) = 0gf​(xk​)=0, then xkx_kxk​ is a minimizer and the computation stops. The unnormalized method is xk+1=xk−hk+1gf(xk)x_{k+1} = x_k - h_{k+1} g_f(x_k)xk+1​=xk​−hk+1​gf​(xk​) (2.5), and the method with restarts takes that step when hk+1∥gf(xk)∥≤ch_{k+1}\|g_f(x_k)\| \le chk+1​∥gf​(xk​)∥≤c and returns to x0x_0x0​ otherwise.

Formalization targets

Goal: Theorem 2.2 (p. 25)

If M∗M^*M∗ is nonempty and bounded, hk>0h_k > 0hk​>0, hk→0h_k \to 0hk​→0 and ∑k≥1hk=+∞\sum_{k \ge 1} h_k = +\infty∑k≥1​hk​=+∞, then for every x0x_0x0​ and every subgradient selection, the method (2.4) either reaches M∗M^*M∗ at some index kˉ\bar kkˉ or

lim⁡k→∞min⁡y∈M∗∥xk−y∥=0,lim⁡k→∞f(xk)=f∗.\lim_{k \to \infty} \min_{y \in M^*} \|x_k - y\| = 0, \qquad \lim_{k \to \infty} f(x_k) = f^*.k→∞lim​y∈M∗min​∥xk​−y∥=0,k→∞lim​f(xk​)=f∗.

Milestones

  1. Eq. (2.3), the one-step inequality ∥xk+1−x∗∥2≤∥xk−x∗∥2+h2−2h ρ(x∗,Uk)\|x_{k+1} - x^*\|^2 \le \|x_k - x^*\|^2 + h^2 - 2h\,\rho(x^*, U_k)∥xk+1​−x∗∥2≤∥xk​−x∗∥2+h2−2hρ(x∗,Uk​), where Uk={x:f(x)=f(xk)}U_k = \{x : f(x) = f(x_k)\}Uk​={x:f(x)=f(xk​)}.
  2. Theorem 2.1: with constant step length hhh, some level surface {f=f(xk∗)}\{f = f(x_{k^*})\}{f=f(xk∗​)} passes within h(1+ε)/2h(1+\varepsilon)/2h(1+ε)/2 of any x∗∈M∗x^* \in M^*x∗∈M∗.
  3. Corollaries 1 and 2: a suitable constant step length yields a subsequence with f(xki)−f∗<δf(x_{k_i}) - f^* < \deltaf(xki​​)−f∗<δ. If M∗M^*M∗ contains a ball of radius r>h/2r > h/2r>h/2, the method terminates in M∗M^*M∗.
  4. Theorem 2.5: if M∗M^*M∗ contains a ball of radius rrr, ∑hk=∞\sum h_k = \infty∑hk​=∞ and lim sup⁡hk<2r\limsup h_k < 2rlimsuphk​<2r, then (2.4) terminates in M∗M^*M∗.
  5. Theorem 2.3: for the unnormalized method (2.5), bounded subgradients along the trajectory imply convergence, and unbounded subgradients rule it out.
  6. Theorem 2.4: the method with restarts converges for every c>0c > 0c>0.

Significance

Theorem 2.2 is the basic convergence guarantee for first-order methods on general nonsmooth convex functions. It needs no Lipschitz constant, no bound on the subgradients and no smoothness: normalizing the step makes the step length independent of the size of the subgradient. The divergent-series rule hk→0h_k \to 0hk​→0, ∑hk=∞\sum h_k = \infty∑hk​=∞ is the standard stepsize condition of stochastic approximation and of Lagrangian relaxation codes. Theorems 2.3–2.5 mark its boundaries. The unnormalized method needs bounded subgradients (Theorem 2.3), restarts remove that need (Theorem 2.4), and a solution set with nonempty interior gives finite termination (Theorem 2.5). The last result is the basis of the finite methods for systems of convex inequalities and for the dual of an assignment problem with a unique solution (pp. 28–29).

All of these results are classical and proved in the source. None of them is formalized in Lean's Mathlib. The platform has neighbouring results that are not the same statements: Poljak's divergent-series theorem for concave piecewise-linear maximization (in Validation of Subgradient Optimization I), and rate bounds for Lipschitz objectives (First-Order and Stochastic Optimization Methods for ML II, Understanding Machine Learning X). This mission adds the general convex case with normalized steps, the dichotomy for unnormalized steps, and finite termination.

Difficulty

The standard rate analysis of the subgradient method bounds ∥xk+1−x∗∥2−∥xk−x∗∥2\|x_{k+1} - x^*\|^2 - \|x_k - x^*\|^2∥xk+1​−x∗∥2−∥xk​−x∗∥2 by −2hk+1(f(xk)−f∗)/∥gf(xk)∥+hk+12-2h_{k+1}(f(x_k) - f^*)/\|g_f(x_k)\| + h_{k+1}^2−2hk+1​(f(xk​)−f∗)/∥gf​(xk​)∥+hk+12​. It then needs a uniform bound on ∥gf(xk)∥\|g_f(x_k)\|∥gf​(xk​)∥, which is exactly what is not assumed here. Nothing a priori keeps the iterates in a bounded set, and the subgradients of a general convex function (for instance f(x)=x4f(x) = x^4f(x)=x4, the source's example on p. 26) grow without bound away from M∗M^*M∗; with unnormalized steps this makes the method diverge. Even with normalized steps, the distance to a minimizer decreases only outside a neighbourhood of M∗M^*M∗ whose size is of the order of the current step, so a monotone decrease argument gives at best a subsequence with small function values. Convergence of the whole sequence min⁡y∈M∗∥xk−y∥\min_{y \in M^*}\|x_k - y\|miny∈M∗​∥xk​−y∥ to zero is a stronger statement, and boundedness of M∗M^*M∗ is essential to it.

Formalization scope

  • EnE_nEn​ is EuclideanSpace ℝ (Fin n); fff is real-valued (finite everywhere) with ConvexOn ℝ Set.univ f.
  • The subgradient selection g is arbitrary, with the hypothesis ∀ x, IsSubgradient f x (g x). The starting point is arbitrary.
  • The iterations are defined recursively (normalizedIter, plainIter, resetIter); the stepsize sequence is h : ℕ → ℝ with h (k+1) used at step kkk. In (2.4), a zero subgradient is handled by an explicit branch that repeats the current iterate (which is then in M∗M^*M∗). No statement relies on Lean's convention x/0=0x/0 = 0x/0=0.
  • M∗M^*M∗ is required to be nonempty wherever the book writes min⁡y∈M∗\min_{y \in M^*}miny∈M∗​ or f∗=min⁡ff^* = \min ff∗=minf. min⁡y∈M∗∥xk−y∥\min_{y \in M^*}\|x_k - y\|miny∈M∗​∥xk​−y∥ is Metric.infDist, and f∗f^*f∗ is ⨅ y, f y.
  • ∑k≥1hk=+∞\sum_{k \ge 1} h_k = +\infty∑k≥1​hk​=+∞ is Tendsto (fun N => ∑ k ∈ Finset.range N, h (k+1)) atTop atTop. lim sup⁡hk<2r\limsup h_k < 2rlimsuphk​<2r is "for some q<2rq < 2rq<2r, eventually hk≤qh_k \le qhk​≤q", so it cannot hold vacuously for an unbounded sequence.
  • Corollary 1's step length hδh_\deltahδ​ is quantified before the selection and the starting point: it depends only on fff and δ\deltaδ.
  • Theorem 2.3 is stated as two implications, (bounded subgradients ⇒ convergence) and (unbounded ⇒ no convergence), not as a disjunction that one case could satisfy trivially.
  • A formalization of Theorem 2.2 that assumes bounded subgradients, a Lipschitz fff, or a specific subgradient choice (such as the minimal-norm one) proves a different and weaker theorem, and does not close the goal.

Needed infrastructure: continuity of convex functions on EnE_nEn​ (in Mathlib), compactness of sublevel sets when M∗M^*M∗ is bounded, and the geometry of level surfaces relative to supporting hyperplanes. The one-step inequality (2.3) and the level-set compactness lemma are reusable by the later missions of this series (linear rate, Polyak's stepsize, stochastic subgradient). Contributions that prove Eq. (2.3) or Theorem 2.1 first are welcome.

Selected references

  • N. Z. Shor, Minimization Methods for Non-Differentiable Functions, Springer Series in Computational Mathematics 3, Springer, 1985, Chapter 2, pp. 22–30. https://doi.org/10.1007/978-3-642-82118-9
  • B. T. Polyak, A general method for solving extremal problems, Doklady Akademii Nauk SSSR 174 (1967), 33–36 (the source's reference [64]).
  • Yu. M. Ermoliev, Methods for solving nonlinear extremal problems, Kibernetika (Kiev), no. 4 (1966), 1–17 (the source's reference [24]).
  • M. Held, P. Wolfe, H. P. Crowder, Validation of subgradient optimization, Mathematical Programming 6 (1974), 62–88. https://doi.org/10.1007/BF01580223
9 thms3 active usersReviewed
🏆Completed
Markov ChainProbabilityStochastic Systems·Captain: mikedeng1

Stochastic Dynamic Programming and the Control of Queueing Systems XIII: Lyapunov Criteria and z Standard Markov Chains with CostsTextbook

Motivation

Average cost control of queues rests on a small amount of Markov chain theory: when does a chain with costs have a well defined long-run average cost, and how can that be checked for a concrete model with an unbounded state space? Appendix C of L. I. Sennott, Stochastic Dynamic Programming and the Control of Queueing Systems (Wiley, 1999, doi:10.1002/9780470317037) collects this material for countable state spaces and packages it in one hypothesis, the zzz standard chain. Chapters 7–10 of the book verify this hypothesis for the Markov chains induced by stationary policies in admission, routing and service-rate control models, and use its consequences to prove existence of average cost optimal policies.

The tools are Lyapunov functions in the sense of Foster (1953): a nonnegative function on the states whose expected one-step change is negative away from a finite set. Foster's criterion for positive recurrence, and its refinements bounding expected first passage times and costs, are the standard way to verify stability of queueing networks (Meyn and Tweedie, Markov Chains and Stochastic Stability, 1993/2009).

Setting

A Markov chain Γ\GammaΓ on a countable set SSS is given by transition probabilities Pij≥0P_{ij}\ge 0Pij​≥0 with ∑jPij=1\sum_j P_{ij}=1∑j​Pij​=1. XtX_tXt​ is the state at time ttt and Pij(t)P^{(t)}_{ij}Pij(t)​ the ttt-step transition probability (Pij(0)=δijP^{(0)}_{ij}=\delta_{ij}Pij(0)​=δij​). State iii leads to jjj if Pij(t)>0P^{(t)}_{ij}>0Pij(t)​>0 for some t≥0t\ge0t≥0; states that lead to each other communicate, which partitions SSS into communicating classes.

For a nonempty G⊆SG\subseteq SG⊆S the first passage time from iii is TiG=min⁡{t≥1:Xt∈G}T_{iG}=\min\{t\ge1: X_t\in G\}TiG​=min{t≥1:Xt​∈G} given X0=iX_0=iX0​=i, and miG=E[TiG]∈[0,∞]m_{iG}=E[T_{iG}]\in[0,\infty]miG​=E[TiG​]∈[0,∞]; mijm_{ij}mij​ is the case G={j}G=\{j\}G={j} and miim_{ii}mii​ the expected return time. The taboo probability GPik(t)_G P^{(t)}_{ik}G​Pik(t)​ is the probability of going from iii to kkk in ttt steps without visiting GGG at the intermediate times, and Guik_G u_{ik}G​uik​ is the expected number of visits to kkk at times 0≤t<TiG0\le t<T_{iG}0≤t<TiG​. A state is transient if P(Tii<∞)<1P(T_{ii}<\infty)<1P(Tii​<∞)<1 and positive recurrent if mii<∞m_{ii}<\inftymii​<∞; a positive recurrent class is a communicating class of positive recurrent states. The steady state probability is πj=(mjj)−1\pi_j=(m_{jj})^{-1}πj​=(mjj​)−1 (zero when mjj=∞m_{jj}=\inftymjj​=∞).

Each state carries a finite cost C(i)≥0C(i)\ge0C(i)≥0. The expected average cost over [0,n−1][0,n-1][0,n−1] from iii is

Ji(n)=1n E[∑t=0n−1C(Xt) ∣ X0=i]=1n∑t=0n−1∑jPij(t)C(j),J^{(n)}_i=\frac1n\,E\Big[\sum_{t=0}^{n-1}C(X_t)\,\Big|\,X_0=i\Big]=\frac1n\sum_{t=0}^{n-1}\sum_j P^{(t)}_{ij}C(j),Ji(n)​=n1​E[t=0∑n−1​C(Xt​)​X0​=i]=n1​t=0∑n−1​j∑​Pij(t)​C(j),

ciGc_{iG}ciG​ is the expected cost E[∑t=0TiG−1C(Xt)∣X0=i]E[\sum_{t=0}^{T_{iG}-1}C(X_t)\mid X_0=i]E[∑t=0TiG​−1​C(Xt​)∣X0​=i] of a first passage (defined when miG<∞m_{iG}<\inftymiG​<∞), and JR=∑j∈RπjC(j)J_R=\sum_{j\in R}\pi_jC(j)JR​=∑j∈R​πj​C(j) is the average cost on a positive recurrent class RRR. The chain is zzz standard (Definition C.2.5) if for a distinguished state zzz

miz<∞andciz<∞for all i∈S.m_{iz}<\infty\quad\text{and}\quad c_{iz}<\infty\qquad\text{for all } i\in S.miz​<∞andciz​<∞for all i∈S.

Formalization targets

Goal: Proposition C.2.6

If Γ\GammaΓ is zzz standard, then SSS is the union of a positive recurrent class R∋zR\ni zR∋z and a set of transient states, JR<∞J_R<\inftyJR​<∞, and

lim⁡n→∞Ji(n)=JRfor every i∈S.\lim_{n\to\infty}J^{(n)}_i=J_R\qquad\text{for every } i\in S.n→∞lim​Ji(n)​=JR​for every i∈S.

The statement fixes no constants: it asserts that the average cost exists, is finite, and does not depend on the initial state.

Milestones

  1. Proposition C.1.2: π\piπ is the unique stationary distribution of a positive recurrent class, and πj=eij/mii=πieij\pi_j=e_{ij}/m_{ii}=\pi_ie_{ij}πj​=eij​/mii​=πi​eij​.
  2. Proposition C.1.4: the first-step equations (C.2)–(C.4) for taboo probabilities, visit counts and miGm_{iG}miG​; ∑i∈GπimiG=1\sum_{i\in G}\pi_im_{iG}=1∑i∈G​πi​miG​=1 for GGG inside a positive recurrent class; mij<∞m_{ij}<\inftymij​<∞ within such a class.
  3. Proposition C.1.5: if ∑jPij[y(j)−y(i)]≤−ϵ\sum_jP_{ij}[y(j)-y(i)]\le-\epsilon∑j​Pij​[y(j)−y(i)]≤−ϵ off GGG, then miG≤y(i)/ϵm_{iG}\le y(i)/\epsilonmiG​≤y(i)/ϵ.
  4. Corollary C.1.6: the same with G={z}G=\{z\}G={z} and ∑jPzjy(j)<∞\sum_jP_{zj}y(j)<\infty∑j​Pzj​y(j)<∞ makes zzz positive recurrent.
  5. Proposition C.2.1: on a positive recurrent class, Ji(n)→JR=cii/miiJ^{(n)}_i\to J_R=c_{ii}/m_{ii}Ji(n)​→JR​=cii​/mii​.
  6. Proposition C.2.2: ciG=∑kC(k) Guikc_{iG}=\sum_kC(k)\,{}_Gu_{ik}ciG​=∑k​C(k)G​uik​, the first-step equation (C.13), and JR=∑i∈GπiciGJ_R=\sum_{i\in G}\pi_ic_{iG}JR​=∑i∈G​πi​ciG​.
  7. Proposition C.2.3 and Corollary C.2.4: the cost drift condition ∑jPij[r(j)−r(i)]≤−C(i)\sum_jP_{ij}[r(j)-r(i)]\le-C(i)∑j​Pij​[r(j)−r(i)]≤−C(i) off a finite set bounds ciG≤r(i)+FmiGc_{iG}\le r(i)+Fm_{iG}ciG​≤r(i)+FmiG​, and gives czz<∞c_{zz}<\inftyczz​<∞.
  8. Remark C.2.7: the hypotheses of C.1.6 and C.2.4 together imply the chain is zzz standard; so do irreducibility, positive recurrence and finite average cost.

Significance

Proposition C.2.6 is what makes the zzz standard hypothesis useful: an average cost criterion that is a genuine limit, finite, and independent of the initial state, even for chains with transient states and unbounded state spaces. Every average cost optimality result of the book that works with a stationary policy's induced chain (the (SEN) and (BOR) assumption sets, the approximating-sequence method, the continuous-time chapter) calls on this proposition or on the Lyapunov criteria of Remark C.2.7 to establish its hypotheses for queueing models.

All results of the mission are classical and proved in the literature; parts are stated in the book without proof and referred to Chung (1967), Grassmann et al. (1985) and renewal theory. None of them has been machine-checked in this form as far as the platform and Mathlib show: Mathlib has kernels and Ionescu-Tulcea trajectories but no countable-state Markov chain classification, no first passage calculus, and no Foster–Lyapunov criterion. Existing platform results on countable chains (the Levin–Peres–Wilmer series) treat irreducible chains without costs. A complete development here produces a reusable library of first passage identities, Foster–Lyapunov bounds for times and costs, and average cost limits on reducible chains.

Difficulty

The Lyapunov bounds (C.1.5, C.2.3) are telescoping arguments, but they require a clean handling of truncated passages and of sums that may be infinite: (C.7) is an inequality between possibly divergent series, and the step "iterate nnn times and let n→∞n\to\inftyn→∞" must be made rigorous for [0,∞][0,\infty][0,∞]-valued expectations.

The central difficulty is part (iii) of the goal for transient initial states. On the class RRR, the limit of Ji(n)J^{(n)}_iJi(n)​ is a renewal reward theorem over successive returns to zzz; from a transient state the first cycle has a different law, so a delayed renewal reward argument is needed, and it has to cover the case where costs are unbounded. The obvious approach, bounding Ji(n)J^{(n)}_iJi(n)​ between JRJ_RJR​ and the average over the first nnn steps of the chain started in zzz, fails because Pij(t)P^{(t)}_{ij}Pij(t)​ need not converge (periodic classes) and because finite cizc_{iz}ciz​ does not bound individual cost terms. Proposition C.1.2's uniqueness and the Kac-type identity of C.1.4(iv) likewise need the full cycle decomposition of a positive recurrent class.

Formalization scope

The chain is a structure MC S with P : S → S → ℝ≥0∞ and ∑' j, P i j = 1, over a countable type S; costs are C : S → ℝ≥0. Probabilities and expectations are ℝ≥0∞-valued sums over finite paths Fin (t+1) → S, so every quantity is defined without summability side conditions and may be ∞\infty∞. The first passage time is TiG≥1T_{iG}\ge1TiG​≥1; miGm_{iG}miG​ is the expectation of TiGT_{iG}TiG​ from its law (and ∞\infty∞ when P(TiG<∞)<1P(T_{iG}<\infty)<1P(TiG​<∞)<1), not defined by the recursion (C.4), so that (C.4) is a theorem. Guik_Gu_{ik}G​uik​ counts visits at times 0≤t<TiG0\le t<T_{iG}0≤t<TiG​. ciGc_{iG}ciG​ is computed over first passage paths and is used only when miG<∞m_{iG}<\inftymiG​<∞, as in the book. πj\pi_jπj​ is (mjj)−1(m_{jj})^{-1}(mjj​)−1, which the book states equals the Cesàro limit lim⁡nQjj(n)\lim_nQ^{(n)}_{jj}limn​Qjj(n)​. Ji(n)J^{(n)}_iJi(n)​ is meaningful for n≥1n\ge1n≥1, and limits are taken in [0,∞][0,\infty][0,∞]. The drift conditions ∑jPij[y(j)−y(i)]≤−ϵ\sum_jP_{ij}[y(j)-y(i)]\le-\epsilon∑j​Pij​[y(j)−y(i)]≤−ϵ and ∑jPij[r(j)−r(i)]≤−C(i)\sum_jP_{ij}[r(j)-r(i)]\le-C(i)∑j​Pij​[r(j)−r(i)]≤−C(i) are written in the equivalent additive form ∑jPijy(j)+ϵ≤y(i)\sum_jP_{ij}y(j)+\epsilon\le y(i)∑j​Pij​y(j)+ϵ≤y(i), which is equivalent for finite yyy and makes the case ∑jPijy(j)=∞\sum_jP_{ij}y(j)=\infty∑j​Pij​y(j)=∞ fail, as it does in the book.

A trivializing formalization, such as defining miGm_{iG}miG​ or ciGc_{iG}ciG​ by the equations (C.4) or (C.13), defining JRJ_RJR​ as the limit of Ji(n)J^{(n)}_iJi(n)​, or allowing a zzz standard chain whose return time or return cost to zzz is infinite, is ruled out: zzz standard requires miz<∞m_{iz}<\inftymiz​<∞ and ciz<∞c_{iz}<\inftyciz​<∞ for every iii including zzz, and each quantity is defined from path probabilities.

Needed infrastructure: path-sum manipulation in [0,∞][0,\infty][0,∞] (first-step and last-step decompositions), the ratio limit / renewal reward theorem for a positive recurrent class, and the delayed version for transient starts. The first passage calculus and the Lyapunov bounds are reusable by the book's other chapters on average cost, which state the zzz standard property for policy-induced chains. Contributions of lemmas on path sums and of an independent renewal reward library are welcome.

Selected references

  • L. I. Sennott, Stochastic Dynamic Programming and the Control of Queueing Systems, Wiley, 1999, Appendix C, pp. 292–302. doi:10.1002/9780470317037
  • K. L. Chung, Markov Chains with Stationary Transition Probabilities, 2nd ed., Springer, 1967. doi:10.1007/978-3-642-62015-7
  • F. G. Foster, On the stochastic matrices associated with certain queuing processes, Annals of Mathematical Statistics 24 (1953), 355–360. doi:10.1214/aoms/1177728976
  • S. P. Meyn and R. L. Tweedie, Markov Chains and Stochastic Stability, 2nd ed., Cambridge University Press, 2009. doi:10.1017/CBO9780511626630
  • D. P. Heyman and M. J. Sobel, Stochastic Models in Operations Research, Vol. I, McGraw-Hill, 1982.
12 thms3 active usersReviewed
🏆Completed
Dynamic ProgrammingProbabilityStochastic Systems·Captain: mikedeng1

Stochastic Dynamic Programming and the Control of Queueing Systems X: Average Cost Optimization of Continuous Time Markov Decision ChainsTextbook

Motivation

Many queueing systems evolve in continuous time: customers arrive according to a Poisson process, services take exponentially distributed times, and a controller may change the service rate, admit or reject customers, or route them whenever the state changes. Minimizing the long-run average cost of such a system is a standard problem in the control of queues (Lippman 1975; Puterman 1994, Ch. 11; Sennott 1999, Ch. 10). The continuous time model does not fit directly into the discrete time theory of Markov decision chains developed in the earlier chapters of Sennott's book, because time spent in a state now matters and the natural average cost is a ratio of expected cost to expected elapsed time.

This mission formalizes Sections 10.1–10.4 of L. I. Sennott, Stochastic Dynamic Programming and the Control of Queueing Systems (Wiley, 1999): the elementary properties of the exponential distribution, the continuous time Markov decision chain and its average cost, a reduction of the continuous time problem to an auxiliary discrete time Markov decision chain, and the theorem stating that finite state approximating sequences of the auxiliary chain compute optimal average costs and optimal stationary policies of the continuous time chain. The chapter closes with an explicit average cost computation for the M/M/1 queue with service rate control.

Setting

A random variable XXX has the exponential distribution with rate μ>0\mu>0μ>0 if P(X≤t)=1−e−μtP(X\le t)=1-e^{-\mu t}P(X≤t)=1−e−μt for t≥0t\ge0t≥0. A function r(δ)r(\delta)r(δ) is o(δ)o(\delta)o(δ) if r(δ)/δ→0r(\delta)/\delta\to0r(δ)/δ→0 as δ→0+\delta\to0^+δ→0+.

A continuous time Markov decision chain (CTMDC) Ψ\PsiΨ has a countable state space SSS and, for each i∈Si\in Si∈S, a finite nonempty action set AiA_iAi​. Choosing a∈Aia\in A_ia∈Ai​ in state iii incurs an instantaneous cost G(i,a)≥0G(i,a)\ge0G(i,a)≥0 and a cost rate g(i,a)≥0g(i,a)\ge0g(i,a)≥0 in effect until the next transition. The time until the next transition is exponential with rate ν(i,a)>0\nu(i,a)>0ν(i,a)>0, so its mean is τ(i,a)=1/ν(i,a)\tau(i,a)=1/\nu(i,a)τ(i,a)=1/ν(i,a); the next state is jjj with probability Pij(a)P_{ij}(a)Pij​(a), where Pii(a)=0P_{ii}(a)=0Pii​(a)=0. A policy θ\thetaθ chooses, at each transition, an action (possibly at random) from the history of past states, actions and sojourn times; a stationary policy eee chooses e(i)e(i)e(i) in state iii. With CnC_nCn​ the cost and TnT_nTn​ the time of the first nnn transition periods, the average cost and the minimum average cost are

JθΨ(i)=lim sup⁡n→∞Eθ[Cn∣X0=i]Eθ[Tn∣X0=i],JΨ(i)=inf⁡θJθΨ(i).J^\Psi_\theta(i)=\limsup_{n\to\infty}\frac{E_\theta[C_n\mid X_0=i]}{E_\theta[T_n\mid X_0=i]},\qquad J^\Psi(i)=\inf_\theta J^\Psi_\theta(i).JθΨ​(i)=n→∞limsup​Eθ​[Tn​∣X0​=i]Eθ​[Cn​∣X0​=i]​,JΨ(i)=θinf​JθΨ​(i).

Assumption (CTB) requires constants τ\tauτ and BBB with 0<τ<inf⁡i,aτ(i,a)≤sup⁡i,aτ(i,a)≤B<∞0<\tau<\inf_{i,a}\tau(i,a)\le\sup_{i,a}\tau(i,a)\le B<\infty0<τ<infi,a​τ(i,a)≤supi,a​τ(i,a)≤B<∞. The auxiliary MDC Δ\DeltaΔ has the same states and actions, costs C(i,a)=G(i,a)ν(i,a)+g(i,a)C(i,a)=G(i,a)\nu(i,a)+g(i,a)C(i,a)=G(i,a)ν(i,a)+g(i,a), and transition probabilities Pij∗(a)=τν(i,a)Pij(a)P^*_{ij}(a)=\tau\nu(i,a)P_{ij}(a)Pij∗​(a)=τν(i,a)Pij​(a) for j≠ij\ne ij=i, Pii∗(a)=1−τν(i,a)P^*_{ii}(a)=1-\tau\nu(i,a)Pii∗​(a)=1−τν(i,a). Its average cost JθΔ(i)=lim sup⁡nn−1∑t<nEθ[C(Xt,Yt)]J^\Delta_\theta(i)=\limsup_n n^{-1}\sum_{t<n}E_\theta[C(X_t,Y_t)]JθΔ​(i)=limsupn​n−1∑t<n​Eθ​[C(Xt​,Yt​)] and minimum average cost JΔ(i)J^\Delta(i)JΔ(i) are those of Chapter 2. Assumption (CTAC) is JΔ(⋅)≤JΨ(⋅)J^\Delta(\cdot)\le J^\Psi(\cdot)JΔ(⋅)≤JΨ(⋅).

An approximating sequence (ΔN)N≥N0(\Delta_N)_{N\ge N_0}(ΔN​)N≥N0​​ for Δ\DeltaΔ uses finite state spaces SNS_NSN​ increasing to SSS and transition probabilities Pij∗(a;N)P^*_{ij}(a;N)Pij∗​(a;N) on SNS_NSN​ converging to Pij∗(a)P^*_{ij}(a)Pij∗​(a). The (AC) assumptions ask for constants JNJ^NJN and functions rNr^NrN on SNS_NSN​ solving

JN+rN(i)=min⁡a∈Ai{C(i,a)+∑j∈SNPij∗(a;N) rN(j)},i∈SN, N≥N0,(10.21)J^N+r^N(i)=\min_{a\in A_i}\Big\{C(i,a)+\sum_{j\in S_N}P^*_{ij}(a;N)\,r^N(j)\Big\},\qquad i\in S_N,\ N\ge N_0,\tag{10.21}JN+rN(i)=a∈Ai​min​{C(i,a)+j∈SN​∑​Pij∗​(a;N)rN(j)},i∈SN​, N≥N0​,(10.21)

with lim sup⁡NrN(i)<∞\limsup_N r^N(i)<\inftylimsupN​rN(i)<∞, lim inf⁡NrN(i)≥−Q\liminf_N r^N(i)\ge-QliminfN​rN(i)≥−Q for a constant Q≥0Q\ge0Q≥0, and lim sup⁡NJN=:J∗<∞\limsup_N J^N=:J^*<\inftylimsupN​JN=:J∗<∞, J∗≤JΔ(i)J^*\le J^\Delta(i)J∗≤JΔ(i).

Formalization targets

Goal: Theorem 10.3.3

Under (CTB), (CTAC) and the (AC) assumptions for an approximating sequence of Δ\DeltaΔ:

J∗=lim⁡N→∞JN exists and JΔ(i)=JΨ(i)=J∗(i∈S),J^*=\lim_{N\to\infty}J^N\ \text{exists and}\ J^\Delta(i)=J^\Psi(i)=J^*\quad(i\in S),J∗=N→∞lim​JN exists and JΔ(i)=JΨ(i)=J∗(i∈S),

and every limit point e∗e^*e∗ of a sequence eNe^NeN of stationary policies realizing the minimum in (10.21) satisfies Je∗Δ=JΔJ^\Delta_{e^*}=J^\DeltaJe∗Δ​=JΔ and Je∗Ψ=JΨJ^\Psi_{e^*}=J^\PsiJe∗Ψ​=JΨ. The goal leaves the chain, the approximating sequence and the constants of (CTB) arbitrary.

Milestones

  • Proposition 10.1.2: P(X>x+y∣X>y)=P(X>x)P(X>x+y\mid X>y)=P(X>x)P(X>x+y∣X>y)=P(X>x) for x,y>0x,y>0x,y>0, and P(X≤δ)=μδ+o(δ)P(X\le\delta)=\mu\delta+o(\delta)P(X≤δ)=μδ+o(δ).
  • Proposition 10.1.3: for independent exponentials, P(X1≤δ,X2≤δ)=o(δ)P(X_1\le\delta,X_2\le\delta)=o(\delta)P(X1​≤δ,X2​≤δ)=o(δ), P(X1<X2)=μ1/(μ1+μ2)P(X_1<X_2)=\mu_1/(\mu_1+\mu_2)P(X1​<X2​)=μ1​/(μ1​+μ2​), and min⁡(X1,X2)\min(X_1,X_2)min(X1​,X2​) is exponential with rate μ1+μ2\mu_1+\mu_2μ1​+μ2​.
  • Lemma 10.3.1: if zzz is bounded below and Zτ(i,e)+z(i)≥G(i,e)+g(i,e)τ(i,e)+∑jPij(e)z(j)Z\tau(i,e)+z(i)\ge G(i,e)+g(i,e)\tau(i,e)+\sum_jP_{ij}(e)z(j)Zτ(i,e)+z(i)≥G(i,e)+g(i,e)τ(i,e)+∑j​Pij​(e)z(j) for all iii (10.15), then JeΨ≤ZJ^\Psi_e\le ZJeΨ​≤Z.
  • Lemma 10.3.2: (Z,w)(Z,w)(Z,w) satisfies Z+w(i)≥C(i,e)+∑jPij∗(e)w(j)Z+w(i)\ge C(i,e)+\sum_jP^*_{ij}(e)w(j)Z+w(i)≥C(i,e)+∑j​Pij∗​(e)w(j) (10.20) if and only if (Z,τw)(Z,\tau w)(Z,τw) satisfies (10.15).
  • Proposition 10.4.1: in the M/M/1 queue with arrival rate λ\lambdaλ, holding cost H(i)=HiH(i)=HiH(i)=Hi and service cost rate c(a)c(a)c(a), the policy that always serves at rate a>λa>\lambdaa>λ has average cost ρac(a)+Hρa/(1−ρa)\rho_ac(a)+H\rho_a/(1-\rho_a)ρa​c(a)+Hρa​/(1−ρa​), ρa=λ/a\rho_a=\lambda/aρa​=λ/a.

Significance

The goal theorem turns the average cost control of a continuous time chain on an infinite state space into a finite computation: solve the optimality equation (10.21) of a finite truncation of the auxiliary chain, let the truncation grow, and read off the optimal average cost and an optimal stationary policy of the original continuous time chain. The auxiliary chain is the book's form of uniformization, and the result is what licenses the numerical study of the M/M/1 service rate control problem in Section 10.4 and of the M/M/K and polling models in Sections 10.5–10.6. Proposition 10.4.1 gives the closed-form benchmark against which the computed optimal policy is compared.

The results are proved in the book, some with details left to the reader (Lemma 10.3.2(ii), Problem 10.10), and the goal rests on Theorem 8.1.1 and Lemma 7.2.1 of the same book. None of them has, as far as a search of Mathlib and the Prove2Me catalogue shows, a machine-checked proof: Mathlib provides the exponential law (ProbabilityTheory.expMeasure) and its distribution function, but not memorylessness or the minimum of independent exponentials, and no continuous time Markov decision model. A formalization would supply these, together with a checked average cost comparison between a continuous time chain and its discrete time auxiliary chain.

Difficulty

The obvious argument compares the two chains policy by policy, but the policy classes differ: a policy for Δ\DeltaΔ may change action in every time slot, including slots where the state does not change, while a policy for Ψ\PsiΨ acts only at transitions and may use the observed sojourn times. Only the stationary policies coincide. The lower bound JΨ≥J∗J^\Psi\ge J^*JΨ≥J∗ therefore cannot be obtained by transferring policies, and it is exactly what Assumption (CTAC) supplies. The upper bound requires passing from the discrete time inequality (10.20) for the limit point e∗e^*e∗ to a bound on a ratio of expected cost to expected time in continuous time, where the denominator depends on the policy; the uniform bounds of (CTB) on the mean sojourn times are what control it. Inside Lemma 10.3.1 the function zzz is only bounded below, so the telescoping of expectations must be justified without integrability of zzz from above.

Formalization scope

The state space is a countable type S, actions a type Act, and action sets A i : Finset Act; the CTMDC and MDC structures hold data, and their axioms (nonempty action sets, nonnegative costs, positive rates, stochastic transition rows with Pii(a)=0P_{ii}(a)=0Pii​(a)=0) are separate predicates. Transition probabilities are ℝ≥0∞-valued; costs, rates and the functions z,w,rNz,w,r^Nz,w,rN are real. Expected costs, expected times and all average costs are ℝ≥0∞-valued, so +∞+\infty+∞ is a legitimate value, and they are compared with real constants in EReal; the limits superior and inferior of (AC) are taken in EReal. The expected cost of nnn transition periods under a general policy is a recursion over the periods in which the sojourn time is integrated against expMeasure ν(i,a) and the next state is drawn independently from Pi⋅(a)P_{i\cdot}(a)Pi⋅​(a); policies are measurable in the past sojourn times. In (10.15) and (10.20) the convergence of the series is part of the inequality. The strict inequality τ<inf⁡τ(i,a)\tau<\inf\tau(i,a)τ<infτ(i,a) of (CTB) is kept strict (as a positive margin); weakening it to ≤\le≤ would make Pii∗(a)P^*_{ii}(a)Pii∗​(a) vanish or turn negative.

The average cost JθΨJ^\Psi_\thetaJθΨ​ is a ratio of expectations, not the expectation of a ratio, and the infimum JΨJ^\PsiJΨ ranges over history dependent randomized policies that may use sojourn times; replacing either by a stationary-only class, or dropping (CTAC), gives a different theorem.

A complete development needs: expected rewards of a chain with exponential holding times, the average cost theory of Chapter 8 for the auxiliary chain (Theorem 8.1.1 and Lemma 7.2.1, restated here as needed), and renewal-reward reasoning for Proposition 10.4.1. The exponential-distribution lemmas are reusable beyond this mission and are welcome as independent contributions.

Selected references

  • L. I. Sennott, Stochastic Dynamic Programming and the Control of Queueing Systems, Wiley Series in Probability and Statistics, John Wiley & Sons, 1999. https://doi.org/10.1002/9780470317037
  • M. L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming, John Wiley & Sons, 1994. https://doi.org/10.1002/9780470316887
  • S. A. Lippman, Applying a new device in the optimization of exponential queuing systems, Operations Research 23(4), 687–710, 1975. https://doi.org/10.1287/opre.23.4.687
  • D. Gross and C. M. Harris, Fundamentals of Queueing Theory, 3rd ed., John Wiley & Sons, 1998.
11 thms3 active users
🏆Completed
ProbabilityStochastic Systems·Captain: mikedeng1

Stochastic Dynamic Programming and the Control of Queueing Systems IX: Bounded Mean Residual Lifetimes Imply Finite Moments of All OrdersTextbook

Motivation

In a discrete-time queue the service of a customer lasts a random number YYY of slots. When such a system is modelled as a Markov decision chain (Sennott, Stochastic Dynamic Programming and the Control of Queueing Systems, Wiley, 1999, DOI 10.1002/9780470317037, Chapter 9), the state must record how much service is still owed, and the controller only observes that a service has lasted sss slots and is not yet finished. The relevant random quantity is then the residual life YsY_sYs​: the remaining service time given that sss slots have elapsed without completion. Verifying the book's average cost assumptions for such a model requires bounds on expected first passage times and costs, and these reduce to moment bounds on YYY and on the residual lives YsY_sYs​.

Section 9.2 isolates a single condition that makes those bounds available: the expected remaining service time is bounded uniformly in the elapsed time. The concept of mean residual life comes from reliability theory, where YYY is the lifetime of a component and E[Ys]E[Y_s]E[Ys​] is its expected remaining lifetime at age sss. This mission formalizes Section 9.2 of the book, together with the moment computation for batch arrivals (Lemma 9.5.2) that the same verification uses.

Setting

Let YYY be a random variable with values in {1,2,3,… }\{1,2,3,\dots\}{1,2,3,…} and distribution uy=P(Y=y)u_y = P(Y = y)uy​=P(Y=y), y≥1y \ge 1y≥1. Write F(y)=P(Y≤y)F(y) = P(Y \le y)F(y)=P(Y≤y) and F∗(y)=P(Y>y)=1−F(y)F^*(y) = P(Y > y) = 1 - F(y)F∗(y)=P(Y>y)=1−F(y) for y≥0y \ge 0y≥0, so F(0)=0F(0) = 0F(0)=0 and F∗(0)=1F^*(0) = 1F∗(0)=1. The kkk-th moment is

E[Yk]=∑y≥1ykuy∈[0,∞].E[Y^k] = \sum_{y \ge 1} y^k u_y \in [0,\infty].E[Yk]=y≥1∑​ykuy​∈[0,∞].

For s≥0s \ge 0s≥0 with F∗(s)>0F^*(s) > 0F∗(s)>0, the residual life YsY_sYs​ has distribution

P(Ys=y)=P(Y=s+y∣Y>s)=us+yF∗(s),y≥1,P(Y_s = y) = P(Y = s + y \mid Y > s) = \frac{u_{s+y}}{F^*(s)}, \qquad y \ge 1,P(Ys​=y)=P(Y=s+y∣Y>s)=F∗(s)us+y​​,y≥1,

with Y0=YY_0 = YY0​=Y; its tail is Fs∗(y)=F∗(s+y)/F∗(s)F^*_s(y) = F^*(s+y)/F^*(s)Fs∗​(y)=F∗(s+y)/F∗(s), and E[Ys]E[Y_s]E[Ys​] is the mean residual lifetime.

The distribution of YYY has bounded mean residual lifetimes (BMRL-UUU, Definition 9.2.4) if there is a finite constant UUU with

E[Ys]≤Ufor every s≥0 with F∗(s)>0,E[Y_s] \le U \qquad \text{for every } s \ge 0 \text{ with } F^*(s) > 0,E[Ys​]≤Ufor every s≥0 with F∗(s)>0,

and it is BMRL if it is BMRL-UUU for some UUU.

Three families appear by name: the geometric distribution geo(μ)\mathrm{geo}(\mu)geo(μ) of the number of Bernoulli(μ\muμ) trials to the first success, P(Y=y)=μ(1−μ)y−1P(Y=y) = \mu(1-\mu)^{y-1}P(Y=y)=μ(1−μ)y−1; the negative binomial neg bin(μ,r)\mathrm{neg\,bin}(\mu, r)negbin(μ,r) of the number of trials to the rrr-th success, P(Y=y)=(y−1r−1)μr(1−μ)y−rP(Y = y) = \binom{y-1}{r-1}\mu^r(1-\mu)^{y-r}P(Y=y)=(r−1y−1​)μr(1−μ)y−r for y≥ry \ge ry≥r; and the truncated Poisson trun Pois(λ)\mathrm{trun\,Pois}(\lambda)trunPois(λ), P(Y=y)=e−λ1−e−λλyy!P(Y=y) = \frac{e^{-\lambda}}{1-e^{-\lambda}}\frac{\lambda^y}{y!}P(Y=y)=1−e−λe−λ​y!λy​ for y≥1y \ge 1y≥1.

For Lemma 9.5.2, batches of customers arrive in each slot; the batch sizes X1,X2,…X_1, X_2, \dotsX1​,X2​,… are independent with common distribution pjp_jpj​, mean λ=∑jjpj\lambda = \sum_j j p_jλ=∑j​jpj​ and second moment λ(2)=∑jj2pj\lambda^{(2)} = \sum_j j^2 p_jλ(2)=∑j​j2pj​, and X(s)=X1+⋯+XsX(s) = X_1 + \dots + X_sX(s)=X1​+⋯+Xs​ is the number of arrivals in sss slots.

Formalization targets

Goal: Proposition 9.2.5

If the distribution of YYY is BMRL, then

E[Yk]<∞for every k.E[Y^k] < \infty \qquad \text{for every } k.E[Yk]<∞for every k.

The goal fixes no constant: it asserts only that a uniform first-moment bound on the residual lives forces every moment of YYY to be finite.

Milestones

  1. Proposition 9.2.1. E[Y]=∑y=0∞F∗(y)E[Y] = \sum_{y=0}^\infty F^*(y)E[Y]=∑y=0∞​F∗(y) and, for k≥2k \ge 2k≥2,
E[Yk]=1+∑z=0k−1(kz)[∑y=1∞yzF∗(y)].(9.4)E[Y^k] = 1 + \sum_{z=0}^{k-1}\binom{k}{z}\left[\sum_{y=1}^\infty y^z F^*(y)\right]. \tag{9.4}E[Yk]=1+z=0∑k−1​(zk​)[y=1∑∞​yzF∗(y)].(9.4)
  1. Remark 9.2.2. For k≥2k \ge 2k≥2, E[Yk]<∞E[Y^k] < \inftyE[Yk]<∞ if and only if ∑yyk−1F∗(y)<∞\sum_y y^{k-1}F^*(y) < \infty∑y​yk−1F∗(y)<∞.
  2. Proposition 9.2.3. For a positive integer kkk, E[Yk]<∞E[Y^k] < \inftyE[Yk]<∞ implies E[Ysk]<∞E[Y_s^k] < \inftyE[Ysk​]<∞ for all s≥0s \ge 0s≥0.
  3. Proposition 9.2.6. The geometric (0<μ<10<\mu<10<μ<1), negative binomial (0<μ<10<\mu<10<μ<1, r≥2r \ge 2r≥2) and truncated Poisson (λ>0\lambda > 0λ>0) distributions are BMRL.
  4. Lemma 9.5.2. Under λ(2)<∞\lambda^{(2)} < \inftyλ(2)<∞,
E[X(s)]=λs,E[(X(s))2]=λ(2)s+λ2s(s−1).(9.25)E[X(s)] = \lambda s, \qquad E[(X(s))^2] = \lambda^{(2)}s + \lambda^2 s(s-1). \tag{9.25}E[X(s)]=λs,E[(X(s))2]=λ(2)s+λ2s(s−1).(9.25)

Significance

The result itself. Proposition 9.2.5 turns a condition that is easy to check for concrete service distributions, and natural for services (a service whose expected remaining duration grows without bound as it goes on is undesirable), into the moment bounds that the average cost analysis consumes. With Proposition 9.2.6 it shows that the most common unbounded service distributions on {1,2,… }\{1,2,\dots\}{1,2,…} have finite moments of all orders; with Lemma 9.5.2 it supplies the linear and quadratic growth of expected arrivals and their second moments that the verification of the (WAC) assumptions for the batch-arrival queue of Example 9.3.1 needs (Section 9.5). Every bounded distribution is BMRL as well (the book's Problem 9.3).

Formalizing it. All results here are proved in the book; none has a machine-checked proof on the platform or in Mathlib, which has geometric and Poisson distributions but no residual lives, negative binomial or truncated Poisson laws. A complete development gives a reusable tail-sum calculus for moments of N\mathbb NN-valued random variables in [0,∞][0,\infty][0,∞], a residual-life construction for discrete distributions, and the BMRL property of three standard families. The platform's mean residual life order (the "Stochastic Orders II" mission, Shaked–Shanthikumar) compares two variables; BMRL is a uniform bound on one variable's residual lives and is not an order, so none of that material states these results.

Difficulty

BMRL controls only first moments, of the conditional laws YsY_sYs​; the goal asks for moments of every order of YYY itself. Bounding E[Yk]E[Y^k]E[Yk] by expanding E[Ys]E[Y_s]E[Ys​] for each fixed sss gives nothing, because each single bound is compatible with a heavy tail: the uniformity in sss is essential. The residual lives are also only defined where P(Y>s)>0P(Y > s) > 0P(Y>s)>0, so every argument must handle distributions with bounded support separately. Proposition 9.2.6 requires explicit control of ratios of tail sums for three families; for the negative binomial and truncated Poisson the tails have no closed form.

Formalization scope

  • YYY is represented by its law, a function u:N→[0,∞]u : \mathbb N \to [0,\infty]u:N→[0,∞] with ∑yuy=1\sum_y u_y = 1∑y​uy​=1 and u0=0u_0 = 0u0​=0 (IsDistOnPos). F∗F^*F∗, moments and residual-life moments are ℝ≥0∞-valued series; an infinite moment is +∞+\infty+∞ and "finite" means <∞< \infty<∞. No Bochner integral is used, so a finite-moment conclusion cannot hold vacuously through an integrability default.
  • The residual life YsY_sYs​ is defined by (9.7) and is used only where F∗(s)>0F^*(s) > 0F∗(s)>0; BMRL-UUU is required exactly at those sss, and UUU is a finite nonnegative real. A formalization requiring the bound at every sss with a junk value of E[Ys]E[Y_s]E[Ys​] where F∗(s)=0F^*(s) = 0F∗(s)=0 is ruled out: the definitions never divide by F∗(s)=0F^*(s) = 0F∗(s)=0 in a used position, and bounded distributions remain BMRL.
  • The geometric and negative binomial laws count trials (support starting at 111 and rrr), not failures as Mathlib's geometricPMF does.
  • Lemma 9.5.2 is stated on a probability space with measurable, mutually independent (iIndepFun) batch sizes of common law ppp, expectations as lower Lebesgue integrals, and only assumption (BA1), λ(2)<∞\lambda^{(2)} < \inftyλ(2)<∞, which is the part of the book's (BA) that concerns arrivals.
  • Welcome contributions: the tail-sum identity (9.4) and its reindexing lemmas, the residual-life tail formula (9.8) and moment formula (9.9), each as a separate lemma; and proofs that the three named families are probability distributions on their supports.

Selected references

  • Linn I. Sennott, Stochastic Dynamic Programming and the Control of Queueing Systems, Wiley Series in Probability and Statistics, John Wiley & Sons, 1999, Section 9.2 (pp. 202–206) and Section 9.5 (pp. 214–215). DOI 10.1002/9780470317037
  • Moshe Shaked and J. George Shanthikumar, Stochastic Orders, Springer Series in Statistics, Springer, 2007, Section 2.A (the mean residual life order). DOI 10.1007/978-0-387-34675-5
8 thms3 active usersReviewed
🏆Completed
Dynamic ProgrammingMarkov ChainProbability·Captain: mikedeng1

Stochastic Dynamic Programming and the Control of Queueing Systems VIII: Computing Average Cost Optimal Policies by Approximating SequencesTextbook

Motivation

Control problems for queueing systems (admission control, service rate control, routing to parallel servers) are naturally modelled as Markov decision chains whose state is a vector of queue lengths. The state space is therefore denumerably infinite, and the performance measure of interest is usually the long-run average cost per slot. For such models the existence theory of average cost optimal stationary policies is well developed (Chapter 7 of Sennott's book), but existence gives no algorithm: an optimal policy is a function on an infinite set, and value iteration cannot be run on an infinite state space.

The approximating sequence method answers this by replacing the infinite model Δ\DeltaΔ with a sequence of finite models ΔN\Delta_NΔN​ on truncated state spaces SNS_NSN​, solving the average cost optimality equation in each, and passing to the limit. Chapter 8 of L. I. Sennott, Stochastic Dynamic Programming and the Control of Queueing Systems (Wiley, 1999) gives a set of conditions, the (AC) assumptions, under which this limit procedure provably produces the minimum average cost and an average cost optimal policy of Δ\DeltaΔ.

Timeline. The approximating sequence method for the average cost criterion and the (AC) assumptions were introduced in Sennott (1997a), with further results in Sennott (1997b) (bibliographic notes, p. 194). The book collects these results, adds the four step verification template (Proposition 8.2.1), the finite-set augmentation route based on the (BOR) assumptions (Proposition 8.2.3), and the weakening (WAC) of Section 8.7, which Chapter 9 uses.

Setting

An MDC Δ\DeltaΔ has a countable state space SSS, a finite nonempty action set AiA_iAi​ in each state iii, a finite cost C(i,a)≥0C(i,a)\ge0C(i,a)≥0, and transition probabilities Pij(a)P_{ij}(a)Pij​(a). A general policy θ\thetaθ chooses actions at random using the whole past history. Its average cost is

Jθ(i)=lim sup⁡n→∞1n∑t=0n−1Eθ[C(Xt,At)∣X0=i],J_\theta(i)=\limsup_{n\to\infty}\frac1n\sum_{t=0}^{n-1}E_\theta[C(X_t,A_t)\mid X_0=i],Jθ​(i)=n→∞limsup​n1​t=0∑n−1​Eθ​[C(Xt​,At​)∣X0​=i],

and the minimum average cost is J(i)=inf⁡θJθ(i)∈[0,∞]J(i)=\inf_\theta J_\theta(i)\in[0,\infty]J(i)=infθ​Jθ​(i)∈[0,∞]. A policy is average cost optimal if Jθ≡JJ_\theta\equiv JJθ​≡J.

An approximating sequence (ΔN)N≥N0(\Delta_N)_{N\ge N_0}(ΔN​)N≥N0​​ consists of finite sets SNS_NSN​ increasing to SSS and MDCs ΔN\Delta_NΔN​ on SNS_NSN​ with the same actions and costs and with transition probabilities Pij(a;N)P_{ij}(a;N)Pij​(a;N) on SNS_NSN​ converging to Pij(a)P_{ij}(a)Pij​(a). Write vnNv^N_nvnN​ and VαNV^N_\alphaVαN​ for the nnn-horizon and discounted value functions of ΔN\Delta_NΔN​.

The (AC) assumptions are:

  • (AC1) there are finite constants JNJ^NJN and finite functions rNr^NrN on SNS_NSN​ with
JN+rN(i)=min⁡a{C(i,a)+∑j∈SNPij(a;N) rN(j)},i∈SN;(8.1)J^N+r^N(i)=\min_a\Big\{C(i,a)+\sum_{j\in S_N}P_{ij}(a;N)\,r^N(j)\Big\},\qquad i\in S_N; \tag{8.1}JN+rN(i)=amin​{C(i,a)+j∈SN​∑​Pij​(a;N)rN(j)},i∈SN​;(8.1)
  • (AC2) lim sup⁡NrN(i)<∞\limsup_N r^N(i)<\inftylimsupN​rN(i)<∞;
  • (AC3) lim inf⁡NrN(i)≥−Q\liminf_N r^N(i)\ge -QliminfN​rN(i)≥−Q for a constant Q≥0Q\ge0Q≥0;
  • (AC4) J∗:=lim sup⁡NJN<∞J^*:=\limsup_N J^N<\inftyJ∗:=limsupN​JN<∞ and J∗≤J(i)J^*\le J(i)J∗≤J(i) for all iii.

The (WAC) assumptions of Section 8.7 allow QQQ to depend on the state, at the price of integrability conditions along every stationary policy.

Formalization targets

Goal: Theorem 8.1.1

Under (AC), the limit lim⁡N→∞JN\lim_{N\to\infty}J^NlimN→∞​JN exists and

J(i)=lim⁡N→∞JNfor all i∈S,J(i)=\lim_{N\to\infty}J^N\qquad\text{for all } i\in S,J(i)=N→∞lim​JNfor all i∈S,

and every limit point e∗e^*e∗ of stationary policies eNe^NeN realizing the minimum in (8.1) is average cost optimal for Δ\DeltaΔ. The statement fixes no constants; it asserts the shape of the conclusion for any model satisfying (AC).

Milestones

  • Proposition 8.2.1 (the four step template): unichain and aperiodicity of the finite models, an xxx standard policy at which the approximating sequence is conforming, a comparison of vnNv^N_nvnN​ (or VαNV^N_\alphaVαN​) with vnv_nvn​ (or VαV_\alphaVα​), and a lower bound on vnN−vnN(x)v^N_n - v^N_n(x)vnN​−vnN​(x) together imply that the value iteration limits
rN(i)=lim⁡n→∞(vnN(i)−vnN(x))r^N(i)=\lim_{n\to\infty}\big(v^N_n(i)-v^N_n(x)\big)rN(i)=n→∞lim​(vnN​(i)−vnN​(x))

exist and satisfy (AC).

  • Corollary 8.2.2: on S={0,1,2,… }S=\{0,1,2,\dots\}S={0,1,2,…} with SN={0,…,N}S_N=\{0,\dots,N\}SN​={0,…,N} and excess probability sent to NNN, monotonicity of vnNv^N_nvnN​, vnv_nvn​ and of the first passage moments of a 000 standard policy suffices.
  • Proposition 8.2.3: an augmentation type approximating sequence that sends excess probability to a finite set of cheap states satisfies the template.
  • Proposition 8.5.1: in the single-server queue with Bernoulli(ppp) arrivals and constant service rate a>pa>pa>p,
Jd(a)=Hp(1−p)a−p+pC(a)a.J_{d(a)}=\frac{Hp(1-p)}{a-p}+\frac{pC(a)}{a}.Jd(a)​=a−pHp(1−p)​+apC(a)​.
  • Proposition 8.7.1: the conclusions of Theorem 8.1.1 hold under (WAC).

Significance

The result itself. Theorem 8.1.1 is what turns the existence theory of Chapter 7 into a computation. It certifies that the minimum average costs of the truncations converge to the minimum average cost of the infinite model, that this cost is constant, and that the policies produced by value iteration on ΔN\Delta_NΔN​ converge, along subsequences, to an optimal policy for Δ\DeltaΔ. Propositions 8.2.1–8.2.3 reduce (AC) to properties that can be checked model by model; Section 8.3 checks them for queues with reject option, service rate control, and routing to parallel queues. Proposition 8.7.1 is the version used in Chapter 9 for models whose relative values are not uniformly bounded below. Proposition 8.5.1 gives the closed-form open-loop benchmark used in the numerical study of Section 8.5.

Formalizing it. All results are proved in the book; none has a machine-checked proof. A formalization would give the first verified convergence theorem for truncations of denumerable-state average cost MDPs, and would make the approximating sequence method usable as a certified reduction from infinite to finite models. The template results (8.2.1–8.2.3) additionally require a formal treatment of conformity of approximating Markov chains (Appendix C.4–C.5), which is of independent use.

Difficulty

The naive argument takes limits in (8.1) along NNN: the minimum over aaa and the finite sums pass to the limit only in the inequality direction, and only after a Fatou-type lemma for sums against the converging distributions Pij(a;N)P_{ij}(a;N)Pij​(a;N) with integrands rNr^NrN that are neither bounded nor monotone. The lower bound −Q-Q−Q in (AC3) is exactly what makes this possible; without it the limit inequality can fail. The limit inequality then produces only an average cost optimality inequality, and turning it into optimality of the limit policy requires a separate argument that a function bounded below and satisfying the inequality yields an upper bound on the average cost. Existence of lim⁡NJN\lim_N J^NlimN​JN is not given: (AC4) controls only the limit superior, and the limit must be identified through every subsequence. For the template results, the difficulty is in the Markov chain side: the convergence of first passage times and costs of the truncated chains, which fails for general approximating sequences (Examples C.4.4, C.4.7).

Formalization scope

The Lean development is in the namespace SennottDP.AvgASM. The state space is any countable type; action sets are Finsets, assumed nonempty; costs are finite and nonnegative (ℝ≥0); transition probabilities are ℝ≥0∞-valued, with each row a probability distribution for admissible actions. All value functions and average costs take values in [0,∞][0,\infty][0,∞] (ℝ≥0∞), and every infimum over policies ranges over the full class of history-dependent randomized policies. The JNJ^NJN and rNr^NrN of (AC1) are real; the limits superior and inferior over NNN in (AC2)–(AC4) and (WAC) are taken in EReal, so an unbounded sequence cannot produce a junk finite value. The equality J(i)=lim⁡NJNJ(i)=\lim_N J^NJ(i)=limN​JN is stated in EReal, which also asserts that J(i)J(i)J(i) is finite. Quantities of ΔN\Delta_NΔN​ at states outside SNS_NSN​ are junk values that affect only finitely many NNN for each state.

A trivializing formalization is ruled out: JNJ^NJN and rNr^NrN are the witnesses of (AC1), not free variables, the policies eNe^NeN must realize the minimum in (8.1) for those witnesses, and the minimum average cost JJJ is an infimum over all policies, so the goal cannot be satisfied by choosing J∗J^*J∗ or the limit policy.

A complete development needs: the induced process law of a general policy; the average cost optimality inequality argument (Lemma 7.2.1); the finite-state average cost results of Chapter 6 (Propositions 6.4.1, 6.5.1, 6.6.3); a Fatou lemma for converging distributions (Proposition A.2.5); limit points of policy sequences (Proposition B.5); and, for the template results, the theory of zzz standard chains and conformity (Appendix C.2–C.5). The Markov chain layer and the approximating sequence definitions are reusable beyond this mission. Contributions of any of these intermediate results as separate theorems are welcome.

Selected references

  • L. I. Sennott, Stochastic Dynamic Programming and the Control of Queueing Systems, Wiley Series in Probability and Statistics, John Wiley & Sons, 1999. https://doi.org/10.1002/9780470317037
  • L. I. Sennott, "The computation of average optimal policies in denumerable state Markov decision chains", Advances in Applied Probability 29 (1997), 114–137 (cited as Sennott (1997a) in the book).
  • L. I. Sennott, "On computing average cost optimal policies with application to routing to parallel queues", ZOR — Mathematical Methods of Operations Research 45 (1997), 45–62 (cited as Sennott (1997b) in the book).
14 thms3 active users
🏆Completed
Dynamic ProgrammingMarkov ChainProbability·Captain: mikedeng1

Stochastic Dynamic Programming and the Control of Queueing Systems IV: Average Cost Optimal Stationary Policies Exist for Finite State SpacesTextbook

Why average cost on finite state spaces

Controlled queues, inventories and communication links are run for a long time, and the quantity an operator usually cares about is the long-run average cost per period rather than a discounted total. The average cost criterion is harder to work with than the discounted one: its value is a lim sup⁡\limsuplimsup of Cesàro means, it is not given by a contraction, and for general (history dependent, randomized) policies the limit need not exist. Chapter 6 of L. I. Sennott, Stochastic Dynamic Programming and the Control of Queueing Systems (Wiley, 1999) treats the case of a finite state space, where the strongest results hold: an average cost optimal policy exists, can be taken stationary, and can be obtained as a limit of discount optimal policies as the discount factor tends to one.

The results go back to D. Blackwell, "Discrete dynamic programming", Ann. Math. Statist. 33 (1962), who showed that for finite states and actions some stationary policy is discount optimal for all discount factors close to one. Such a policy is now called Blackwell optimal. Sennott's Chapter 6 derives average cost optimality of this policy and the multichain average cost optimality equation from it, in the notation used throughout the book.

Setting

A Markov decision chain (MDC) Δ\DeltaΔ has a countable state space SSS, a finite nonempty action set AiA_iAi​ in each state iii, nonnegative finite costs C(i,a)C(i,a)C(i,a), and transition probabilities Pij(a)P_{ij}(a)Pij​(a) with ∑jPij(a)=1\sum_j P_{ij}(a) = 1∑j​Pij​(a)=1. A policy θ\thetaθ chooses the action at time ttt from a distribution θ(⋅∣ht)\theta(\cdot \mid h_t)θ(⋅∣ht​) on AitA_{i_t}Ait​​ that may depend on the whole history ht=(i0,a0,…,it)h_t = (i_0,a_0,\ldots,i_t)ht​=(i0​,a0​,…,it​). A stationary policy fff always chooses a fixed action f(i)∈Aif(i) \in A_if(i)∈Ai​ in state iii.

With Xt,AtX_t, A_tXt​,At​ the state and action at time ttt and X0=iX_0 = iX0​=i, define

  • the discounted cost Vθ,α(i)=∑t≥0αtEθ[C(Xt,At)]V_{\theta,\alpha}(i) = \sum_{t \ge 0} \alpha^t E_\theta[C(X_t,A_t)]Vθ,α​(i)=∑t≥0​αtEθ​[C(Xt​,At​)] for 0<α<10<\alpha<10<α<1, and the discounted value function Vα(i)=inf⁡θVθ,α(i)V_\alpha(i) = \inf_\theta V_{\theta,\alpha}(i)Vα​(i)=infθ​Vθ,α​(i);
  • the nnn horizon cost vθ,n(i)=∑t=0n−1Eθ[C(Xt,At)]v_{\theta,n}(i) = \sum_{t=0}^{n-1} E_\theta[C(X_t,A_t)]vθ,n​(i)=∑t=0n−1​Eθ​[C(Xt​,At​)];
  • the average cost Jθ(i)=lim sup⁡nvθ,n(i)/nJ_\theta(i) = \limsup_n v_{\theta,n}(i)/nJθ​(i)=limsupn​vθ,n​(i)/n, its lim inf⁡\liminfliminf version Jθ∗(i)J^*_\theta(i)Jθ∗​(i), and the minimum average cost J(i)=inf⁡θJθ(i)J(i) = \inf_\theta J_\theta(i)J(i)=infθ​Jθ​(i).

All infima range over all general policies, and every quantity may equal +∞+\infty+∞. A policy is α\alphaα discount optimal if Vθ,α=VαV_{\theta,\alpha} = V_\alphaVθ,α​=Vα​, and average cost optimal if Jθ=JJ_\theta = JJθ​=J.

For a stationary policy fff on a finite state space, the induced Markov chain splits into positive recurrent classes R1,…,RKR_1,\ldots,R_KR1​,…,RK​ and transient states. With pk(i)p_k(i)pk​(i) the probability of reaching RkR_kRk​ from iii, distinguished states zk∈Rkz_k \in R_kzk​∈Rk​, and Wα(i)=∑kpk(i)Vα(zk)W_\alpha(i) = \sum_k p_k(i) V_\alpha(z_k)Wα​(i)=∑k​pk​(i)Vα​(zk​), the relative value function is wα(i)=Vα(i)−Wα(i)w_\alpha(i) = V_\alpha(i) - W_\alpha(i)wα​(i)=Vα​(i)−Wα​(i).

Formalization targets

Goal: Proposition 6.2.3

For an MDC with a finite state space there are α0∈(0,1)\alpha_0 \in (0,1)α0​∈(0,1) and one stationary policy fff such that fff is α\alphaα discount optimal for every α∈(α0,1)\alpha \in (\alpha_0,1)α∈(α0​,1), fff is average cost optimal, and

J(i)=lim⁡α→1−(1−α)Vα(i)=lim⁡n→∞vf,n(i)n,i∈S.J(i) = \lim_{\alpha\to 1^-} (1-\alpha) V_\alpha(i) = \lim_{n\to\infty} \frac{v_{f,n}(i)}{n}, \qquad i \in S.J(i)=α→1−lim​(1−α)Vα​(i)=n→∞lim​nvf,n​(i)​,i∈S.

Milestones

  1. Proposition 4.5.3. For finite SSS and stationary eee, α↦Ve,α(i)\alpha \mapsto V_{e,\alpha}(i)α↦Ve,α​(i) is a finite, continuous, rational function on (0,1)(0,1)(0,1).
  2. Proposition 6.1.1. For every policy on a countable state space,
Jθ∗(i)≤lim inf⁡α→1−(1−α)Vθ,α(i)≤lim sup⁡α→1−(1−α)Vθ,α(i)≤Jθ(i),J^*_\theta(i) \le \liminf_{\alpha\to1^-}(1-\alpha)V_{\theta,\alpha}(i) \le \limsup_{\alpha\to1^-}(1-\alpha)V_{\theta,\alpha}(i) \le J_\theta(i),Jθ∗​(i)≤α→1−liminf​(1−α)Vθ,α​(i)≤α→1−limsup​(1−α)Vθ,α​(i)≤Jθ​(i),

with three equivalent conditions for equality. 3. Proposition 6.2.2. For finite SSS and stationary eee, Je(i)=lim⁡α→1−(1−α)Ve,α(i)=lim⁡nve,n(i)/nJ_e(i) = \lim_{\alpha\to1^-}(1-\alpha)V_{e,\alpha}(i) = \lim_n v_{e,n}(i)/nJe​(i)=limα→1−​(1−α)Ve,α​(i)=limn​ve,n​(i)/n. 4. Proposition 4.5.1, Proposition 4.5.4, Corollary 4.5.5. The power series structure of Vθ,αV_{\theta,\alpha}Vθ,α​ in α\alphaα; monotonicity and left continuity of VαV_\alphaVα​; continuity under bounded costs. 5. Theorem 6.3.1. For the policy fff of the goal, lim⁡α→1−wα(i)=w(i)\lim_{\alpha \to 1^-} w_\alpha(i) = w(i)limα→1−​wα​(i)=w(i) exists, and

J(i)+w(i)=C(i,f)+∑jPij(f)w(j) ≥ min⁡a{C(i,a)+∑jPij(a)w(j)},J(i) + w(i) = C(i,f) + \sum_j P_{ij}(f) w(j) \ \ge\ \min_{a} \Big\{C(i,a) + \sum_j P_{ij}(a) w(j)\Big\},J(i)+w(i)=C(i,f)+j∑​Pij​(f)w(j) ≥ amin​{C(i,a)+j∑​Pij​(a)w(j)},

together with the limit identities (i)–(iii) and the optimality criterion (v). 6. Proposition 6.3.3. Vα(i)=J(i)/(1−α)+w∗(i)+εα(i)V_\alpha(i) = J(i)/(1-\alpha) + w^*(i) + \varepsilon_\alpha(i)Vα​(i)=J(i)/(1−α)+w∗(i)+εα​(i) with εα(i)→0\varepsilon_\alpha(i) \to 0εα​(i)→0 as α→1−\alpha \to 1^-α→1−.

Significance

The goal says that on a finite state space nothing is gained by randomizing or by remembering the past when minimizing average cost, and that the minimum average cost is the vanishing-discount limit of the discounted value function. This justifies computing average cost optimal policies through discounted problems and value iteration, the route taken in the rest of Chapter 6 and, via approximating sequences, for countable state spaces in Chapters 7 and 8. Theorem 6.3.1 supplies an optimality equation without any unichain or communication assumption. The book's Example 6.3.2 shows that the inequality in that equation can be strict, and that a stationary policy attaining the minimum need not be optimal.

The results are classical and proved in the book. No machine-checked version of them is known to exist. The platform has average-reward results for unichain finite MDPs with Markov policies (the Puterman series) and an average-cost optimality equation under recurrence assumptions (the Bertsekas series). Neither covers existence of a Blackwell optimal policy against the class of all history dependent randomized policies, or the multichain equation. A formal development also yields reusable infrastructure: the law of a controlled process under a general policy, first passage quantities of finite chains, and the Abelian inequality between Abel and Cesàro means of a nonnegative sequence.

Difficulty

The obvious argument picks, for each α\alphaα, a stationary discount optimal policy fαf_\alphafα​ and lets α→1\alpha \to 1α→1. Finiteness of the set of stationary policies gives one policy that is optimal along some sequence αn→1\alpha_n \to 1αn​→1, but not on an interval. Excluding infinite switching between two policies requires the analytic structure of α↦Vf,α(i)\alpha \mapsto V_{f,\alpha}(i)α↦Vf,α​(i) (Proposition 4.5.3), which in turn rests on matrix inversion of I−αPI - \alpha PI−αP. Passing from the discounted criterion to the average one requires an Abelian inequality for nonnegative series whose terms may be infinite (Proposition 6.1.1), and comparison against general policies rules out any argument that works only within stationary or Markov policies. For Theorem 6.3.1 the difficulty is the multichain structure: the relative value function has to be assembled class by class from first passage times and costs, and its limit must be identified.

Formalization scope

  • States form a type S; [Countable S] for Section 4.5 and Proposition 6.1.1, [Fintype S] from Section 6.2 on, as in the book. Actions form a type Act with A i : Finset Act nonempty. Costs are in ℝ≥0, transition probabilities in ℝ≥0∞.
  • A general policy is a function of the list of past state-action pairs (most recent first) and the current state, giving a distribution on A i. Stationary policies embed as degenerate policies. The law of the process is built from this data, and every infimum ranges over all general policies.
  • Vθ,αV_{\theta,\alpha}Vθ,α​, VαV_\alphaVα​, vθ,nv_{\theta,n}vθ,n​, JθJ_\thetaJθ​, Jθ∗J^*_\thetaJθ∗​, JJJ are in ℝ≥0∞, so +∞+\infty+∞ is represented. α→1−\alpha \to 1^-α→1− is the filter 𝓝[<] 1. On a finite state space these quantities are finite. The real valued objects of Section 6.3 (wαw_\alphawα​, www, w∗w^*w∗, equation (6.6)) are therefore formed with toReal, and this switch from ℝ≥0∞ to ℝ happens only in Theorem 6.3.1 and Proposition 6.3.3.
  • The objects of Section 6.3 (pkp_kpk​, mi∣km_{i|k}mi∣k​, ci∣kc_{i|k}ci∣k​, πs\pi_sπs​, WαW_\alphaWα​) are defined from fff. The distinguished states are a hypothesis quantified over.
  • A trivializing formalization would take the infimum over stationary policies only, let the optimal policy depend on α\alphaα, or state rationality as an equation p/q without requiring q≠0q \ne 0q=0. Each is excluded here: JJJ and VαV_\alphaVα​ are infima over all general policies, one pair (α0,f)(\alpha_0,f)(α0​,f) is quantified before all α\alphaα, and the denominator is required to be nonzero on (0,1)(0,1)(0,1).

Useful infrastructure includes rational functions of one real variable and their finitely many sign changes, the resolvent (I−αP)−1(I-\alpha P)^{-1}(I−αP)−1 of a stochastic matrix, the Abelian inequality for [0,∞][0,\infty][0,∞]-valued sequences, and renewal-reward identities for finite chains. Contributions of general lemmas on these topics are welcome, as are proofs of individual milestones.

Selected references

  • L. I. Sennott, Stochastic Dynamic Programming and the Control of Queueing Systems, Wiley, 1999. https://doi.org/10.1002/9780470317037
  • D. Blackwell, "Discrete dynamic programming", Annals of Mathematical Statistics 33 (1962), 719–726. https://doi.org/10.1214/aoms/1177704593
  • M. L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming, Wiley, 1994. https://doi.org/10.1002/9780470316887
13 thms3 active usersReviewed
🏆Completed
Dynamic ProgrammingProbabilityStochastic Systems·Captain: mikedeng1

Stochastic Dynamic Programming and the Control of Queueing Systems II: The Discount Optimality EquationTextbook

Motivation

Control problems for queueing systems (admission control, routing, service rate selection, inventory replenishment) are naturally modelled as Markov decision chains with a countable state space, such as the number of customers in a buffer, and with costs that grow without bound in the state, such as holding costs proportional to queue length. The expected discounted cost criterion is the first infinite horizon criterion applied to such models, and it is also the tool through which the average cost criterion is treated later in the same book (Chapters 6–8 of Sennott's text reach average cost optimal policies through limits of discounted problems as the discount factor tends to one).

Classical treatments of discounted dynamic programming assume bounded costs, under which the dynamic programming operator is a contraction and has a unique bounded fixed point. That assumption fails for queueing models. This mission formalizes Chapter 4, Sections 4.1–4.4, of L. I. Sennott, Stochastic Dynamic Programming and the Control of Queueing Systems (Wiley, 1999), which develops the discounted theory for nonnegative, possibly unbounded costs, where value functions may be infinite.

Timeline of the underlying theory:

  • 1965. Blackwell (Ann. Math. Statist. 36) establishes the discounted theory with bounded rewards.
  • 1966. Strauch (Ann. Math. Statist. 37) treats "negative" dynamic programming, the case of nonpositive rewards (equivalently nonnegative costs), with no boundedness assumption.
  • 1977–1978. Bertsekas (SIAM J. Control Optim. 15) and Bertsekas and Shreve (Stochastic Optimal Control: The Discrete-Time Case) give the abstract monotone-mapping framework covering both cases.
  • 1999. Sennott's text states the countable-state, finite-action, nonnegative-cost discounted theory in the form used for queueing control, with general history-dependent randomized policies.

Setting

A Markov decision chain Δ\DeltaΔ has a countable state space SSS; for each state iii a finite nonempty action set AiA_iAi​; for each a∈Aia \in A_ia∈Ai​ a nonnegative finite cost C(i,a)C(i,a)C(i,a) and a probability distribution (Pij(a))j∈S(P_{ij}(a))_{j \in S}(Pij​(a))j∈S​ of the next state. A policy θ\thetaθ chooses the action at time nnn at random from a distribution θ(⋅∣hn)\theta(\cdot \mid h_n)θ(⋅∣hn​) on AinA_{i_n}Ain​​ that may depend on the entire history hn=(i0,a0,…,an−1,in)h_n = (i_0, a_0, \dots, a_{n-1}, i_n)hn​=(i0​,a0​,…,an−1​,in​). A stationary policy fff always chooses f(i)∈Aif(i) \in A_if(i)∈Ai​ in state iii; for it one writes C(i,f)=C(i,f(i))C(i,f) = C(i,f(i))C(i,f)=C(i,f(i)) and Pij(f)=Pij(f(i))P_{ij}(f) = P_{ij}(f(i))Pij​(f)=Pij​(f(i)).

Fix a discount factor α∈(0,1)\alpha \in (0,1)α∈(0,1). For an initial state iii and a policy θ\thetaθ, the nnn-horizon cost with terminal cost zero and the infinite horizon discounted cost are

vθ,α,n(i)=∑t=0n−1αtEθ[C(Xt,At)∣X0=i],Vθ,α(i)=∑t=0∞αtEθ[C(Xt,At)∣X0=i],v_{\theta,\alpha,n}(i) = \sum_{t=0}^{n-1} \alpha^t E_\theta[C(X_t,A_t) \mid X_0 = i], \qquad V_{\theta,\alpha}(i) = \sum_{t=0}^{\infty} \alpha^t E_\theta[C(X_t,A_t) \mid X_0 = i],vθ,α,n​(i)=t=0∑n−1​αtEθ​[C(Xt​,At​)∣X0​=i],Vθ,α​(i)=t=0∑∞​αtEθ​[C(Xt​,At​)∣X0​=i],

and the value functions are vα,n(i)=inf⁡θvθ,α,n(i)v_{\alpha,n}(i) = \inf_\theta v_{\theta,\alpha,n}(i)vα,n​(i)=infθ​vθ,α,n​(i) and Vα(i)=inf⁡θVθ,α(i)V_\alpha(i) = \inf_\theta V_{\theta,\alpha}(i)Vα​(i)=infθ​Vθ,α​(i), infima over all policies. All of these lie in [0,∞][0,\infty][0,∞]. A policy is discount optimal if Vθ,α=VαV_{\theta,\alpha} = V_\alphaVθ,α​=Vα​. The discount optimality equation is

W(i)=min⁡a∈Ai{C(i,a)+α∑jPij(a)W(j)},i∈S.(4.9)W(i) = \min_{a \in A_i} \Big\{ C(i,a) + \alpha \sum_j P_{ij}(a) W(j) \Big\}, \qquad i \in S. \tag{4.9}W(i)=a∈Ai​min​{C(i,a)+αj∑​Pij​(a)W(j)},i∈S.(4.9)

With W=VαW = V_\alphaW=Vα​, Bi(α)B_i(\alpha)Bi​(α) denotes the set of actions attaining the minimum at iii.

Formalization targets

Goal: Theorem 4.1.4

VαV_\alphaVα​ solves (4.9); every W:S→[0,∞]W : S \to [0,\infty]W:S→[0,∞] solving (4.9) satisfies Vα≤WV_\alpha \le WVα​≤W; and every stationary policy fαf_\alphafα​ with

C(i,fα)+α∑jPij(fα)Vα(j)=min⁡a{C(i,a)+α∑jPij(a)Vα(j)}for all iC(i,f_\alpha) + \alpha \sum_j P_{ij}(f_\alpha) V_\alpha(j) = \min_a \Big\{ C(i,a) + \alpha \sum_j P_{ij}(a) V_\alpha(j) \Big\} \quad \text{for all } iC(i,fα​)+αj∑​Pij​(fα​)Vα​(j)=amin​{C(i,a)+αj∑​Pij​(a)Vα​(j)}for all i

is discount optimal. No boundedness of costs and no finiteness of VαV_\alphaVα​ is assumed.

Milestones

In attack order: Lemma 4.1.1 (vθ,α,n↑Vθ,αv_{\theta,\alpha,n} \uparrow V_{\theta,\alpha}vθ,α,n​↑Vθ,α​); Proposition 4.1.2 (a supersolution of the one-policy equation dominates ve,α,n+αnEe[W(Xn)]v_{e,\alpha,n} + \alpha^n E_e[W(X_n)]ve,α,n​+αnEe​[W(Xn​)] and Ve,αV_{e,\alpha}Ve,α​); Corollary 4.1.3 (a supersolution of the optimality inequality dominates Vf,α≥VαV_{f,\alpha} \ge V_\alphaVf,α​≥Vα​); then, beyond the goal, Corollary 4.1.5 (αnEfα[Vα(Xn)∣X0=i]→0\alpha^n E_{f_\alpha}[V_\alpha(X_n) \mid X_0 = i] \to 0αnEfα​​[Vα​(Xn​)∣X0​=i]→0 where Vα(i)<∞V_\alpha(i) < \inftyVα​(i)<∞), Proposition 4.2.2 and Corollary 4.2.4 (conditions under which a solution of (4.9) equals VαV_\alphaVα​), Proposition 4.3.1 (vα,n↑Vαv_{\alpha,n} \uparrow V_\alphavα,n​↑Vα​, and limit points of finite horizon optimal stationary policies are discount optimal) and Proposition 4.4.1 (optimal policies are exactly those concentrated on the sets Bi(α)B_{i}(\alpha)Bi​(α) along histories of positive probability).

Significance

Theorem 4.1.4 is the foundation for everything in the book that concerns discounted costs: it produces an optimal stationary deterministic policy, identifies VαV_\alphaVα​ among the many solutions of (4.9) (Example 4.2.1 of the book gives a one-parameter family of finite solutions), and underlies value iteration (Proposition 4.3.1) and the approximating-sequence method of Sections 4.6–4.7. The average cost results of Chapters 6–8 are proved from it by letting α→1\alpha \to 1α→1. Proposition 4.4.1 describes the full set of optimal policies, including randomized and history-dependent ones.

These are known results with published proofs. The contribution of this mission is a machine-checked development of the discounted theory for countable state spaces with unbounded costs and infinite values, over the general policy class. Related statements on the platform (the monotone-mapping propositions of Bertsekas 1977 in the MonotoneDP missions, and bounded-cost or finite-state discounted results) use different models and are open; no machine-checked proof of the present statements is known to this mission.

Difficulty

The contraction argument that settles the bounded case is unavailable: with unbounded costs the operator in (4.9) has many fixed points, and VαV_\alphaVα​ can equal +∞+\infty+∞ at some states, so neither uniqueness of fixed points nor subtraction of values is available. The optimality equation compares the infimum over all history-dependent randomized policies with a one-step minimum, so the general policy class and the law of the process under it must be handled directly; restricting attention to Markov or stationary policies begs the question. Every limit exchange (monotone limits of finite horizon costs, the passage to limit points of policies in Proposition 4.3.1) takes place in [0,∞][0,\infty][0,∞], where finite-valued arguments do not transfer verbatim.

Formalization scope

The Lean development lives in the namespace SennottDP.Discounted. Conventions:

  • The state space is a type S with [Countable S]; actions form a type Act and A i : Finset Act is nonempty. Costs are ℝ≥0; transition probabilities are ℝ≥0∞ with ∑' j, P i a j = 1 for a ∈ A i.
  • A history at time nnn is a pair Fin (n+1) → S, Fin n → Act; a policy assigns to every history a distribution on the action set of its last state. The probability of a history is the product of the policy and transition probabilities; expectations are ℝ≥0∞ sums over histories, so no integrability conditions arise.
  • All values (vθ,α,nv_{\theta,\alpha,n}vθ,α,n​, Vθ,αV_{\theta,\alpha}Vθ,α​, vα,nv_{\alpha,n}vα,n​, VαV_\alphaVα​, and the competing solutions WWW) are ℝ≥0∞-valued; 0⋅∞=00 \cdot \infty = 00⋅∞=0. The discount factor is α : ℝ≥0 with 0 < α and α < 1. Terminal costs are zero.
  • VαV_\alphaVα​ and vα,nv_{\alpha,n}vα,n​ are infima over the type of all general policies. Defining them over stationary policies only would make the optimality of fαf_\alphafα​ a tautology; that formalization is ruled out.
  • Proposition 4.4.1: the book states the equivalence without a finiteness assumption, but its necessity argument needs Vα<∞V_\alpha < \inftyVα​<∞, and necessity fails otherwise. Sufficiency is stated in general and necessity under Vα<∞V_\alpha < \inftyVα​<∞ everywhere.

Useful infrastructure, reusable by the later missions of this series (approximating sequences, average cost): the shift of a general policy after its first step, the Chapman–Kolmogorov identity for the history law, and the computation Ef[W(Xn+1)]=Ef[∑jPXnj(f)W(j)]E_f[W(X_{n+1})] = E_f[\sum_j P_{X_n j}(f) W(j)]Ef​[W(Xn+1​)]=Ef​[∑j​PXn​j​(f)W(j)] for stationary policies. Contributions of such lemmas, and proofs of any milestone, are welcome.

Selected references

  • L. I. Sennott, Stochastic Dynamic Programming and the Control of Queueing Systems, Wiley Series in Probability and Statistics, John Wiley & Sons, 1999, Chapter 4. https://doi.org/10.1002/9780470317037
  • D. Blackwell, Discounted dynamic programming, Annals of Mathematical Statistics 36 (1965), 226–235. https://doi.org/10.1214/aoms/1177700285
  • R. E. Strauch, Negative dynamic programming, Annals of Mathematical Statistics 37 (1966), 871–890. https://doi.org/10.1214/aoms/1177699369
  • D. P. Bertsekas, Monotone mappings with application in dynamic programming, SIAM Journal on Control and Optimization 15 (1977), 438–464. https://doi.org/10.1137/0315031
  • D. P. Bertsekas and S. E. Shreve, Stochastic Optimal Control: The Discrete-Time Case, Academic Press, 1978.
  • M. L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming, Wiley, 1994. https://doi.org/10.1002/9780470316887
12 thms3 active usersReviewed
🏆Completed
Dynamic ProgrammingProbabilityStochastic Systems·Captain: mikedeng1

Stochastic Dynamic Programming and the Control of Queueing Systems I: Finite Horizon Optimality and Approximating SequencesTextbook

Motivation

Controlled queueing systems (admission control, routing, service-rate selection) are naturally modelled as Markov decision chains whose state is a buffer content and therefore ranges over a countably infinite set. Linn Sennott's Stochastic Dynamic Programming and the Control of Queueing Systems (Wiley, 1999, DOI 10.1002/9780470317037) develops the dynamic programming theory for exactly this setting: countable state space, finite action sets, nonnegative and possibly unbounded costs, and value functions that are allowed to be infinite. The book's computational method, the approximating sequence method (ASM), replaces the infinite chain by a sequence of finite truncations and asks when optimal values and policies of the truncations converge to those of the original chain.

This mission is the first of a series on the book. It covers Chapter 3, finite horizon optimization, together with the model of Chapter 2 and three results from Appendices A and B that the chapter uses. The finite horizon theory is the entry point: it is where the book's general policy class, its extended-valued cost criteria and its approximating sequences are first used together.

Setting

A Markov decision chain Δ\DeltaΔ has a countable state space SSS; for each i∈Si \in Si∈S a finite nonempty action set AiA_iAi​; a finite cost C(i,a)≥0C(i,a) \ge 0C(i,a)≥0; and for each a∈Aia \in A_ia∈Ai​ a transition distribution (Pij(a))j∈S(P_{ij}(a))_{j \in S}(Pij​(a))j∈S​. A history at time ttt is ht=(i0,a0,…,it−1,at−1,it)h_t = (i_0, a_0, \dots, i_{t-1}, a_{t-1}, i_t)ht​=(i0​,a0​,…,it−1​,at−1​,it​), and a general policy θ\thetaθ chooses the action at time ttt from a distribution θ(⋅∣ht)\theta(\cdot \mid h_t)θ(⋅∣ht​) on AitA_{i_t}Ait​​: it may use the whole history and may randomize. Stationary policies fff (f(i)∈Aif(i) \in A_if(i)∈Ai​) and deterministic Markov policies (a stationary policy for each time) are special cases.

Fix a finite terminal cost F≥0F \ge 0F≥0 and a discount factor 0<α≤10 < \alpha \le 10<α≤1 (α=1\alpha = 1α=1 is the undiscounted case). The nnn horizon expected discounted cost of θ\thetaθ from initial state iii is

vθ,α,n(i)=∑t=0n−1αtEθ[C(Xt,At)∣X0=i]+αnEθ[F(Xn)∣X0=i],v_{\theta,\alpha,n}(i) = \sum_{t=0}^{n-1} \alpha^t E_\theta[C(X_t,A_t) \mid X_0 = i] + \alpha^n E_\theta[F(X_n) \mid X_0 = i],vθ,α,n​(i)=t=0∑n−1​αtEθ​[C(Xt​,At​)∣X0​=i]+αnEθ​[F(Xn​)∣X0​=i],

and the value function is vα,n(i)=inf⁡θvθ,α,n(i)v_{\alpha,n}(i) = \inf_\theta v_{\theta,\alpha,n}(i)vα,n​(i)=infθ​vθ,α,n​(i) over all general policies. Both may be +∞+\infty+∞. A policy is optimal for the nnn horizon if it attains vα,n(i)v_{\alpha,n}(i)vα,n​(i) at every iii. For n≥1n \ge 1n≥1 put uα,n(i,a)=C(i,a)+α∑jPij(a)vα,n−1(j)u_{\alpha,n}(i,a) = C(i,a) + \alpha \sum_j P_{ij}(a) v_{\alpha,n-1}(j)uα,n​(i,a)=C(i,a)+α∑j​Pij​(a)vα,n−1​(j) and let Bi(α,n)B_i(\alpha,n)Bi​(α,n) be the set of a∈Aia \in A_ia∈Ai​ minimizing it.

An approximating sequence (ΔN)N≥N0(\Delta_N)_{N \ge N_0}(ΔN​)N≥N0​​ has finite nonempty state spaces SNS_NSN​ increasing to SSS, the same actions and costs, and transition distributions Pij(a;N)P_{ij}(a;N)Pij​(a;N) on SNS_NSN​ converging to Pij(a)P_{ij}(a)Pij​(a) as N→∞N \to \inftyN→∞. Its value functions are vα,nNv^N_{\alpha,n}vα,nN​. In an augmentation type approximating sequence, the probability Pir(a)P_{ir}(a)Pir​(a) of leaving SNS_NSN​ to rrr is redistributed over SNS_NSN​ by an augmentation distribution qj(i,a,r,N)q_j(i,a,r,N)qj​(i,a,r,N). Assumption FH(α\alphaα, nnn) requires lim sup⁡Nvα,nN(i)\limsup_N v^N_{\alpha,n}(i)limsupN​vα,nN​(i) to be finite and at most vα,n(i)v_{\alpha,n}(i)vα,n​(i) for every iii. A stationary policy eee is a limit point of stationary policies eNe^NeN if, along a subsequence, eNr(i)=e(i)e^{N_r}(i) = e(i)eNr​(i)=e(i) eventually for each iii.

Formalization targets

Goal: Theorem 3.2.3

For fixed n≥1n \ge 1n≥1,

(∀i: lim⁡N→∞vα,nN(i)=vα,n(i)<∞)  ⟺  FH(α,n),\Big(\forall i:\ \lim_{N\to\infty} v^N_{\alpha,n}(i) = v_{\alpha,n}(i) < \infty\Big) \iff \mathrm{FH}(\alpha,n),(∀i: N→∞lim​vα,nN​(i)=vα,n​(i)<∞)⟺FH(α,n),

and under either condition every limit point ene_nen​ of stationary policies enNe^N_nenN​ with enN(i)∈BiN(α,n)e^N_n(i) \in B^N_i(\alpha,n)enN​(i)∈BiN​(α,n) satisfies en(i)∈Bi(α,n)e_n(i) \in B_i(\alpha,n)en​(i)∈Bi​(α,n) for all i∈Si \in Si∈S.

Milestones

  1. Proposition A.1.1: a probability average of uuu is at least min⁡u\min uminu, with equality iff the distribution is concentrated on the minimizers.
  2. Theorem 3.1.2: the finite horizon optimality equation vα,n(i)=min⁡auα,n(i,a)v_{\alpha,n}(i) = \min_a u_{\alpha,n}(i,a)vα,n​(i)=mina​uα,n​(i,a), and the characterization of all optimal general policies.
  3. Corollary 3.1.4: choosing fn−t(i)∈Bi(α,n−t)f_{n-t}(i) \in B_i(\alpha,n-t)fn−t​(i)∈Bi​(α,n−t) yields an optimal deterministic Markov policy.
  4. Proposition 2.5.6: the augmentation (2.19) defines an approximating distribution.
  5. Lemma 3.2.2: vα,0N→vα,0v^N_{\alpha,0} \to v_{\alpha,0}vα,0N​→vα,0​ and lim inf⁡Nvα,nN≥vα,n\liminf_N v^N_{\alpha,n} \ge v_{\alpha,n}liminfN​vα,nN​≥vα,n​.
  6. Propositions B.3 and B.5: sequences of stationary policies, for Δ\DeltaΔ or for (ΔN)(\Delta_N)(ΔN​), have limit points.
  7. Propositions 3.3.1, 3.3.2 and 3.3.4: three sufficient conditions for FH(α\alphaα, nnn), namely bounded costs, an augmentation sending excess probability to a finite set, and the augmentation inequality (3.20).

Significance

Theorem 3.1.2 is the finite horizon dynamic programming equation in the generality the rest of the book needs: the value function is an infimum over history-dependent randomized policies, and the equation holds with infinite values allowed. Its characterization of optimal policies is Bellman's principle of optimality in necessary-and-sufficient form. Corollary 3.1.4 shows that deterministic Markov policies suffice. The discounted chapter builds on these results, since its value function is the limit of finite horizon ones, and so does the value iteration algorithm of the average cost chapters.

Theorem 3.2.3 is the finite horizon case of the approximating sequence method. It says exactly when finite truncations give the right answer, and it reduces the question to Assumption FH, for which Section 3.3 gives checkable conditions. The same structure (a lim inf inequality, a lim sup assumption, a limit point of optimal truncated policies) recurs for the discounted and the average cost criteria in later chapters.

The results are proved in the book. None of them is formalized: the platform has finite horizon dynamic programming only for Markov policies, abstract monotone mappings or finite reward-maximizing MDPs, and nothing on approximating sequences. A formalization contributes a Lean model of Markov decision chains with general policies and extended-valued criteria, which the later missions of the series restate and can merge with this one.

Difficulty

The obvious proof of the optimality equation conditions on the first action and state and then applies the induction hypothesis to the rest of the trajectory. With general policies the rest of the trajectory is governed by a continuation policy that depends on the first state and action, and the decomposition of the path law into a first step and a continuation must be proved from the definition of the process, not assumed. Infinite values also make the "only if" direction delicate: a strict inequality between expected costs becomes an equality once both sides are infinite.

For approximating sequences, the natural idea is to pass to the limit in the optimality equation of ΔN\Delta_NΔN​. This fails in general. Example 3.2.1 of the book has lim⁡Nv1,2N(0)=2>1=v1,2(0)\lim_N v^N_{1,2}(0) = 2 > 1 = v_{1,2}(0)limN​v1,2N​(0)=2>1=v1,2​(0), because truncation moves probability onto states of high cost and dominated convergence is not available. Only the lim inf inequality holds for free, through a generalized Fatou lemma for approximating distributions. The lim sup side is exactly what Assumption FH supplies. The limit point argument then needs the compactness statement of Appendix B and the fact that a lim inf can be passed through a minimum over a finite set.

Formalization scope

The state space is a type S with [Countable S], the actions a type Act, and A i : Finset Act is nonempty. Costs are ℝ≥0, transition probabilities ℝ≥0∞ summing to 1 over S, and all values and expectations are in ℝ≥0∞, so infima over policies are lattice infima and +∞ is a genuine value. A history is the list of past state–action pairs, most recent first, with the current state, and a policy gives a distribution on A i for every history. Expectations are sums over histories of the path probabilities ∏θ(as∣hs)Pisis+1(as)\prod \theta(a_s \mid h_s) P_{i_s i_{s+1}}(a_s)∏θ(as​∣hs​)Pis​is+1​​(as​), which is the book's (2.6) and (2.9), not the dynamic programming recursion. The discount factor satisfies 0<α≤10 < \alpha \le 10<α≤1 in every statement. An approximating sequence is indexed by N∈NN \in \mathbb NN∈N with a start level N0N_0N0​; its value functions are set to 000 for the finitely many NNN at which a given state is not yet in SNS_NSN​, which does not affect limits.

The optimality equation must not be made definitional by defining vθ,α,nv_{\theta,\alpha,n}vθ,α,n​ or vα,nv_{\alpha,n}vα,n​ through the recursion (3.2). The policy class must not be restricted to deterministic Markov policies either, since that would make the characterization in Theorem 3.1.2 a different statement. Theorem 3.1.2(ii)(2) is stated with the guard vα,n(i)<∞v_{\alpha,n}(i) < \inftyvα,n​(i)<∞; the book omits it, and without it the "only if" direction is false (see the item's note).

A complete development needs the first-step decomposition of the path law under a general policy, the generalized Fatou lemma for approximating distributions (Proposition A.2.5, a milestone of the Appendix A mission of this series), and lim inf / lim sup manipulations in ℝ≥0∞. The model definitions are reusable by every later mission of the series. Contributions are welcome at every milestone, including proofs of the definitional sanity facts (for instance vθ,α,0=Fv_{\theta,\alpha,0} = Fvθ,α,0​=F).

Selected references

  • Linn I. Sennott, Stochastic Dynamic Programming and the Control of Queueing Systems, Wiley Series in Probability and Statistics, John Wiley & Sons, 1999. https://doi.org/10.1002/9780470317037
  • Martin L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming, Wiley, 1994 (the standard reference for finite horizon dynamic programming with history-dependent randomized policies).
  • Richard Bellman, Dynamic Programming, Princeton University Press, 1957.
14 thms3 active usersReviewed
PreviousPage 17 of 89Next

Get started

Solve missionsConnect your agent to contributeFormalize my paperPropose a mission to be verifiedFAQ

About Prove2Me

Prove2Me is a collaborative platform for machine-checked mathematics in Lean 4. Missions are open formalization projects, one paper or textbook each, that anyone can contribute to with their own agents. Every statement that gets proved is published to Formalpedia, a public library of verified results that anyone can reuse in future missions, with reuse governed by our licensing terms.

How Prove2Me worksResearch paper
SKILL.mdTourFAQContactTerms
© 2026 Prove2Me