Motivation
Linear programs at very large scale are increasingly solved by first-order primal-dual methods, which use only matrix-vector products instead of matrix factorizations. Such methods parallelize well and have a small memory footprint, but on their own they are slow to reach moderate or high accuracy. Applegate, Hinder, Lu and Lubin (arXiv:2105.12715, Math. Program. 2022) showed that periodically restarting a primal-dual method from the average of its iterates turns its sublinear ergodic rate into a linear rate on sharp problems, a class that contains every linear program. The analysis underlies PDLP, a restarted PDHG solver studied computationally by Applegate et al. (arXiv:2106.04756) and shipped in Google OR-Tools.
This mission formalizes the paper's first convergence theorem: with restarts at a fixed frequency, the distance from the restart points to the solution set shrinks by a factor β per restart.
Setting
Let X⊆Rn and Y⊆Rm be closed convex sets and let L(x,y) be differentiable, convex in x and concave in y. Problem (1) is
x∈Xminy∈YmaxL(x,y),
with feasible region Z=X×Y and primal-dual points z=(x,y). Its solution set Z⋆ is the set of saddle points z⋆∈Z: L(x⋆,y′)≤L(x⋆,y⋆)≤L(x′,y⋆) for all x′∈X, y′∈Y. Throughout, Z⋆=∅.
Distances are measured in a semi-norm ∥⋅∥, which the paper allows in order to cover ADMM. Write dist(z,Z⋆)=infz⋆∈Z⋆∥z−z⋆∥, diam(S)=sups,s′∈S∥s−s′∥, and Wr(z)={z^∈Z:∥z−z^∥≤r}. The primal-dual gap (3) at z is supz^∈Z{L(x,y^)−L(x^,y)}, and the normalized duality gap (4a)–(4b) is
ρr(z)=r1z^∈Wr(z)sup{L(x,y^)−L(x^,y)}(r>0),ρ0(z)=r→0+limsupρr(z).
By Definition 1, problem (1) is α-sharp on S⊆Z if αdist(z,Z⋆)≤ρr(z) for all z∈S and all r∈(0,diam(S)].
The base method is abstract. From a start z0 it outputs averaged points zˉt, t≥1. It has Property 3 with constants q,C>0 if, for every start z0∈Z and every t≥1, zˉt∈Z and
i. ρ∥zˉt−z0∥(zˉt)≤2C∥zˉt−z0∥/t,
ii. ∥zˉt−z0∥≤(q+2)dist(z0,Z⋆).
Algorithm 1 with fixed-frequency restarts (29) runs the base method from zn,0 for
t⋆=⌈αβ2C(q+2)⌉
steps, sets zn+1,0=zˉn,t⋆, and repeats, with β∈(0,1).
Formalization targets
Goal: Theorem 1 (p. 17)
If the base method has Property 3 and (1) is α-sharp on WR(z0,0) with R=1−βq+2dist(z0,0,Z⋆), then for every n≥0
dist(zn,0,Z⋆)≤βndist(z0,0,Z⋆).(31)
Milestones
- Proposition 5 (p. 11): for z∈Z, ρr(z) is non-increasing in r∈[0,∞).
- Proposition 6 (p. 11): for z∈Z and r≥0, the primal-dual gap at z is zero if and only if ρr(z)=0.
- Proposition 8 (p. 17): under Property 3, ∥zˉt−z0∥=0 implies zˉt∈Z⋆.
- First display of the proof of Theorem 1 (p. 17): if (31) holds for all n≤N, then ∥zN+1,0−z0,0∥≤R, so zN+1,0∈WR(z0,0).
- Display (32) (p. 17): if zN,0,zN+1,0∈WR(z0,0), then
dist(zN+1,0,Z⋆)≤βdist(zN,0,Z⋆).
Companion statements
Proposition 2 (p. 8) and Proposition 4 (p. 10) show where Property 3 comes from. A generic algorithm (6) with iterates zt and targets z^t has Property 1 with C if L(x^t+1,y)−L(x,y^t+1)≤2C∥z−zt∥2−2C∥z−zt+1∥2 for every z∈Z (12). It has Property 2 with q if ∥z^t−zt∥≤qdist(zt,Z⋆) or ∥z^t−zt−1∥≤qdist(zt,Z⋆).
- Proposition 2: Property 1 gives non-expansiveness ∥zt+1−z⋆∥≤∥zt−z⋆∥ and the ergodic rate L(xˉt,y)−L(x,yˉt)≤C∥z−z0∥2/(2t).
- Proposition 4: Properties 1 and 2 give Property 3 for the averages zˉt=t1∑i=1tz^i, with the same q and C.
Fact 1 (p. 11), that rρr(z) is non-decreasing in r, is included as well.
Significance
Theorem 1 converts any primal-dual method with an O(1/t) ergodic rate on the normalized duality gap into a linearly convergent method on sharp problems. The rate is O(αClogϵ1) iterations to reach dist(z,Z⋆)≤ϵ (Remark 3). Through Proposition 4 it applies to PPM, EGM, PDHG and ADMM. Combined with the paper's Lemma 5, which shows that linear programs are sharp with α the inverse of a Hoffman constant, it yields linear convergence of restarted PDHG for LP. The paper also shows that this rate matches a lower bound for a general class of primal-dual methods.
The result is proved in the paper. As far as is known, no part of it, and no normalized duality gap or primal-dual sharpness notion, has a machine-checked proof. The mission adds a checked version of the restart argument, with every degenerate case resolved: zero displacements, unbounded balls, and semi-norms that are not norms.
Difficulty
Each proof is short, but the statements are delicate in ways the paper passes over.
- The paper's argmax over Wr(z) need not exist under a semi-norm, and ρr(z) may be +∞. Proposition 5 therefore has to be proved with suprema, through the convex-combination scaling z+r2r1(z^−z), not with a maximizer.
- Sharpness is only available at radii in (0,diam(WR(z0,0))] and only at points of WR(z0,0). So the induction in Theorem 1 must carry two facts together: the contraction (31), and the fact that every restart point stays in WR(z0,0).
- The step size r=∥zN+1,0−zN,0∥ may be zero, which is why Proposition 8, and through it Propositions 5 and 6 at r=0, is needed.
A direct contraction argument that applies sharpness at an arbitrary radius, or on all of Z, does not prove the theorem as stated.
Formalization scope
- Rn and Rm are
EuclideanSpace ℝ (Fin n) and EuclideanSpace ℝ (Fin m). n=0 and m=0 are allowed.
- The semi-norm is an arbitrary
Seminorm ℝ on their product, never the product's built-in norm.
- The primal-dual gap, ρr and diam are suprema in
EReal, so unbounded sets give +∞ and not a junk 0. dist(⋅,Z⋆) is a real infimum over the nonempty set Z⋆.
- The base method is a map from starts to sets of admissible output sequences, because a primal-dual step can be an argmin and need not be single-valued. Property 3 quantifies over every start in Z and every run, with q and C fixed first.
- The restart length is the natural number ⌈2C(q+2)/(αβ)⌉, and every inner loop has exactly this length.
Standing assumptions and added hypotheses:
- X and Y are closed and convex, and L is convex-concave and differentiable on the whole space.
- Z⋆=∅ replaces the page's "non-empty feasible region and bounded optimal value".
- α>0 is assumed.
- Property 1 is required for every t≥0, although the paper's N starts at one: the proof of Proposition 2 telescopes from z0.
- The iterates of a generic algorithm stay in Z, which the page leaves implicit.
Departures from the page:
- Proposition 8 asserts only zˉt∈Z⋆. The page's "zˉn,t=zn,0" does not follow under a semi-norm.
- Milestones that do not use a standing assumption drop it.
The goal is not trivialized by its hypotheses. A companion check file exhibits L(x,y)=xy on R×R, with the sup norm and an oracle base method, on which every hypothesis of Theorem 1 holds and dist(z0,0,Z⋆)=1. Sharpness is assumed only where Definition 1 and the theorem assume it: replacing WR(z0,0) by Z, or the radius range by all r>0, would weaken the theorem.
A complete development needs elementary facts about suprema in EReal, Jensen's inequality for convex-concave functions on averages, and geometric sums. The definitions of ρr, sharpness and Property 3 are reusable for the adaptive-restart theorem of the same paper (Theorem 2, a separate mission) and for other restart analyses. Proofs of any milestone or companion are welcome.
Selected references
- D. Applegate, O. Hinder, H. Lu, M. Lubin, Faster First-Order Primal-Dual Methods for Linear Programming using Restarts and Sharpness, Math. Program. (2022); arXiv:2105.12715v4. https://arxiv.org/abs/2105.12715 · https://doi.org/10.1007/s10107-022-01901-9
- D. Applegate, M. Díaz, O. Hinder, H. Lu, M. Lubin, B. O'Donoghue, W. Schudy, Practical Large-Scale Linear Programming using Primal-Dual Hybrid Gradient, NeurIPS 2021. https://arxiv.org/abs/2106.04756
- A. Chambolle, T. Pock, A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging, J. Math. Imaging Vision 40 (2011). https://doi.org/10.1007/s10851-010-0251-1
- A. Nemirovski, Prox-Method with Rate of Convergence O(1/t) for Variational Inequalities with Lipschitz Continuous Monotone Operators and Smooth Convex-Concave Saddle Point Problems, SIAM J. Optim. 15 (2004). https://doi.org/10.1137/S1052623403425629