Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers I: Under a Lagrangian Saddle Point, ADMM Residuals Vanish and Objective Values ConvergeTextbook
Why ADMM convergence matters
The alternating direction method of multipliers (ADMM) is one of the most widely used algorithms for large-scale convex optimization in statistics, machine learning and signal processing. Its appeal is decomposition: a problem whose objective splits into two parts, coupled only by a linear constraint, is solved by alternately minimizing over each part and updating a dual variable. Each subproblem is often a proximity operator, a projection or a small linear system, so ADMM turns problems such as the lasso, sparse inverse covariance selection and consensus fitting across many machines into sequences of simple steps. The survey of Boyd, Parikh, Chu, Peleato and Eckstein (DOI 10.1561/2200000016) made the method standard, and every algorithm in its later chapters is justified by one convergence result, stated in §3.2.1 and proved in Appendix A. This mission formalizes that result and the inequalities behind it.
The method goes back to Gabay and Mercier (1976). Eckstein and Bertsekas (1992) proved convergence through the theory of maximal monotone operators, by identifying ADMM with Douglas–Rachford splitting applied to the dual problem. The proof in Appendix A of the survey is different: it is a direct Lyapunov argument in finite dimensions that uses only convexity and elementary algebra.
Setting
Let and , let , and . The problem is
with optimal value . The augmented Lagrangian with parameter is
and is the ordinary Lagrangian. For , ADMM generates iterates by
The state is ; plays no role. The primal residual is , the dual residual is , and .
Two assumptions are made. Assumption 1: and are closed, proper and convex. Assumption 2: has a saddle point , i.e. for all . Nothing is assumed about the ranks of and . The convergence proof uses the Lyapunov function
Formalization targets
Goal: residual and objective convergence (§3.2.1, p. 17; Appendix A, p. 106)
Under Assumptions 1 and 2 and for , every ADMM run satisfies
The statement fixes no rate and no constant. It does not claim convergence of or , which fails in general (p. 17).
Milestones
In the order of Appendix A:
- (3.10) holds along the iteration: (§3.3, p. 18);
- the dual residual inclusion (§3.3, p. 18);
- (A.3) ;
- (A.2) ;
- (3.11) ;
- the monotonicity step for (p. 110);
- (A.6) ;
- (A.1) for ;
- the summed bound , with and ;
- the stopping-rule bound when (§3.3.1, p. 19).
Significance
The theorem is what licenses every specialized ADMM of the survey (lasso, basis pursuit, covariance selection, consensus and sharing, distributed model fitting): each of these chapters only computes the subproblem solutions, and correctness of the overall method is inherited from §3.2.1. Inequality (3.11) and its corollary in §3.3.1 justify the primal/dual residual stopping criterion (3.12) used in practice: small residuals certify small suboptimality.
The result itself is classical and proved; it is not open. To our knowledge no machine-checked proof of convex two-block ADMM convergence exists in Lean's Mathlib. Formalizing it produces a reusable development of the augmented Lagrangian method with explicit domain handling for extended-valued convex functions, and checks a proof whose index bookkeeping the printed text leaves loose (the monotonicity step and (A.1) need , see below).
Difficulty
The obvious argument, "the subproblem optimality conditions plus the saddle point give a decreasing quantity", works only once the right Lyapunov function is found and the cross term is controlled. That term has no sign from the optimality conditions of a single iteration, and at the first iteration, where is an arbitrary starting point, the decrease (A.1) can genuinely fail. A second obstacle is that and need not converge or even be bounded when or is rank deficient, so objective convergence cannot pass through limits of the primal iterates; it has to come from the two-sided bounds (A.2) and (A.3). Finally, the subdifferential sum rule used to linearize each subproblem must be handled for functions taking the value .
Formalization scope
- Vectors are
EuclideanSpace ℝ (Fin n); matrices act throughMatrix.toEuclideanLin; the problem data are bundled in a structureProblem n m p. - An extended-valued is encoded by its effective domain and its real values on . Assumption 1 is: nonempty, convex on , epigraph over closed. All minimizations, and the saddle-point inequality in , range over the domains; this is equivalent to the book's formulation with .
- is the infimum over feasible points of the domains, not by definition.
- The run is a hypothesis. An ADMM run is any triple of sequences satisfying (3.2)–(3.4) exactly. The book asserts on p. 16 that Assumption 1 makes the subproblems solvable; this is false in general (, ), so no statement constructs iterates. A formalization that defined the iterates by choice, or required of a run, would trivialize the residual claim and is ruled out.
- Indices: starts at the book's . Statements about , , , are for (written with ). The monotonicity step and (A.1) are stated for ; the book states (A.1) without a range, and at with an arbitrary it can fail. The summed bound accordingly starts at and is bounded by instead of ; it is stated as a bound on every partial sum.
- Subdifferentials (milestones 1–2) use the published
ShorNonsmooth.Subdiff.subdifferentialrelative to the domain. - The dual-variable convergence listed in §3.2.1 is not proved in the book and is not a target.
A complete development needs the subdifferential of a convex function plus a differentiable quadratic, the first-order characterization of a constrained minimizer, and elementary limits; all of this is reusable for the later missions of the series, which take ADMM runs as given. Proofs of individual milestones are welcome independently.
Selected references
- S. Boyd, N. Parikh, E. Chu, B. Peleato, J. Eckstein, Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers, Foundations and Trends in Machine Learning 3(1), 2011, pp. 1–122. https://doi.org/10.1561/2200000016
- D. Gabay, B. Mercier, A dual algorithm for the solution of nonlinear variational problems via finite element approximation, Computers & Mathematics with Applications 2(1), 1976, pp. 17–40. https://doi.org/10.1016/0898-1221(76)90003-1
- J. Eckstein, D. P. Bertsekas, On the Douglas–Rachford splitting method and the proximal point algorithm for maximal monotone operators, Mathematical Programming 55, 1992, pp. 293–318. https://doi.org/10.1007/BF01581204
- R. T. Rockafellar, Convex Analysis, Princeton University Press, 1970. https://doi.org/10.1515/9781400873173