A Dual Algorithm for the Solution of Nonlinear Variational Problems via Finite Element Approximation 1: For 0 < ρ < 2r, the Modified Dual Algorithm Converges Strongly to (v*, Av*)Research Paper
Motivation
Many problems in mechanics and optimal control are posed as the minimization of a convex functional of the form , where is a differential operator and is non-smooth: an obstacle, a friction term, a plasticity threshold. Splitting the variable, , separates the linear operator from the non-smooth function, and a Lagrange multiplier for the constraint turns the problem into a sequence of two simpler ones: a linear system in and a pointwise nonlinear problem in . The scheme that alternates between the two and then updates the multiplier is now called the alternating direction method of multipliers (ADMM). It is one of the standard algorithms of large-scale convex optimization, statistics and imaging (Boyd, Parikh, Chu, Peleato and Eckstein, 2011).
Gabay and Mercier (IRIA report RR-126, 1975; Comput. Math. Appl., 1976) gave a convergence proof of this method in Hilbert space, for a stepsize anywhere in , where is the penalty parameter of the augmented Lagrangian.
Timeline.
- 1969: Hestenes and Powell introduce the augmented Lagrangian (method of multipliers).
- 1975: Glowinski and Marrocco propose the alternating scheme for nonlinear Dirichlet problems.
- 1975–76: Gabay and Mercier prove strong convergence of the primal iterates for under strong monotonicity of (this mission).
- 1983: Gabay identifies ADMM with Douglas–Rachford splitting applied to the dual.
- 1992: Eckstein and Bertsekas extend convergence to maximal monotone operators, without strong monotonicity, for .
Setting
and are real Hilbert spaces; and are the inner product and norm of , the norm of . is a continuous linear operator and is a continuous linear functional on , with value at . The dual of is identified with . The function is a sum where
- is convex and continuously differentiable, and its gradient is strongly monotone: for some (2.3);
- is proper (never , somewhere finite), convex and lower semicontinuous;
- for some (2.5);
- some lies in the interior of (qualification).
The problem is
and a solution is a point where this value is finite and minimal. The Lagrangian and augmented Lagrangian are, for and ,
The modified dual algorithm (3.4) starts from any and, for ,
- finds with for all (minimization of in );
- finds with (minimization of in );
- sets .
denotes the orthogonal projection of onto the range , which is closed by (2.5).
Formalization targets
Goal: Theorem 3.1
For and every stepsize with
every run of the algorithm satisfies
where is the unique solution of . The multipliers need not converge.
Milestones
- Proposition 2.1: has a unique solution.
- Theorem 2.1: a saddle point of consists of the solution , and with ; and a saddle point exists.
- Theorem 2.2: for , and have the same saddle points.
- (3.8)–(3.10): a saddle point is a fixed point of Steps 1–2.
- (3.11)–(3.12): .
- (3.19)–(3.20): the energy estimate
and its sum over . 7. for . 8. (3.22)–(3.23): a contraction-plus-summable-forcing recursion for , and the scalar lemma that such recursions tend to . 9. and for .
Significance
The theorem says that ADMM converges without any step-size tuning beyond , and that the primal iterates converge in norm in infinite dimension, which is what makes the method usable for finite element discretizations of variational problems (the companion mission on internal approximations treats the discretization). It covers obstacle problems, Bingham fluids and elasto-plastic torsion, where is an indicator or a non-differentiable norm.
The result is proved in the 1975 report. To our knowledge it has no machine-checked proof. Formalizing it adds a library-level convergence theorem for ADMM in Hilbert space with -valued convex functions, a saddle-point existence theorem for linearly constrained convex problems, and the energy estimate (3.19) as a reusable lemma. The formalization also fixes two defects of the printed text (see Formalization scope): a sign in Step 1 and a qualification hypothesis that is too weak.
Difficulty
The energy estimate (3.19) only controls the multiplier through its component in , and only the -error through . The natural first idea, a Lyapunov function in alone that decreases at each step, does not work for : the component obeys a separate recursion whose contraction factor is below one only for , and whose forcing term must be shown summable from the -estimate. The existence of a multiplier, i.e. of a saddle point, is a separate difficulty: it needs the subdifferential chain rule in infinite dimension, under a constraint qualification.
Formalization scope
Lean conventions:
- , are
InnerProductSpace ℝwithCompleteSpace; multipliers are elements of ; is aStrongDual ℝ V. - (a Gateaux-differentiable function is finite), with a gradient
f₁'at every point (HasGradientAt) that is continuous; "weakly continuous on finite-dimensional subspaces" is then automatic. -
EReal, withIsProperFn,IsConvexFn,IsSubgradientfrom the publishedInertialFB.IFB.ConvexAnalysisand Mathlib'sLowerSemicontinuous. No subtraction inERealoccurs: the variational inequalities (3.6), (3.9) are stated as subgradient inclusions, and the Lagrangians as a real part plus . - A solution of has finite value. A run is any triple of sequences indexed by satisfying Steps 1–3 for every ; the start is arbitrary, and the well-posedness of the steps is not assumed or used.
- is the orthogonal projection onto the closure of , equal to under (2.5).
- Strong convergence is convergence in norm; "bounded" is
Bornology.IsBounded (Set.range λ).
Corrections to the printed text, each explained in the item's statement:
- Step 1 (3.5) and (3.8) carry . The paper prints there and in Remark 1 of p. 15, which is inconsistent with and : with the printed sign the iteration solves the problem with .
- The qualification (2.4), "", is strengthened to . With (2.4) alone, Theorem 2.1's saddle point need not exist and the multipliers of Theorem 3.1 need not be bounded: , , , , the indicator of the disc of centre and radius , .
- Proposition 2.1 assumes that is somewhere finite, which its proof requires.
Ruled out: the solution is a hypothesis of the goal, so the hypotheses must not be contradictory. They are met by , , , , , and a proper excludes the degenerate reading in which every point "solves" with value .
A complete development needs: existence of minimizers of coercive convex l.s.c. functions on a Hilbert space; the subdifferential sum and chain rules under a continuity qualification; orthogonal projections onto closed ranges; and elementary facts about summable sequences. The saddle-point theory (Theorems 2.1–2.2) and the scalar lemma (3.23) are reusable beyond this mission. Proofs of any milestone are welcome, as are proofs of the goal by a different route.
Selected references
- D. Gabay and B. Mercier, A dual algorithm for the solution of non linear variational problems via finite element approximation, IRIA Rapport de Recherche 126, 1975. https://hal.science/hal-04716124v1 ; journal version: Computers & Mathematics with Applications 2 (1976) 17–40, https://doi.org/10.1016/0898-1221(76)90003-1
- R. Glowinski and A. Marrocco, Sur l'approximation, par éléments finis d'ordre un, et la résolution, par pénalisation-dualité, d'une classe de problèmes de Dirichlet non linéaires, RAIRO Analyse Numérique 9 (1975) 41–76. https://doi.org/10.1051/m2an/197509R200411
- M. R. Hestenes, Multiplier and gradient methods, J. Optim. Theory Appl. 4 (1969) 303–320. https://doi.org/10.1007/BF00927673
- I. Ekeland and R. Temam, Analyse convexe et problèmes variationnels, Dunod, 1974 (the paper's [E]).
- J. Eckstein and D. P. Bertsekas, On the Douglas–Rachford splitting method and the proximal point algorithm for maximal monotone operators, Math. Programming 55 (1992) 293–318. https://doi.org/10.1007/BF01581204
- S. Boyd, N. Parikh, E. Chu, B. Peleato and J. Eckstein, Distributed optimization and statistical learning via the alternating direction method of multipliers, Found. Trends Mach. Learn. 3 (2011) 1–122. https://doi.org/10.1561/2200000016