Multivariate Monotone Inclusions in Saddle Form: The Asynchronous Block-Iterative Saddle Splitting Algorithm Converges Weakly to a Primal-Dual Kuhn–Tucker PointResearch Paper
Motivation
Many problems in convex optimization, variational inequalities, game theory and signal processing are systems of monotone inclusions coupled by linear operators: several unknowns in different Hilbert spaces, each subject to its own maximally monotone, cocoercive and Lipschitzian operators, and tied together through linear maps and composite terms. Bùi and Combettes (arXiv:2002.06135, Mathematics of Operations Research, 2022) introduce a single model, Problem 1.1, that contains a large part of these formulations and design algorithms for it that are fully split (each operator is used separately, set-valued operators through their resolvents and single-valued ones through forward evaluations), block-iterative (only some blocks are updated at each iteration) and asynchronous (updates may use information computed up to iterations earlier).
The paper places itself after a line of primal-dual splitting methods: Briceño-Arias and Combettes (SIAM J. Optim. 2011, doi:10.1137/10081602x) for a two-operator monotone+skew model, Alotaibi, Combettes and Shahzad (SIAM J. Optim. 2014) for Fejér approximation of a Kuhn–Tucker set, Combettes and Eckstein (Math. Program. B168, 2018) for asynchronous block-iterative primal-dual decomposition, and the projective splitting methods of Johnstone and Eckstein (2020–2021). These earlier methods work in the primal-dual space . The new idea is to reformulate the problem as finding a zero of a saddle operator on a larger space and to approximate that zero set from outside by half-spaces.
Setting
Let and be finite index sets and let , be real Hilbert spaces, with Hilbert direct sums and . For every and the data are:
- maximally monotone, cocoercive with constant , monotone and -Lipschitzian, and such that is monotone and -Lipschitzian on ;
- , maximally monotone on , , cocoercive with constants , and , monotone and Lipschitzian with constants ;
- bounded linear , and vectors , .
The parallel sum of is . Write and . The primal problem (1.1) asks for with
Its set of solutions is . The dual problem (1.2) asks for such that some satisfies and ; its set of solutions is . The Kuhn–Tucker set (2.13) collects the pairs satisfying the first condition at and .
On the saddle operator (2.11) maps to the product of the sets , , and . It splits as (Definition 2.4), where collects the cocoercive operators.
Algorithm 3.4 runs, at iteration , over activated blocks , . For it computes a resolvent step from the delayed iterate ; for it computes , from resolvents of , at the delayed iterate . Blocks that are not activated keep their previous values. From these quantities it forms a half-space containing , measured by a number that involves . If , it moves toward that half-space with relaxation ; otherwise the iterate stays put.
Formalization targets
Goal: Theorem 3.5
Under Assumptions 3.1–3.3 (step-size ranges tied to , and by (3.1); every block activated at least once in every consecutive iterations; delays at most ), if the dual problem has a solution, then every run of Algorithm 3.4 satisfies
together with the analogous statements for and . There exist and with such that
Milestones
In attack order:
- Proposition 2.2(i)–(ii): is maximally monotone and is closed and convex.
- Proposition 2.2(iii)–(iv): zeros of project into , and .
- Proposition 2.5: , maximally monotone, -cocoercive, and the half-spaces containing .
- Lemma A.4: the relaxed half-space outer-approximation iteration (A.3) is Fejér monotone and converges weakly when every weak sequential cluster point lies in .
- Proposition 2.6: the iteration (2.32) on converges weakly to a zero of when its three stated residual conditions hold.
- Lemmas A.1, A.2 and A.3: boundedness of resolvent sequences, strong monotonicity of , and the vanishing of the delayed differences .
Significance
Theorem 3.5 gives weak convergence of a method that uses each operator of Problem 1.1 separately, needs no inversion of linear operators and no knowledge of , and updates nonempty subsets of blocks subject to the coverage windows and bounded delays of Assumptions 3.2–3.3. It produces primal and dual solutions together. Section 4 of the paper derives from it algorithms for variational inequalities over sums of sets and for multivariate convex minimization with infimal convolutions. The hypothesis is dual solvability. Remark 2.3 shows that primal solvability alone is not enough: there are instances with and .
The theorem is proved in the paper. A formalization adds machine-checked foundations that later work in splitting methods can reuse: the saddle operator and its maximal monotonicity; the Fejér outer-approximation principle in Hilbert space (Lemma A.4, cited by the paper from an encyclopedia entry); and the asynchrony lemma (Lemma A.3). To our knowledge none of these results has been formalized before.
Difficulty
The obvious approach would be to show that Algorithm 3.4 is a Fejér-monotone iteration toward in . That approach fails: for the general Problem 1.1, no half-space separating the current point from can be computed from separate operator evaluations (Remark 2.3(iii)). The algorithm instead targets in the larger space . The link back to , and is Proposition 2.2.
The second difficulty is asynchrony. The half-space at iteration is built from points computed at earlier iterations, with stale step sizes. To show that the weak cluster points are zeros of , one has to control these delays (Lemma A.3) and use the cocoercive and Lipschitzian terms uniformly over the step-size ranges (Lemmas A.1–A.2).
Formalization scope
- Spaces. Every space is a complete real inner product space. and are
PiLp 2over finite index types, and is a nestedWithLp 2product, so all norms and inner products are the Hilbert ones (not sup norms). - Operators. Set-valued operators are maps
E → Set E. Monotonicity, maximal monotonicity,zer, cocoercivity, resolvents and weak convergence are the publishedThreeOpSplitting.Convergencedefinitions; inverses and sums of set-valued maps are the publishedDouglasRachfordPPA.GenDRones. - Runs. A run of Algorithm 3.4 is a predicate on given sequences that pins every assignment of (3.3). This includes the delayed indices inside the step sizes, the carry-over of inactive blocks (stated at ), the recomputation of for every , and with the factor at the current iterates. The resolvents are given maps constrained by the resolvent identity at every positive step.
- Constants and assumptions. is pinned as the least cocoercivity constant. The delay bound is written without natural-number subtraction. "" in Proposition 2.6 is stated as "eventually for every ".
- Ruling out trivial readings. The run predicate determines every assignment of (3.3), so no free choice of outputs makes the theorem trivial. The hypothesis is dual solvability exactly as in (1.2), not the stronger-looking (equivalent only by Proposition 2.2). The conclusion is weak convergence to a pair that is simultaneously primal, dual and Kuhn–Tucker. A sorry-free check shows that the goal's hypotheses are jointly satisfiable.
- Not included. Theorem 3.7 (strong convergence of a Haugazeau-type variant) with Proposition 2.7 and Lemma A.5, Proposition 2.2(v), and the applications of Section 4.
- Reusable parts. The saddle operator and its splitting, Lemma A.4, Lemma A.3, and Lemmas A.1–A.2 are stated in general form and can serve other splitting missions. Contributions of auxiliary results are welcome, for example demiclosedness of maximally monotone operators, Minty's theorem in the published encoding, and Fejér-monotone weak convergence (Opial).
Selected references
- M. N. Bùi and P. L. Combettes, Multivariate monotone inclusions in saddle form, Mathematics of Operations Research, 2022; preprint arXiv:2002.06135v3, 2021. https://arxiv.org/abs/2002.06135
- H. H. Bauschke and P. L. Combettes, Convex Analysis and Monotone Operator Theory in Hilbert Spaces, 2nd ed., Springer, 2017. https://doi.org/10.1007/978-3-319-48311-5
- L. M. Briceño-Arias and P. L. Combettes, A monotone+skew splitting model for composite monotone inclusions in duality, SIAM Journal on Optimization 21, 2011. https://doi.org/10.1137/10081602x
- P. L. Combettes and J. Eckstein, Asynchronous block-iterative primal-dual decomposition methods for monotone inclusions, Mathematical Programming B168, pp. 645–672, 2018 (reference [25] of arXiv:2002.06135).
- P. L. Combettes, Fejér-monotonicity in convex optimization, in: Encyclopedia of Optimization, vol. 2, pp. 106–114, Springer, 2001 (reference [22] of arXiv:2002.06135; source of Lemma A.4).