Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Łojasiewicz Condition III: Randomized Coordinate Descent Has Expected Rate (1 − μ/(dL))^kResearch Paper
Motivation
Coordinate descent changes one component of a parameter vector at a time. This matters when evaluating or updating a single component is cheaper than handling the full vector, for example in problems with sparse features. The coordinate chosen at each step can be random or selected from the current gradient. Karimi, Nutini, and Schmidt study these choices for objectives that satisfy a Polyak–Łojasiewicz (PL) inequality, a condition that controls objective error by the size of the gradient without assuming convexity. Their Theorem 3 gives an explicit expected rate for uniform random coordinates, while §3.2 gives a rate for the greedy Gauss–Southwell choice. Karimi, Nutini, and Schmidt (2016), §3
The question is how much of the familiar full-gradient linear rate survives when an update sees only one coordinate. The theorem answers this with a dimension factor: each uniform draw accesses one of coordinates, and the rate depends on . The same paper relates the greedy rule to earlier work on coordinate descent and boosting; this mission treats its stated greedy rate as a companion result, using the same coordinate update. Karimi, Nutini, and Schmidt (2016), §§3.1–3.2
Setting
Let , and let be differentiable. Write for coordinate of its gradient and for the corresponding standard basis vector. The paper's coordinate-wise upper bound (5), with , is
Here ranges over the coordinates. The bound concerns motion along one coordinate. It is the exact smoothness condition used in Theorem 3. Let be a global minimizer and put ; thus the solution set is nonempty and is an attained optimal value. The PL inequality (3) with says
The randomized coordinate-descent step (6) draws uniformly from the coordinates, independently at each time, and sets . The initial point is arbitrary. For draws, expectation means the average over all words , each assigned weight . In particular, at the empty word has weight one and the iterate is .
The Gauss–Southwell rule instead chooses a coordinate with the largest ; ties may be resolved in any way. Its rate uses a PL inequality in the maximum-coordinate norm, , with constant . Karimi, Nutini, and Schmidt (2016), pp. 5–6
Formalization targets
Theorem 3: uniform random coordinates
For every , the goal is the paper's expected suboptimality bound
The target includes the specific update (6), the coordinate condition (5), and the PL inequality (3). The three milestones are the paper's displayed one-coordinate decrease, its uniform expected decrease involving , and the resulting conditional expected gap contraction. They are indexed to the proof of Theorem 3 on page 5. Karimi, Nutini, and Schmidt (2016), Theorem 3 and proof
Greedy coordinate choice
The companion target is the rate stated in §3.2 when the maximum-norm PL inequality holds:
This bound applies to any run that chooses a maximizing coordinate at every step. The different PL modulus is part of the statement; it need not equal the modulus in Theorem 3. Karimi, Nutini, and Schmidt (2016), §3.2
Significance
Theorem 3 gives a quantitative guarantee for a method that reads one gradient component per update. The guarantee depends on the attained optimum, the coordinate smoothness constant, the PL modulus, and the number of coordinates. It applies to objectives satisfying PL even when convexity is unavailable. The companion result records how the stated rate changes when the coordinate is selected by the largest gradient component and the PL condition is measured in the corresponding dual norm. Karimi, Nutini, and Schmidt (2016), §3
The paper proves these claims, but this mission asks for machine-checked proofs of their exact hypotheses and constants. A complete development will make the finite random experiment explicit and give reusable definitions for coordinate updates and expectations of finite coordinate-choice histories. The PL predicate is already published in the theorem library; the coordinate model and the results here remain to be formalized.
Difficulty
A bound for a single coordinate update depends on one component of the gradient. An arbitrary coordinate can have a zero component even when the full gradient is large, so a pointwise decrease with the full gradient norm cannot be asserted for every choice. The theorem instead concerns the expectation over a uniform choice and then over the complete history of choices. The main formal difficulty is keeping those two levels of expectation aligned with a state that itself depends on earlier draws. For the greedy companion, the selected coordinate may not be unique; the statement must cover every maximizing choice without assuming a fixed tie-breaking rule.
Formalization scope
The vector space is EuclideanSpace ℝ (Fin d). The shared definitions give (5), the step (6), the iterate produced by a word Fin k → Fin d, and the expected gap as a real finite sum. The published ShiOptRates.PolyakLojasiewicz predicate represents (3) after multiplying both sides by two. Every theorem using uniform draws assumes ; every step with assumes . The goal also states differentiability, an explicit global minimizer, and . No convexity is assumed. A free real number called would change the PL condition and could invalidate the rate, so the formal statement uses throughout.
The index-word average is an exact finite representation of independent uniform sampling, including . It does not sample coordinates without replacement. The companion theorem represents the maximum-norm PL condition by requiring its inequality at every maximizing coordinate, which is equivalent for , and quantifies over every run satisfying the Gauss–Southwell update relation. Useful contributions include proofs of the three displayed milestones, the passage from one draw to the word average, and the greedy companion. The paper's optional remark on sampling proportional to coordinate-specific constants and its exact-coordinate-minimization remark are outside this proposal.
Selected references
- H. Karimi, J. Nutini, and M. Schmidt, Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak–Łojasiewicz Condition, ECML PKDD, 2016; arXiv:1608.04636v4.