A Singular Value Thresholding Algorithm for Matrix Completion 1: The SVT Iteration Converges to the Unique Solution of the Proximal ProblemResearch Paper
Motivation
Matrix completion asks to recover an matrix from a subset of its entries. When has low rank, a standard convex surrogate is to minimize the nuclear norm (the sum of the singular values) subject to agreeing with on ; Candès and Recht showed that this recovers exactly under incoherence and sampling conditions (Candès–Recht 2009). Generic interior-point solvers for this semidefinite program do not scale beyond matrices of a few hundred rows.
Cai, Candès and Shen (SIAM J. Optim. 2010) proposed the singular value thresholding (SVT) algorithm: a first-order iteration whose only nonlinear step is a soft-thresholding of singular values, and whose other iterate is a sparse matrix supported on . The algorithm has become a standard baseline in low-rank matrix recovery and a model example of dual (Uzawa-type) methods for nuclear-norm problems. This mission formalizes its convergence theorem.
Setting
All matrices are real. For write and . The nuclear norm is the sum of the singular values of .
For an index set , the sampling projector keeps the entries with indices in and sets the others to zero.
A reduced singular value decomposition of a matrix of rank is with () and () having orthonormal columns and with . For the singular value shrinkage operator is
Fix , a sequence of step sizes and data . The SVT iteration (2.7) starts from and sets, for ,
The proximal problem (2.8) is
More generally, for a linear map with adjoint and spectral norm , and , problem (3.1) is to minimize subject to , and Uzawa's iteration (3.3) starts from and sets , .
Formalization targets
Goal: Theorem 4.2, second sentence (p. 1968)
If , then (2.8) has a unique solution and the SVT iterates satisfy
Theorem 4.2, first sentence (p. 1968)
If (3.1) is feasible and , then (3.1) has a unique solution and the iterates of (3.3) converge to it.
Supporting results (milestones, in attack order)
- Well-definedness of (§2.1, p. 1960): the output does not depend on the chosen SVD.
- Theorem 2.1 (p. 1960): .
- Sparsity of the iterates (§2.2, p. 1961): since , every vanishes outside .
- Eq. (2.14) (p. 1964): the minimizers of the Lagrangian are those of .
- Lemma 4.1 (p. 1968): for , , .
- The §3.1 reduction (p. 1964): for a sampling operator, and (3.3) becomes (2.7) under .
- Theorem 4.2, first sentence, as above.
Significance
The theorem certifies that SVT, run with any step sizes in a fixed interval , computes the unique minimizer of the strongly convex surrogate (2.8). Together with the separate fact that the solution of (2.8) tends to the minimum nuclear norm completion as (the paper's Theorem 3.1, a companion mission), this is what justifies using SVT as a solver for nuclear-norm matrix completion. Theorem 2.1, the proximal characterization of singular value soft-thresholding, is used throughout the literature on proximal methods for low-rank problems.
The paper's proof of Theorem 4.2 consists of the reduction to Uzawa's method and a citation of a general convergence theorem for projected gradient methods on the dual. The formalization produces a self-contained, machine-checked chain: the proximal characterization of , the Lagrangian identity, strong monotonicity of , and the convergence argument itself. To our knowledge none of these results has a machine-checked proof; Mathlib at the pinned revision has singular values of linear maps but no SVD structure, no nuclear norm and no subgradient calculus.
Difficulty
Nothing in the iteration is a gradient step of a smooth function in : the -update is a nonsmooth proximal map, and the convergence of is not visible from the recursion itself. The paper's argument cites a general theorem on projected gradient methods ([25, Theorem 2.1]) and takes for granted that "strong duality holds" for (2.8) (p. 1963), so the existence of a Lagrange multiplier is part of what must be formalized. Theorem 2.1 depends on the subdifferential of the nuclear norm, which Mathlib does not provide, and therefore on the singular value decomposition and the duality between the nuclear and spectral norms. Convergence of objective values or of a subsequence would not suffice: the target is convergence of the whole sequence to the unique solution.
Formalization scope
Matrices are Matrix (Fin n₁) (Fin n₂) ℝ; convergence is Mathlib's topology on matrices, which coincides with the Frobenius-norm topology. and are defined entrywise; is the sum of Mathlib's LinearMap.singularValues of viewed as a map . The shrinkage operator is a relation IsShrink τ Y X defined, as in (2.1)–(2.2), through some reduced SVD of ; well-definedness is a milestone. It is not defined as the minimizer of (2.3), which would make Theorem 2.1 definitional. Linear maps are given by matrices with , and a sampling operator by an injective enumeration of . Subgradients are those of (2.4).
Sequences are indexed by : Lean's step is the paper's step , so and are unused. Committed conventions:
- and are hypotheses; with a start that is nonzero outside the iterates converge to a different matrix.
- The standing is kept, except in Theorem 2.1 and the well-definedness statement, which are printed for .
- The step-size conditions are explicit bounds , with for , together with , respectively . The multiplicative form avoids Lean's : for the condition does not become unsatisfiable.
- Feasibility of (3.1) is an added hypothesis of Theorem 4.2's first sentence, since "the unique solution" presupposes it.
- "Converges to the unique solution" is stated as existence and uniqueness of the solution together with convergence of the whole sequence to it.
A small unprinted helper, for sampling operators, is included to pass from the first sentence of Theorem 4.2 to the second; it is not a milestone. The subdifferential formula (2.6) of the nuclear norm and the Fejér-type condition of §5.1.2 are not stated. Reusable infrastructure welcome from solvers: existence and uniqueness properties of the reduced SVD, the nuclear/spectral norm duality, the subdifferential of the nuclear norm, and a general convergence theorem for Uzawa's method with a strongly convex objective.
Selected references
- J.-F. Cai, E. J. Candès, Z. Shen, A Singular Value Thresholding Algorithm for Matrix Completion, SIAM J. Optim. 20(4):1956–1982, 2010. https://doi.org/10.1137/080738970
- E. J. Candès, B. Recht, Exact Matrix Completion via Convex Optimization, Found. Comput. Math. 9:717–772, 2009. https://doi.org/10.1007/s10208-009-9045-5
- K. J. Arrow, L. Hurwicz, H. Uzawa, Studies in Linear and Non-Linear Programming, Stanford University Press, 1958.