Distributionally Robust Inverse Covariance Estimation: The Wasserstein Shrinkage Estimator: Without Sparsity Information the Robust Estimator Is a Closed-Form Shrinkage of the Sample EigenvaluesResearch Paper
Motivation
The precision matrix of a Gaussian random vector encodes its conditional independence structure: exactly when and are independent given the other coordinates. Estimating it is a basic step in Gaussian graphical models, portfolio selection and linear discriminant analysis. The textbook estimator, the inverse of the sample covariance matrix , fails when the number of samples is smaller than the dimension : then is singular and the maximum likelihood problem has no solution. Even for the inverse sample covariance is badly conditioned.
Nguyen, Kuhn and Mohajerin Esfahani (arXiv:1805.07194, Operations Research 2022) replace the nominal distribution in the likelihood by the worst case over all normal distributions within a type-2 Wasserstein distance of it, and show that the resulting robust problem has a closed-form solution. The solution keeps the eigenvectors of and maps its eigenvalues through an explicit nonlinear shrinkage, and it is invertible for every sample size. Shrinkage estimators of this kind had been proposed earlier for statistical reasons (Ledoit and Wolf, doi:10.1016/S0047-259X(03)00096-4); here the shrinkage follows from the robust model rather than being imposed.
Setting
Fix a dimension . Write and for the positive semidefinite and positive definite symmetric matrices, and for the trace inner product. For distributions on the type-2 Wasserstein distance is
the infimum over couplings with marginals , (Definition 1.1).
Given a sample covariance matrix (possibly singular) and a radius , the ambiguity set is
the centred normal distributions, degenerate ones included, within distance of the nominal one. The worst-case expectation is (7), and the robust estimation problem (5) is
where is the set of admissible precision matrices; without structural information .
The paper reformulates (5) through the metric on (Definition 2.3), the scalar program (8)
and the joint convex program (6), which minimizes plus the objective of (8) over , .
Formalization targets
Goal: Theorem 3.1 (p. 12)
Let , , and with orthonormal . Then the equation
has a unique positive solution , and the unique minimizer of (5) is
Milestones
- The worst case over equals (proof of Proposition 2.8, p. 8).
- Lemma 2.7 (p. 7): partial infima of functions continuous in are upper semicontinuous; under uniform calmness from below they are lower semicontinuous.
- Proposition 2.8 (p. 7): for , , equals the value of (8), and is continuous on .
- Theorem 2.6 (p. 7): for every , equals the value of (6), and is continuous on .
- Equation (16b) has a unique root in , and it is positive (p. 12).
- Identity (17), (p. 12), and (p. 13).
- For , is feasible and optimal in (6) (p. 12).
Significance
The result. Theorem 3.1 gives the robust estimator in closed form up to an eigendecomposition and a scalar root, so it costs no more than the sample covariance itself. It shows that the estimator commutes with , is positive definite for every even when is singular, and depends on the data only through the eigenvalues; the paper's later sensitivity results (Proposition 3.5) and the numerical comparisons with Ledoit–Wolf shrinkage and the graphical lasso rest on this formula. Theorem 2.6 shows more generally, for sparsity-constrained , that the robust problem is a convex program although the ambiguity set is not convex.
Formalizing it. The results are proved on paper; none is machine-checked. The formalization exercises a measure-level definition of the robust problem, with the ambiguity set defined through the coupling Wasserstein distance, and connects it to matrix analysis: the Gelbrich formula for Gaussians, a semidefinite duality, continuity of optimal value functions, and a first-order optimality argument for a matrix-fractional objective. Each milestone is reusable for other Gaussian–Wasserstein robust models, of which the platform already has several (Kalman filtering, minimax mean-square-error estimation).
Difficulty
The obvious argument is to plug into the first-order conditions of (6). This works only when : for singular , every zero eigenvalue gives , so is singular and is not even feasible in (6). The singular case, which is the case of interest (), therefore needs continuity of the optimal value and of the worst-case expectation in , which is the content of Theorem 2.6 and Proposition 2.8. Those in turn require the reduction from distributions to covariance matrices (the Gelbrich formula, which holds for degenerate normals and is an optimal-transport theorem in its own right), a Lagrangian duality with a Lyapunov-equation inner problem, and a priori eigenvalue bounds on near-optimal that vary continuously with . Sufficiency of the first-order conditions also needs joint convexity of , which the paper cites.
Formalization scope
is EuclideanSpace ℝ (Fin p) and matrices are Matrix (Fin p) (Fin p) ℝ with the entrywise topology. is Mathlib's multivariateGaussian 0 S, which allows singular . The Wasserstein distance is the published coupling distance WassersteinDRO.Gelbrich.wassersteinDistance 2, valued in ; the radius enters through ENNReal.ofReal ρ. The square root is the published psdSqrt, and the published WassMMSE.Sandwich.frob. The worst-case expectation, , and the values of (6) and (8) are suprema and infima in EReal. Every distribution in is Gaussian, so the integrand is integrable. Every matrix inverse sits under the constraint inside the infimum. "Unique minimizer" is stated as IsMinOn over all positive definite together with equality of every positive definite minimizer to . The eigendecomposition is an arbitrary orthonormal one, not a basis chosen by the library.
One hypothesis is added: , because for (16b) has no positive solution. Definition 2.1 prints the normal density without the factor in the exponent; the standard normal distribution is used.
The goal is stated for the measure-level problem (5). Stating it on the matrix program (6) would be a different and, for singular , false theorem. Replacing by the covariance ball or by all distributions, or computing suprema as real sSup (which returns on unbounded sets), would trivialize or change the statement.
The mission concerns the zero-mean problem (5) with for the goal; Theorem 2.6 is posed for general . Out of scope: Theorem 2.10 (the unknown-mean problem (4)), Corollary 2.9, Lemma 3.3, Proposition 3.5, the algorithm of §4, the extremal distributions of §5 (Theorem 5.1) and the experiments of §6. The Gelbrich formula (Proposition 2.2) is not posed here; it is Open on the platform as WassKF.Reform.proposition_2_2. The Open theorem WassMMSE.Sandwich.proposition_A_4_i, a covariance-level duality behind Proposition 2.8, is included for reuse. Welcome contributions: the Gelbrich formula for possibly degenerate centred normals, second moments of multivariateGaussian, continuity of optimal value functions (Berge's maximum theorem), and convexity of the matrix-fractional function.
Selected references
- V. A. Nguyen, D. Kuhn, P. Mohajerin Esfahani, Distributionally Robust Inverse Covariance Estimation: The Wasserstein Shrinkage Estimator, Operations Research 70(1), 2022; read as arXiv:1805.07194v1, 2018. https://arxiv.org/abs/1805.07194
- C. R. Givens, R. M. Shortt, A class of Wasserstein metrics for probability distributions, Michigan Mathematical Journal 31(2), 1984. https://doi.org/10.1307/mmj/1029003026
- O. Ledoit, M. Wolf, A well-conditioned estimator for large-dimensional covariance matrices, Journal of Multivariate Analysis 88(2), 2004. https://doi.org/10.1016/S0047-259X(03)00096-4
- S. Boyd, L. Vandenberghe, Convex Optimization, Cambridge University Press, 2004. https://doi.org/10.1017/CBO9780511804441