Robust Regression and Lasso 2: When an Absolute Norm Bounds the Column Disturbance Norms, Robust Regression Is Regression Regularized by the Dual Norm, ‖b − Ax‖ₐ + l‖x‖*ₛResearch Paper
Motivation
Robust regression guards a linear model against errors in the data matrix. The observations are fitted by , but the features (the columns of ) are only known up to a disturbance from an uncertainty set . The robust estimator minimizes the worst-case residual,
Xu, Caramanis and Mannor (arXiv:0811.1790v1; IEEE Trans. Inf. Theory 56(7), 2010) showed that when each column is perturbed independently within a Euclidean ball of radius , this problem is exactly the weighted -regularized problem , an un-squared Lasso (their Theorem 1, the subject of the first mission of this series). Robustness then explains regularization: the penalty is the price of the worst disturbance.
Section III of the paper asks which regularizers arise when the disturbances of different features are coupled, that is, when a single budget limits all columns jointly. The answer it gives is that every norm-based regularizer arises this way: a norm budget on the vector of column-disturbance sizes produces the dual norm as the penalty. This links robust optimization (Ben-Tal, El Ghaoui, Nemirovski 2009) and regularized estimation, and it recovers the robust least-squares formulation of El Ghaoui and Lebret (1997) as the Euclidean special case.
Setting
Fix , , and an arbitrary norm on . The data are the columns of and the response ; for , . A disturbance is with ; the perturbed matrix has columns . For an uncertainty set the robust objective is
Three families of uncertainty sets appear.
- Feature-wise (Theorem 3): for budgets , .
- Coupled (p. 6): for functions , , with the budget set .
- Norm budget (Corollary 1): for a norm on and , . The dual norm is . A norm is absolute if for all .
A fourth, polytope set, for , , is the subject of Corollary 2. Write .
Formalization targets
Goal: Corollary 1 (p. 7)
For an absolute norm and , for every the worst case over the norm-budget set is attained and
and the robust problem and have the same minimizers.
Milestone: Theorem 3 (p. 6)
For and every : , attained, with the same minimizers.
Milestone: Eq. (13) (proof of Theorem 4, Appendix B, p. 19)
For every , in the extended reals,
Milestone: Corollary 2 (p. 7)
If some has , then for every the worst case over the polytope set equals (both sides when the constraints are infeasible), the infimum is attained when feasible, and robust minimizers are exactly the -parts of optimal solutions of that linear program.
The paper's general result, Theorem 4 (p. 6), expresses the coupled robust problem through a Lagrangian function under the assumption that has nonempty relative interior. The mission does not pose it: that assumption is not Slater's condition, and the statement fails without one (an explicit counterexample with is recorded in the series plan). The mission poses the decomposition (13), on which Theorem 4 rests, and the two corollaries, which are true as stated under the readings below.
Significance
The result. Corollary 1 says that any norm penalty is the worst-case cost of column disturbances whose sizes are jointly bounded in the dual norm. Taking gives the penalty (budget per column), gives (a shared budget), and both Euclidean gives the Frobenius-ball model of robust least squares. Corollary 2 shows that polytope budgets keep the problem a linear program apart from the loss, which allows modeling, for instance, corruption of an unknown subset of features. Eq. (13) is the common reduction behind these: the coupled worst case is the uncoupled one maximized over the admissible budget vectors.
Formalizing it. None of these statements has a machine-checked proof. The mission makes precise two points the paper leaves implicit: which "symmetric" norms the corollary is true for, and in what sense "max" holds when the budget set is empty or unbounded. The finished development is a self-contained library of worst-case residual identities for arbitrary norms on finite-dimensional or general normed spaces. Related platform items, credited and not reused: SPOBounds.StronglyConvex.dual_norm_ball_max (maximum of a linear functional over a norm ball, with the continuous-dual norm), and RobustLS.Unstructured.worst_case_residual_closed_form (El Ghaoui–Lebret, which perturbs in a Frobenius ball, a different set).
Difficulty
The upper bound of every identity is a triangle inequality. The content is in the matching lower bound: a disturbance must be exhibited that aligns all perturbed columns with the residual direction, and in (13) each column disturbance must have norm exactly , including columns with , which is where enters. The paper's first line of the proof of (13) rewrites as ; that set identity is false unless is downward closed, so the obvious route through it does not work for (13) as stated. For Corollary 1 the remaining step is the identity , which needs the absolute property and fails for norms that are only permutation invariant. For Corollary 2 the remaining step is linear programming duality, which Mathlib does not yet provide in this form.
Formalization scope
is an arbitrary real normed space with Nontrivial E (the paper's implicit ); finite dimension is not imposed, a harmless generalization. Columns are a family a : Fin m → E, disturbances δ : Fin m → E, coefficients Fin m → ℝ, and . Explicit readings:
- The robust objective is a supremum over the uncertainty set, computed in
EReal; it is never defined by its closed form. "Max" is stated asIsGreatestof the set of residual values (attained) in Theorem 3 and Corollary 1, where the maximum exists, and as anERealsupremum in (13) and Corollary 2, where it need not. - "Is equivalent to" / "the resulting regularized regression problem is" means: equal objectives at every and equal sets of minimizers. Existence of a minimizer is not claimed.
- "Symmetric norm" is read as absolute; is a Mathlib
SeminormonFin m → ℝwith (the type already carries the sup norm, so the notation is not used for it). The dual norm is a realsSupof , a nonempty set that is bounded above because is a norm on . - is added to Corollary 1, and feasibility of , to Corollary 2; without them the sets are empty.
- Eq. (13) is stated for as defined on p. 6 (not through the paper's set rewriting); the convexity of the is kept as the paper's standing assumption and is not used.
A trivializing formalization is ruled out: the robust objective is the EReal supremum (or the residual-value set) over the uncertainty set, so no identity holds by unfolding, and no real sSup junk value enters a robust objective. Welcome contributions: the explicit worst-case disturbance in an arbitrary normed space, the absolute-norm duality lemma, and a finite-dimensional LP strong duality theorem, each reusable beyond this mission.
Selected references
- H. Xu, C. Caramanis, S. Mannor, Robust Regression and Lasso, arXiv:0811.1790v1, 2008; IEEE Transactions on Information Theory 56(7):3561–3574, 2010. https://arxiv.org/abs/0811.1790v1, https://doi.org/10.1109/TIT.2010.2048503
- L. El Ghaoui, H. Lebret, Robust solutions to least-squares problems with uncertain data, SIAM Journal on Matrix Analysis and Applications 18(4):1035–1064, 1997. https://doi.org/10.1137/S0895479896298130
- A. Ben-Tal, L. El Ghaoui, A. Nemirovski, Robust Optimization, Princeton University Press, 2009. https://doi.org/10.1515/9781400831050
- R. Tibshirani, Regression shrinkage and selection via the lasso, Journal of the Royal Statistical Society B 58(1):267–288, 1996. https://doi.org/10.1111/j.2517-6161.1996.tb02080.x