Subset Selection with Shrinkage: Sparse Linear Modeling When the SNR Is Low III: ℓ2-Regularized Best Subset Selection Satisfies Slow-Rate and Fast-Rate Oracle InequalitiesResearch Paper
Motivation
Best subset selection fits a linear model using at most of available predictors, choosing the that minimize the residual sum of squares. Mixed-integer optimization made it computable at moderate scale (Bertsimas, King & Mazumder, 2016), which reopened the statistical question of when it should be used. Empirical comparisons (Hastie, Tibshirani & Tibshirani, 2020) found that best subsets predicts poorly when the signal-to-noise ratio (SNR) is low: by fitting the noise exactly within its chosen support it overfits, and shrinkage methods such as the Lasso do better there.
Mazumder, Radchenko & Dedieu propose to add a shrinkage penalty to the cardinality-constrained problem. Their Theorem 2 quantifies the effect of an (Euclidean-norm) penalty: it proves prediction-error bounds that are linear in the penalty level and therefore remain small in the low-SNR regime where best subsets fails. This mission formalizes that theorem.
Setting
The data follow the model
where , is an unknown deterministic mean and is a noise vector (in the paper, with independent entries, ). The design is deterministic and its columns have unit Euclidean norm. Write for the Euclidean norm, for the number of nonzero coordinates of , and .
For a model size and a penalty level , the -regularized best-subsets estimator is any solution
The penalty is the norm , not its square, and the loss carries no factor . The quantity of interest is the prediction error .
The minimal -sparse eigenvalue of is . For , is the non-increasing rearrangement of . Given , three noise events are sets of noise vectors:
The paper's Theorem 1 shows these events have high probability; Theorem 2 is a deterministic statement on each of them. The symbol means "up to a positive universal multiplicative constant", one that does not depend on or .
Formalization targets
Goal: Theorem 2 (p. 14)
Let .
(A) Slow rate. If , then on
and on
(B) Fast rate. For every , on
The bounds hold for every minimizer and for misspecified models: need not lie in the column span of .
Milestones
The steps of the printed proof (Supplement B.2, pp. 39–40), in the order the proof uses them: the basic inequality (29); the bound (30) on and its consequence (31); the dichotomy on , where (30) implies (31) and (32) implies (33); the elementary inequality (34); the basic inequality on (p. 40); and the bound of through (p. 40).
Significance
The result. The slow-rate bound of part (A) is linear in . When the signal is small compared with , the term is smaller than the rate that best subsets attains, and the paper's Theorem 4 shows that in this regime the prediction error of best subsets is at least of order . Together the two results explain the empirical finding that shrinkage helps best subsets at low SNR. Part (B) shows nothing is lost at high SNR: the fast rate is the minimax rate over -sparse vectors plus the shrinkage term . Combined with Theorem 1, the bounds give the high-probability statements of the paper's Corollaries 1 and 3.
Formalizing it. The theorem is proved in the paper; no machine-checked version is known to exist. The mission produces a checked deterministic core: once the events are granted, every constant (, the factor , the threshold ) is verified exactly. Together with the companion mission on Theorem 1 it would yield fully checked high-probability oracle inequalities for a cardinality-constrained estimator, which no current formal library contains.
Difficulty
Every step is elementary, but the bookkeeping is exact. The obvious reading "insert the event's inequality into (29)" does not by itself work on : the event's right-hand side is a maximum of a sorted weighted sum and a prediction-norm term, and the first branch has to be compared with for the -sparse difference , including when exceeds . That comparison is a combinatorial estimate on sorted sums, not a substitution. On , the penalty difference is not controlled by the event at all and has to be converted into prediction errors through . In both parts the final constant must not depend on , or .
Formalization scope
All objects are explicit Lean definitions in the namespace SubsetShrink.L2:
- vectors are functions on
Fin nandFin pwith 0-based indices, and is a realMatrix (Fin n) (Fin p); - the Euclidean norm, and the inner product are explicit finite sums, since Mathlib's norm on
Fin m → ℝis the sup norm; - is encoded as a maximum over permutations, which equals the sorted sum by the rearrangement inequality;
- is a real infimum over the nonzero vectors of , a nonempty set when .
The model consists of with unit columns, , , and . The estimator is a predicate that every minimizer satisfies; no minimizer is chosen and existence is not assumed. "On the event" is a hypothesis that the noise vector satisfies the event's printed inequality, so no probability measure appears. is the event as printed on p. 13; the correction of its probability bound belongs to the mission on Theorem 1 and does not affect these deterministic statements.
Committed conventions:
- each is an existential constant quantified before all other data;
- an infimum over is stated as a bound for every ;
- is universally quantified;
- the model size satisfies .
Part (B) assumes or . When the paper's bound is vacuous (), but Lean's would turn it into a false claim.
A formalization that lets the constant depend on , or , fixes a particular minimizer, weakens an event (for example drops "for all "), or computes the infimum as a real ⨅ over an unbounded set would trivialize the theorem and is ruled out.
Not formalized here: the probability bounds of Theorem 1, the -penalized Theorem 3, the lower bound of Theorem 4, the data-driven choice of (Theorem 5), and the corollaries and numerical experiments. These are separate missions or follow-ups. Contributions of reusable lemmas (the rearrangement bound for the sorted sum, properties of , the elementary absorption inequality (34)) are welcome.
Selected references
- R. Mazumder, P. Radchenko, A. Dedieu, Subset Selection with Shrinkage: Sparse Linear Modeling When the SNR Is Low, arXiv:1708.03288v4, 2022; Operations Research, 2023. https://arxiv.org/abs/1708.03288
- D. Bertsimas, A. King, R. Mazumder, Best subset selection via a modern optimization lens, Annals of Statistics 44(2), 2016. https://doi.org/10.1214/15-AOS1388
- T. Hastie, R. Tibshirani, R. Tibshirani, Best subset, forward stepwise or Lasso? Analysis and recommendations based on extensive comparisons, Statistical Science 35(4), 2020. https://doi.org/10.1214/19-STS733
- P. C. Bellec, G. Lecué, A. B. Tsybakov, Slope meets Lasso: improved oracle bounds and optimality, Annals of Statistics 46(6B), 2018. https://doi.org/10.1214/17-AOS1670