Subset Selection with Shrinkage: Sparse Linear Modeling When the SNR Is Low VI: A BIC-Type Choice of the Model Size Yields Oracle Inequalities for the Expected Prediction ErrorResearch Paper
Motivation
Best subset selection fits a linear model using at most of available predictors: it minimizes the residual sum of squares over all with at most nonzero coordinates. It is the classical benchmark for sparse regression, and mixed-integer optimization has made it computable at scales that used to be out of reach (Bertsimas, King & Mazumder, 2016). When the signal-to-noise ratio is low, however, best subsets overfits, and Mazumder, Radchenko & Dedieu propose to add a shrinkage penalty () to it. Their oracle inequalities for the penalized estimators hold for a fixed model size , and the practitioner still has to choose .
§3.5 of the paper removes that choice. It replaces the cardinality constraint with BIC-type penalties that depend on the candidate model size, and Theorem 5 shows that the resulting estimators satisfy oracle inequalities for the expected prediction error over all of , with no condition on the design beyond the normalization of its columns. This mission formalizes Theorem 5 and the steps of its printed proof.
Setting
We observe . The mean is deterministic and unknown, and has independent entries with . The design is deterministic, with and columns of unit Euclidean norm. The prediction error of an estimate is . For , is the Euclidean norm, , and is the number of nonzero coordinates.
For nonnegative tuning parameters and , the data-dependent penalties are
with the convention , so the empty model is not penalized. The BIC-type estimators are
where is a fixed number. Both are unconstrained problems. Solving them is equivalent to solving the constrained problems with for each and then choosing by the penalty. A minimizer exists but need not be unique.
The proof works on events of the noise vector. For an integer and the event requires
Formalization targets
Goal: Theorem 5 (p. 21)
There exist universal constants such that, if or ,
and, if or ,
Here hides one universal multiplicative constant. The constants are not specified, so the goal records the shape of the bound and is not invalidated by later improvements of the constants.
Milestones (proof of Theorem 5, B.13, and B.5)
- (51): the deterministic basic inequality for .
- (54): for , with probability at least , .
- The union bound over model sizes: with , where is taken at , we have .
- (55): for , the analogue of (54) with probability at least .
- (56): under either condition, the high-probability oracle inequality with and .
- The tail-to-expectation step of B.5: if with probability at least , then .
- (58): the high-probability counterpart of (56) for .
Significance
The result. Theorems 2 and 3 of the paper give oracle inequalities only over the sparsity class for a fixed in advance. Theorem 5 makes the model size data-driven. Its bounds hold over all of and need no restricted-eigenvalue or sparse-eigenvalue condition on . Corollary 6 of the paper deduces from it that the estimators attain the error rates of Corollaries 3 and 4 while selecting models of the same order as the true model size. The paper also notes that the slow-rate bound for with is a bound for the Lasso.
Formalizing it. Theorem 5 is a published result with a printed proof (Supplement B.13). The proof is written up to unspecified universal constants, and it defers two steps to other parts of the paper ("repeating the argument after inequality (32)" and "repeating the argument in the proof of Corollary 3"). A formal proof makes every constant and every deferred step explicit. No part of the paper has been machine-checked, to the best of our search.
Difficulty
The argument is long rather than deep. The obvious first attempt fixes the selected model size and applies a high-probability bound for that size. This fails because is random and depends on , so a bound for each fixed says nothing about the random . The noise term must be controlled simultaneously over all model sizes, at a total failure probability that does not grow with . The second obstacle is the passage from high-probability bounds, which hold for every and grow like , to a bound in expectation. That passage needs the high-probability statement with a constant that is uniform in .
Formalization scope
-
Representation. Vectors are
Fin n → ℝandFin p → ℝ, and the design is aMatrix (Fin n) (Fin p) ℝ.- The norms , and are explicit definitions (
l2,l1,l0), because Mathlib's norm onFin m → ℝis the sup norm. - The noise law is the product of centred Gaussians with variance (
noiseLaw). - Logarithms are natural, and .
- The norms , and are explicit definitions (
-
Model. The standing hypotheses are:
- unit-norm columns of ;
- and ;
- , and for .
The quantities , , , , , and the penalties are inputs. The randomness is the law of alone, and .
-
Estimators are argmin predicates
IsBIC2,IsBIC1: " minimizes the objective over all of ". Every statement holds for every minimizer.- In Theorem 5 an estimator is a measurable map that is a minimizer for every , and the bound is claimed for every such selection.
- The expectation is a lower Lebesgue integral in , so a non-integrable error cannot make the bound vacuous.
-
Constants.
- "" is one universal constant , quantified before every other parameter.
- "" is written as "for every ".
- "With probability at least " is the existence of a measurable event of probability at least on which the inequality holds.
- The constants , and carry no sign, as on the page.
-
Correction. Theorem 1 of the paper states for an event with coefficient on . That bound is false for small (already for ). The union bound here uses the corrected event with coefficient , which inside changes nothing downstream.
-
Ruled out. None of the following counts as a formalization of the goal:
- an arbitrary or non-measurable selection of the minimizer;
- a Bochner integral, which is for a non-integrable error;
- a constant that depends on , , or the design;
- a fixed or .
-
Not formalized here. The fixed- oracle inequalities (Theorems 2 and 3), the noise events of Theorem 1, the lower bound (Theorem 4), Corollaries 1–6 and the numerical study are not part of this mission. Some of them are the subjects of the other missions of this series. The proof of Theorem 5 needs the probability bounds for the events and of Theorem 1 and the deterministic arguments of Theorems 2 and 3. A solver may prove them here or import them once they are published.
-
Contributions welcome.
- Gaussian chi-square tail bounds (Laurent–Massart);
- the bound for the sorted- event (Bellec, Lecué & Tsybakov);
- the existence of measurable selections of the argmin;
- the layer-cake step.
These are reusable well beyond this paper.
Selected references
- R. Mazumder, P. Radchenko, A. Dedieu, Subset Selection with Shrinkage: Sparse Linear Modeling When the SNR Is Low, arXiv:1708.03288v4, 2022; Operations Research, 2023. https://arxiv.org/abs/1708.03288
- D. Bertsimas, A. King, R. Mazumder, Best subset selection via a modern optimization lens, Annals of Statistics 44(2), 2016. https://doi.org/10.1214/15-AOS1388
- P. C. Bellec, G. Lecué, A. B. Tsybakov, Slope meets Lasso: improved oracle bounds and optimality, Annals of Statistics 46(6B), 2018. https://doi.org/10.1214/17-AOS1670
- B. Laurent, P. Massart, Adaptive estimation of a quadratic functional by model selection, Annals of Statistics 28(5), 2000. https://doi.org/10.1214/aos/1015957395
- L. Birgé, P. Massart, Gaussian model selection, Journal of the European Mathematical Society 3, 2001. https://doi.org/10.1007/s100970100031