Rademacher and Gaussian Complexities: Risk Bounds and Structural Results 6: The Expected Maximum Discrepancy Lies Between R_n(F)/2 − 2√(2/n) and R_n(F) + 4√(2/n)Research Paper
Motivation
Data-dependent risk bounds in statistical learning theory control the gap between the expected loss of a learned function and its empirical loss by a complexity penalty that is computed from the training data. The first such penalties were the maximum discrepancy of a function class (Bartlett, Boucheron and Lugosi, Model selection and error estimation, Machine Learning 48, 2002) and its Rademacher complexity (Koltchinskii, Rademacher penalties and structural risk minimization, IEEE Trans. Inf. Theory 47, 2001; Koltchinskii and Panchenko 2000). The maximum discrepancy compares the behaviour of the class on two fixed halves of the sample; the Rademacher complexity compares it on two random halves. Bartlett and Mendelson (JMLR 3, 2002), Lemma 3, show that these two quantities are equivalent up to a factor 2 and an additive . This mission formalizes that lemma from the published JMLR article (pp. 463–482); the proof is its Appendix A.
Setting
Let be a probability measure on a measurable space and let be independent samples from . Let be a class of measurable functions . Let be independent uniform -valued random variables, independent of the sample.
The Rademacher complexity of is
For even , the maximum discrepancy of is the random variable
with no absolute value, and the expected maximum discrepancy is . The class is closed under negation if implies , and .
The proof works with the conditional supremum function
defined for the values that can take.
Formalization targets
Goal: Lemma 3, first and second displays
For every even ,
and if is closed under negation,
Milestones (Appendix A, pp. 479–480)
- , with equality when is closed under negation.
- .
- .
- for .
- .
- for any (a corrected form of the printed step, see below).
Significance
Lemma 3 makes the maximum discrepancy and the Rademacher complexity interchangeable in risk bounds: a bound in terms of one gives a bound in terms of the other with an explicit additive loss. The maximum discrepancy can be computed by a single empirical risk minimization on a relabelled sample, while the Rademacher complexity has the structural properties (monotonicity, convex-hull invariance, contraction) that make it easy to bound for concrete classes; the lemma transfers the second kind of estimate to the first quantity.
The lemma is proved in the paper; no machine-checked proof of it, or of the comparison between fixed and random half-sample splits, is known to exist. Formalizing it requires the exchangeability argument for i.i.d. samples, the conditioning of a uniform sign vector on its sum, and a moment bound for the Rademacher sum , all with explicit constants.
Difficulty
The heart of the proof is that, conditioned on the number of positive signs, a uniform sign vector splits the i.i.d. sample into two random subsets of fixed sizes, and every split of the same sizes has the same law as the fixed split. Making this precise requires a permutation-invariance argument for product measures applied to a supremum over an arbitrary class, where measurability is not automatic. The step from classes closed under negation to general classes is where the printed argument is loose: since has no absolute value, is a maximum of two suprema that may be negative, and the naive bound fails.
Formalization scope
The Lean development lives in the namespace RadGauss.Discrepancy. Sign vectors are Fin n → Bool (true ↦ 1, false ↦ -1) and expectations over signs are finite averages over all sign vectors; is the average over the sign vectors with sum . takes values in (a lower Lebesgue integral of an -valued supremum), while , and are real, because the maximum discrepancy is signed. Inequalities of the form are written with real terms embedded by ENNReal.ofReal; this is equivalent to the printed form since for nonempty .
Hypotheses added to the page, all disclosed in each item:
- the sample size is even, with , since needs half sums;
- is nonempty (the supremum over the empty class is in the paper and in Lean);
- every is measurable, and for every sign vector the map is measurable. This is the measurability guard: without it the Bochner integrals defining and would silently be .
Corrections of printed statements:
- The Lipschitz bound on is printed for but used for negative values of ; it is stated for every pair of attainable values.
- The printed step is false for the signed of p. 464 (, uniform on , gives ); the milestone states it with the additional term . The goal itself remains true.
- The third display of Lemma 3, , is false as printed ( as above, , : the probability is ) and is not part of the mission.
A formalization in which or is a junk value (non-integrable or non-measurable suprema, an empty class, an odd sample size with truncated ) would make the goal trivial or meaningless; the hypotheses above rule that out, and the class satisfies all of them.
Welcome contributions include general lemmas on the invariance of under permutations of an i.i.d. sample, conditioning of uniform sign vectors on their sum, and the bound . These are reusable well beyond this mission.
Selected references
- P. L. Bartlett, S. Mendelson, Rademacher and Gaussian Complexities: Risk Bounds and Structural Results, Journal of Machine Learning Research 3 (2002) 463–482. https://www.jmlr.org/papers/v3/bartlett02a.html
- P. L. Bartlett, S. Boucheron, G. Lugosi, Model selection and error estimation, Machine Learning 48 (2002) 85–113. https://doi.org/10.1023/A:1013999503812
- V. Koltchinskii, Rademacher penalties and structural risk minimization, IEEE Transactions on Information Theory 47 (2001) 1902–1914. https://doi.org/10.1109/18.930926
- L. Devroye, L. Györfi, G. Lugosi, A Probabilistic Theory of Pattern Recognition, Springer, 1996. https://doi.org/10.1007/978-1-4612-0711-5