From Predictive to Prescriptive Analytics 2: For Costs in [0, c̄] That Are L-Lipschitz, w.p. ≥ 1 − δ Every Decision Rule in F Costs ≤ Its Empirical Cost + c̄√(log(1/δ)/2N) + L·ℜ_N(F)Research Paper
Motivation
Many operations decisions are made after observing side information: an inventory manager sees weather, search trends and calendar effects before ordering; a supply-chain planner sees market signals before shipping. Bertsimas and Kallus, in From Predictive to Prescriptive Analytics (arXiv:1402.5481v4, Management Science 2020), study how to turn such data into decisions. Besides their local, weight-based prescriptions (the subject of mission 1 of this series), their §8 analyses a second approach: choose a decision rule from a structured class, such as norm-bounded linear rules, by empirical risk minimization — minimizing the average historical cost of the rule.
The question this mission formalizes is the one any practitioner of that approach must answer: how far can the true expected cost of the chosen rule exceed its in-sample cost? Classical statistical learning answers it with Rademacher complexity (Bartlett and Mendelson, JMLR 2002), but only for real-valued predictors. Operations decisions are vectors (order quantities for several products, shipments to several locations), so the paper extends the theory to multivariate decision rules.
Setting
Covariates take values in a measurable space , the uncertain quantity in a measurable space , and decisions in , unconstrained. Data are an i.i.d. sample from a probability measure on , and is its covariate part. A cost is incurred when decision meets outcome . A decision rule is a measurable map ; is a class of decision rules. Its expected cost is , and its empirical cost is , the objective of problem (4).
The empirical multivariate Rademacher complexity of (Definition 3) on a sample is
with independent uniform signs, and the marginal complexity is its expectation over the sample. For real-valued classes () these are the usual Rademacher complexities with factor and no absolute value. Lean notation: empRademacher, margRademacher, expCost, empCost, costClass in PrescAnalytics.ERM.
Formalization targets
Goal: Theorem 13 (out-of-sample guarantees)
If and , then for any each of the following holds with probability at least :
In particular they hold for the empirical risk minimizer. The goal is stated for a general class , not only the linear rules (25).
Milestones
- Lemma 1 (comparison): for , and .
- Theorem 14 (29): for a class of functions with values in , with probability at least , for all .
- Theorem 14 (30): the data-dependent version with .
Significance
Theorem 13 says that the quantity empirical risk minimization optimizes is, up to confidence terms that do not depend on the rule, an upper bound on the true expected cost of every rule in the class, uniformly. Bound (27) is computable from data, so it certifies a learned policy without a holdout set. Combined with complexity bounds for norm-restricted linear rules (the paper's Lemmas 2 and 3, not part of this mission), it shows the confidence terms vanish as .
The results are known: (29) and (30) for -valued classes are the standard Rademacher bounds (Mohri, Rostamizadeh and Talwalkar, Foundations of Machine Learning, Thm 3.3, after rescaling). What is new in the paper is the -norm vector comparison inequality of Lemma 1 with constant . None of these statements has a machine-checked proof on the platform. A formal development would supply a reusable symmetrization-plus-McDiarmid pipeline and a vector contraction lemma. Related items posed in this repository are Bartlett–Mendelson's Theorem 8 (RadGauss.RiskBound.theorem_8, with an absolute value in the complexity and constant ), Maurer's Euclidean vector contraction with constant (SPOBounds.Margin.maurer_vector_contraction), and the open uniform law HighDimStat.UniformLaws.uniform_law_rademacher_complexity; none is the same statement as a target here.
Difficulty
The uniform bound needs three ingredients: a concentration inequality for the supremum of the deviation over the class (bounded differences), a symmetrization step relating the expected supremum to the Rademacher complexity, and a comparison inequality to pass from the cost class to the decision rules. The scalar contraction principle of Ledoux and Talagrand does not apply directly, since each cost depends on a -dimensional output; applying it coordinatewise loses a factor of , and Euclidean vector contraction loses and uses the wrong norm. Lemma 1 needs an argument specific to the -norm. Measure theory adds its own difficulty: the supremum over an uncountable class must be shown measurable before its expectation or probability means anything.
Formalization scope
Decisions are Fin d → ℝ, whose Lean norm is the -norm; , are general measurable spaces. Samples are drawn from the product measure Measure.pi (fun _ : Fin N => μ), with . The following readings are explicit:
- Each "with probability at least , … for all " is a bound on the outer measure of the set of samples where the inequality fails for some ; the quantifier over the class is inside the event.
- Empirical complexities are in
EReal(an unbounded class gives ), marginal ones are lower Lebesgue integrals in ; . - The paper assumes only (resp. ). With that alone (26) and (29) are false — a two-point outcome with a constant rule violates them with probability — so the costs are assumed to take values in , keeping every printed constant.
- The undefined are (the i.i.d. case of the paper's Theorem 21); the Lipschitz denominator printed as is .
- Decision rules and the cost are measurable and the class is pointwise separable (a countable subclass approximates every member pointwise). The paper takes probabilities and expectations of suprema over the class without comment; without such a convention the statement fails for pathological classes.
A trivializing formalization is ruled out: the quantifier over the class sits inside the probability, the complexity has no absolute value and is never a junk for unbounded classes, and the hypotheses are satisfied by a clipped newsvendor cost. Solvers will need McDiarmid's inequality, symmetrization with a ghost sample, and the comparison lemma; each is reusable for any Rademacher-based generalization bound. Formalizations of these tools as separate lemmas are welcome.
Selected references
- D. Bertsimas, N. Kallus, From Predictive to Prescriptive Analytics, arXiv:1402.5481v4, 2018; Management Science 66(3), 2020. https://arxiv.org/abs/1402.5481 , https://doi.org/10.1287/mnsc.2018.3253
- P. L. Bartlett, S. Mendelson, Rademacher and Gaussian Complexities: Risk Bounds and Structural Results, JMLR 3, 2002. https://www.jmlr.org/papers/v3/bartlett02a.html
- M. Ledoux, M. Talagrand, Probability in Banach Spaces, Springer, 1991. https://doi.org/10.1007/978-3-642-20212-4
- M. Mohri, A. Rostamizadeh, A. Talwalkar, Foundations of Machine Learning, 2nd ed., MIT Press, 2018. https://mitpress.mit.edu/9780262039406/
- A. Maurer, A Vector-Contraction Inequality for Rademacher Complexities, ALT 2016. https://arxiv.org/abs/1605.00251