Risk Guarantees for End-to-End Prediction and Optimization Processes: The Excess Squared-Loss Risk Bounds the Squared Excess Optimality-Gap Risk, Scaled by the Squared Diameter of the Feasible SetResearch Paper
Motivation
Many operational decisions are optimization problems whose cost vector is unknown when the decision is made but can be predicted from observed features: shortest paths under predicted travel times, portfolios under predicted returns, production plans under predicted prices. The common practice, predict-then-optimize, first fits a regression model mapping features to a cost estimate and then solves the optimization problem with in place of the true cost . What the decision maker cares about is not the prediction error but the optimality gap of the resulting decision under the true cost. Elmachtoub and Grigas (arXiv:1710.08005, Management Science 2022) introduced the SPO+ surrogate to train predictors against that gap directly, and established its Fisher consistency.
Ho-Nguyen and Kılınç-Karzan (arXiv:2012.15046, Management Science 2022) ask which surrogate losses give quantitative guarantees: when does a small excess surrogate risk force a small excess optimality-gap risk, uniformly over a class of data distributions? Their answer adapts the calibration theory of Steinwart (Constructive Approximation 26, 2007) to the end-to-end setting, and its most striking consequence concerns the ordinary squared loss.
Setting
Let be a non-empty convex compact set and a convex function, continuous on . For a cost vector the decision maker solves
An argmin selection is a fixed map with a minimizer of over for every . Using a prediction instead of the true cost incurs the true loss
Features and costs form a random pair with joint law . For a loss and a measurable predictor , the risk is and the Bayes risk is over measurable ; for these are written and . The squared loss is .
Given a set , the class consists of the laws for which every conditional law is concentrated on , is square integrable, and has mean . Assumption 2 asks that be bounded. The constants are , and .
A loss is -uniformly calibrated if for every there is such that, for every law in , every and every , a conditional excess -risk below forces a conditional excess true risk below . The largest such is the calibration function .
Formalization targets
Goal: Corollary 3 (p. 22)
For every law with and every measurable predictor ,
Milestones
- Lemma 2 (p. 10): attains , and .
- Remark 5 (p. 19): the conditional excess true risk equals with , and is at most .
- Lemma 3 (p. 20): uniform calibration holds iff for all .
- Lemma 4 (p. 21): the alternative form (14) of .
- Lemma 5 (p. 21): when .
- Lemma 6 (p. 22): is the purely geometric infimum of .
- Theorem 4 (p. 22): is -uniformly calibrated, with .
- Theorem 3 (p. 20): for any uniformly calibrated , , with the convex minorant of on .
The goal is stated with the constant and no other problem data: it is independent of , of beyond boundedness, and of the law of the data.
Significance
The corollary says that a predictor trained by ordinary least squares, without any information about or , has excess optimality-gap risk at most . Since the excess squared-loss risk separates over coordinates, coordinate-wise regression bounds suffice for end-to-end guarantees (Remark 7). Any learning-theoretic rate for least-squares regression therefore transfers to a rate for the decision problem, up to a square root. Theorem 3 gives the same transfer for any uniformly calibrated surrogate, and is the template for the paper's further results on separable symmetric losses.
The results are proved in the paper; none of them has a machine-checked proof. Formalizing them requires a measure-theoretic treatment of conditional risks through kernels, a Lean version of Steinwart's convexified calibration inequality in this setting, and the geometric bound of Lemma 5. Each milestone is a self-contained target.
Difficulty
The true loss is neither convex nor continuous in (Lemma 1): the decision jumps between faces of . A pointwise comparison between and fails, since a prediction close to can still select a far-away vertex when lies near a tie. The comparison only holds after conditioning on and averaging over , and it must hold uniformly over all conditional laws in , not for a fixed distribution. The step from conditional comparisons to the integrated risks is where the uniform bound on the conditional excess true risk and the convention at in matter; Bayes risks are infima over predictors, so this step also needs the conditional minimizers to assemble into a measurable predictor.
Formalization scope
Vectors live in EuclideanSpace ℝ (Fin m). The standing setting (Standing X f) bundles convexity, compactness and non-emptiness of , convexity of on and continuity of on ; IsArgminSelection X f xstar is the selection property. A joint law is μ ⊗ₘ κ, with μ the law of and κ a Markov kernel giving ; is the Bochner integral against κ w. Risks are -valued lower integrals of ENNReal.ofReal of the non-negative losses, Bayes risks are infima over measurable predictors, and excess risks are truncated differences. The class is lsClass C, a set of conditional laws concentrated on \mathbb P \in \mathcal Pw \in WWB_X$ is Metric.diam X.
Added or pinned conventions:
- measurability of
xstar(the paper uses Lebesgue measurability, Lemma EC.4) and of predictors; - in the goal, explicit though implied by support on bounded , to make the extended-real Bayes-risk subtraction finite;
- multiplied out in the goal; Theorem 4 carries for its printed quotient;
- the conclusion of (12) read conditionally on ;
- constrained by (the convention of p. 19);
- in Theorem 3, the identity and finiteness of .
The Bayes risks are genuine infima over measurable predictors and the conditional excess risks are defined generically. Neither is defined by the closed form that Lemma 2 or Lemma 6 establishes, so neither lemma holds by definition.
Needed infrastructure: lower integrals against μ ⊗ₘ κ and their disintegration, measurability of the conditional mean and of the optimal value , Jensen's inequality for real convex functions on a bounded interval, and the distance from a point to a half-space. The bias–variance decomposition for laws on and the half-space distance formula are reusable beyond this mission. Proofs of any milestone are welcome, as are alternative arguments for the goal that bypass Theorem 3.
Selected references
- N. Ho-Nguyen and F. Kılınç-Karzan, Risk guarantees for end-to-end prediction and optimization processes, Management Science 68(12), 2022. arXiv:2012.15046v1. https://arxiv.org/abs/2012.15046
- A. N. Elmachtoub and P. Grigas, Smart "predict, then optimize", Management Science 68(1), 2022. https://arxiv.org/abs/1710.08005
- I. Steinwart, How to compare different loss functions and their risks, Constructive Approximation 26, 2007. https://doi.org/10.1007/s00365-006-0662-3