The Sample Average Approximation Method for Stochastic Discrete Optimization 1: Under Assumption A, 1 − P(Ŝ^ε_N ⊂ S^ε) Decays Exponentially Fast in the Sample Size NResearch Paper
Motivation
Many decision problems in operations research ask for the best element of a finite set of alternatives when each alternative's cost is random: which projects to fund when resource use is uncertain, which crew pairings to fly when delays are random, which items to pack into a knapsack whose overflow is penalized. The expected cost is rarely available in closed form, because it is a high-dimensional integral over the distribution of the random data. The sample average approximation (SAA) method replaces the expectation by an average over a random sample and solves the resulting deterministic problem with a standard discrete optimization routine. The method is widely used in stochastic programming (see Shapiro, Dentcheva and Ruszczyński, Lectures on Stochastic Programming, SIAM 2009, Ch. 5, https://doi.org/10.1137/1.9780898718751).
Kleywegt and Shapiro (preprint; published with Homem-de-Mello as SIAM J. Optim. 12(2):479–502, 2002, https://doi.org/10.1137/S1052623499363220) ask why the method works so well for discrete problems. Their answer is a large-deviations statement: when the feasible set is finite, the probability that the SAA problem returns a wrong answer decays exponentially in the sample size. This mission formalizes that result, Proposition 2.2 of the two-author preprint, together with its consistency companion, Proposition 2.1.
Setting
Let be a nonempty finite set, a random element of a measurable space , and such that is measurable and for every . The true problem (1.1) is
Let be an independent, identically distributed sample with the law of . The sample average function and the SAA problem (2.1) are
For , the -optimal sets are and ; at they are the optimal sets and . Two numbers drive the analysis: the maximal deviation of (2.2), and the well-conditioning number
of (2.3), the gap between the -optimal solutions and the rest.
For a real random variable with moment generating function and logarithmic moment generating function , the rate function is . Assumption A requires that for every the moment generating function of be finite in a neighbourhood of .
Formalization targets
Goal: Proposition 2.2
Under Assumption A, for every there is with
The constant is not specified: it depends on through rate functions and cannot be computed before the problem is solved. The goal asserts only its existence, which is the paper's claim.
Milestones
- whenever (2.3).
- implies .
- The union bound .
- The large-deviations upper bound (2.4): for i.i.d. averages .
- for above the mean, when is finite near .
- Under Assumption A, with .
Companions
Proposition 2.1 ( w.p.1, and w.p.1 for large enough), the almost-sure convergence of (2.2), the singleton consequence ( implies eventually), the facts , the Example of §3.2 showing that a sample size can be required, and the computation on the hypercube with best value (§4).
Significance
Proposition 2.2 explains the practical behaviour of the SAA method on discrete problems: an exact optimal solution of the true problem is obtained with probability approaching one exponentially fast, so a moderate sample often suffices even when the number of feasible solutions is astronomically large. It is the basis of the paper's replication argument (§3.2: the probability of success of one SAA replication tends to one exponentially fast) and identifies as the quantity that governs the difficulty of a stochastic discrete problem, which the numerical study of §4 then examines. Later sample-size estimates for SAA (Shapiro, Dentcheva and Ruszczyński 2009, §5.3) refine the same argument into explicit, non-asymptotic bounds.
The result is proved in the paper. The mission produces a machine-checked proof of it and of its ingredients, including a Lean statement of the Cramér–Chernoff upper bound with the rate function in the extended reals and the positivity of above the mean under a local moment condition. To our knowledge none of these is formalized on the platform or in Mathlib in this form.
Difficulty
The argument is short on paper; the formal difficulty is in the large-deviations step. Mathlib has the Chernoff inequality and the factorization of moment generating functions of independent sums, but the step from there to requires passing to a supremum over in the extended reals, handling where , and the case . The positivity requires , i.e. differentiating the moment generating function at an interior point of its domain. A naive attempt that works with real logarithms fails at probability zero, where is not , and an attempt that bounds by Chebyshev's inequality gives only a polynomial rate.
Formalization scope
The feasible set is a Finset of an arbitrary type X, assumed nonempty; minima and maxima over it are Finset.inf' and Finset.sup', so they are attained. The sample is a sequence W : ℕ → Ω → 𝒲 of measurable maps on a probability space, mutually independent (iIndepFun) and identically distributed with W 0; the paper's is W (n - 1), and is the expectation of . Measurability of is Borel measurability. Logarithms of probabilities are ENNReal.log, with , and the limit superior in (2.5) is taken in the extended reals; this is the paper's convention and it makes the goal correct when , where the event is sure. Assumption A is stated as integrability of for near , not through the value of Mathlib's mgf, which is where the expectation is infinite. The logarithmic moment generating function is the published definition BellWilliams2001.ThresholdPolicy.logMGF (extended-real valued, off the domain of ), and is the supremum over only, as on p. 3. Wherever appears, the nonemptiness of is a hypothesis; the goal does not mention and needs none.
A trivializing formalization is ruled out: with Real.log the event " is sure" would give and make the goal false, and a rate function built from mgf would count infinite moment generating functions as ; neither encoding is used.
A complete development needs the Cramér–Chernoff bound for i.i.d. sums with an extended-real rate function, differentiability of the moment generating function at interior points, and the strong law of large numbers applied to finitely many functions. The large-deviations items (milestones 4 and 5, and ) are stated for an arbitrary i.i.d. real sequence and are reusable beyond this mission. Proofs of any item, including the optional examples, are welcome.
Selected references
- A. J. Kleywegt and A. Shapiro, The sample average approximation method for stochastic discrete optimization, preprint (two-author version), https://edoc.hu-berlin.de/server/api/core/bitstreams/2dcf2d5b-5fb9-44d1-b36a-5e378c803717/content. Published with T. Homem-de-Mello in SIAM J. Optim. 12(2):479–502, 2002. https://doi.org/10.1137/S1052623499363220
- A. Dembo and O. Zeitouni, Large Deviations Techniques and Applications, 2nd ed., Springer, 1998. https://doi.org/10.1007/978-1-4612-5320-4
- A. Shapiro, D. Dentcheva and A. Ruszczyński, Lectures on Stochastic Programming: Modeling and Theory, SIAM, 2009. https://doi.org/10.1137/1.9780898718751