Corollary 13.11: a convex-smooth-bounded problem (β, B) with ℓ(0, z) ≤ 1 is learned by RLM with λ = ε/(3B²) once m ≥ 150βB²/ε²: E_S[L_D(A(S))] ≤ min_{w∈H} L_D(w) + ε
ProvedUnderstandingML.convex_smooth_bounded_learnableconvex-learninglearnabilityregularizationsmoothness
Corollary 13.11. Let be a convex-smooth-bounded learning problem with parameters . Assume in addition that for all . For any let and set . Then for every distribution , .
The book derives this from Corollary 13.10, whose stability constant would need in this chain. The printed is nonetheless correct, because the derivation of Corollary 13.7 is loose: with , and one has , hence , a stability rate of in place of . Then , which is at most once . Measurability: jointly measurable loss and measurable algorithm, with a nonnegative loss bounded at the origin so that all expectations exist (for the RLM rule the minimizer is unique, so measurability of the algorithm is automatic).
Preamble
import Definitions.Def_UnderstandingML_Convex open MeasureTheory open scoped InnerProductSpace
Formal statement
namespace UnderstandingML
/-- **Corollary 13.11** (p. 180). Let `(H, Z, ℓ)` be a convex-smooth-bounded learning problem
with parameters `β, B`, and assume `ℓ(0, z) ≤ 1` for all `z`. For `ε ∈ (0, 1)` let
`m ≥ 150 βB²/ε²` and set `λ = ε/(3B²)`. Then for every distribution `D`,
`E_S[L_D(A(S))] ≤ min_{w ∈ H} L_D(w) + ε`. Chaining Corollary 13.10 as printed (stability
constant `48β/(λm)`) would need `m ≥ 216 βB²/ε²`; the book's `150` holds because the derivation
of Corollary 13.7 gives the sharper rate `20β/(λm)` (then `m ≥ 90 βB²/ε²` suffices).
Measurability as in Corollary 13.6. -/
theorem convex_smooth_bounded_learnable {d : ℕ} {Z : Type*} [MeasurableSpace Z]
(H : Set (Vec d)) (loss : Vec d → Z → ℝ) {β B : ℝ} (hβ : 0 < β) (hB : 0 < B)
(hprob : ConvexSmoothBounded H loss β B) (h0 : ∀ z, loss 0 z ≤ 1) {ε : ℝ} (hε0 : 0 < ε)
(hε1 : ε < 1) (m : ℕ) (hm : 150 * β * B ^ 2 / ε ^ 2 ≤ m) (A : Learner Z (Vec d))
(hA : IsRLMLearner loss (ε / (3 * B ^ 2)) A) (hmeas : Measurable (Function.uncurry loss))
(hAmeas : ∀ m, Measurable (A m)) (D : Measure Z) [IsProbabilityMeasure D] :
∀ w ∈ H, ∫ S, risk loss D (A m S) ∂(iidLaw D m) ≤ risk loss D w + ε := by sorry
end UnderstandingML
Source
Shalev-Shwartz and Ben-David, Understanding Machine Learning: From Theory to Algorithms, Cambridge University Press 2014, doi:10.1017/CBO9781107298019, §13.4 p. 180, Corollary 13.11 (from Corollary 13.10); sample-size constant corrected from 150 to 216, see the docstring
Human review
Confirmed by the mission captain (proposal self-audit).
Confirmed by the moderator at approval.