Derivation of Result 4, p. 353 — first and second derivatives of E(e^{−y(F(x)+f(x))}|x) at f(x) = 0
ProvedAddLogReg.Gentle.derivation_derivsboostingexponential-lossgentle-adaboostnewton-methodp2o-batch-pfp2ap2o-gran-per-chapterp2o-plan-paperp2o-v1
Let be a probability measure on , the joint law of , let and fix . Write , a function of the real number . Then
- is differentiable at with
- at every the derivative is ;
- this first derivative is itself differentiable at , with
the last equality because .
These are the gradient and Hessian that the Newton step of Result 4 is built from.
Formalization Note Derivatives are stated with HasDerivAt, so differentiability is part of the claim. The second derivative is the derivative at of the explicit function , which conjunct 2 shows to be the derivative of at every . The statement holds for every (conditional expectations are integrals against the conditional law ν.condKernel x on the two-point label set).
Preamble
import Mathlib import Definitions.Def_AddLogReg_Gentle_Setting open MeasureTheory ProbabilityTheory
Formal statement
namespace AddLogReg.Gentle
theorem derivation_derivs {X : Type*} [MeasurableSpace X] (ν : Measure (X × Bool))
[IsProbabilityMeasure ν] (F : X → ℝ) (x : X) :
HasDerivAt (condCritAt ν F x)
(-∫ b, Real.exp (-(AddLogReg.ExpCrit.sgn b * F x)) * AddLogReg.ExpCrit.sgn b ∂(ν.condKernel x)) 0 ∧
(∀ t : ℝ, HasDerivAt (condCritAt ν F x)
(-∫ b, Real.exp (-(AddLogReg.ExpCrit.sgn b * (F x + t))) * AddLogReg.ExpCrit.sgn b ∂(ν.condKernel x)) t) ∧
HasDerivAt (fun t : ℝ => -∫ b, Real.exp (-(AddLogReg.ExpCrit.sgn b * (F x + t))) * AddLogReg.ExpCrit.sgn b ∂(ν.condKernel x))
(∫ b, Real.exp (-(AddLogReg.ExpCrit.sgn b * F x)) * AddLogReg.ExpCrit.sgn b ^ 2 ∂(ν.condKernel x)) 0 ∧
∫ b, Real.exp (-(AddLogReg.ExpCrit.sgn b * F x)) * AddLogReg.ExpCrit.sgn b ^ 2 ∂(ν.condKernel x) =
∫ b, Real.exp (-(AddLogReg.ExpCrit.sgn b * F x)) ∂(ν.condKernel x) := by sorry
end AddLogReg.Gentle
Source
Friedman, Hastie and Tibshirani, Additive logistic regression: a statistical view of boosting, Ann. Statist. 28 (2000), p. 353, §4.4, Derivation of Result 4, first and second displays
Human review
Confirmed by the mission captain (proposal self-audit).
Confirmed by the moderator at approval.