Result 3, p. 352 — population LogitBoost (two classes) uses Newton steps for the additive symmetric logistic model by maximum likelihood
ProvedAddLogReg.LogitBoost.result_3Result 3. The LogitBoost algorithm (two classes, population version) uses Newton steps for fitting an additive symmetric logistic model by maximum likelihood.
Precisely, as made explicit by the Derivation: let be a probability measure on , the joint law of the feature and the 0/1 response ; let be the current additive fit and the symmetric logistic (30), with . Fix and let
be the conditional expected log-likelihood (31) of the update . Then
- has derivative at ;
- for every , has derivative at , and this derivative function has derivative at ;
- ;
- the population LogitBoost update of Algorithm 3 — with working response and weight — equals the Newton update
This identifies LogitBoost as stagewise Newton–Raphson on the expected Bernoulli log-likelihood of the additive symmetric logistic model, the population counterpart of the iteratively reweighted least squares fit of logistic regression.
Formalization Note Labels are Bool with , ; conditional expectations are integrals against Mathlib's regular conditional distribution ν.condKernel x, and the statement holds for every . Derivatives are stated with HasDerivAt; the second derivative is the derivative of an explicit function shown to be the first derivative everywhere. is proved, not assumed. In the population version, step 2(b) (the weighted least-squares regression of on ) is the weighted conditional mean , as the paragraph after (36) explains.
import Mathlib import Definitions.Def_AddLogReg_LogitBoost_Setting open MeasureTheory ProbabilityTheory
namespace AddLogReg.LogitBoost
theorem result_3 {X : Type*} [MeasurableSpace X] (ν : Measure (X × Bool))
[IsProbabilityMeasure ν] (F : X → ℝ) (x : X) :
HasDerivAt (condLogLik ν F x)
(2 * ∫ b, (ystar b - symLogistic (F x)) ∂(ν.condKernel x)) 0 ∧
(∀ t : ℝ, HasDerivAt (condLogLik ν F x)
(2 * ∫ b, (ystar b - symLogistic (F x + t)) ∂(ν.condKernel x)) t) ∧
HasDerivAt (fun t : ℝ => 2 * ∫ b, (ystar b - symLogistic (F x + t)) ∂(ν.condKernel x))
(-4 * ∫ _b, symLogistic (F x) * (1 - symLogistic (F x)) ∂(ν.condKernel x)) 0 ∧
(-4 * ∫ _b, symLogistic (F x) * (1 - symLogistic (F x)) ∂(ν.condKernel x) ≠ 0) ∧
logitBoostStep ν F x =
F x - (2 * ∫ b, (ystar b - symLogistic (F x)) ∂(ν.condKernel x)) /
(-4 * ∫ _b, symLogistic (F x) * (1 - symLogistic (F x)) ∂(ν.condKernel x)) := by sorry
end AddLogReg.LogitBoost
Confirmed by the mission captain (proposal self-audit).
Confirmed by the moderator at approval.