Derivation of Result 6, step 2, p. 357 — the diagonal quasi-Newton update g_j ← E(y*_j − p_j|x)/(p_j(1 − p_j))
ProvedAddLogReg.Multiclass.derivation_step_2logitboostmulticlass-classificationnewton-methodp2o-batch-pfp2ap2o-gran-per-chapterp2o-plan-paperp2o-v1
Let , let be a probability measure on and the current fit with model probabilities given by (40). Fix . With the score and Hessian of step 1:
- every diagonal entry of the Hessian is nonzero, ;
- for every base class and every , the Newton step with the Hessian replaced by its diagonal, , is
This is the quasi-Newton update of the Derivation: a diagonal approximation to the Hessian of step 1. It is not the full Newton step .
Formalization Note The non-vanishing of is proved, not assumed: the model probabilities (40) lie strictly between and when . The base coordinate of the step is set to .
Preamble
import Mathlib import Definitions.Def_AddLogReg_Multiclass_Setting open MeasureTheory ProbabilityTheory ENNReal
Formal statement
namespace AddLogReg.Multiclass
theorem derivation_step_2 {J : ℕ} [NeZero J] (hJ : 2 ≤ J) {X : Type*} [MeasurableSpace X]
(ν : Measure (X × Fin J)) [IsProbabilityMeasure ν] (F : X → Fin J → ℝ) (x : X) :
(∀ j, hess F x j j ≠ 0) ∧
∀ b j, j ≠ b → diagStep ν F x b j =
(∫ y, (ystar y j - softmax (F x) j) ∂(ν.condKernel x)) /
(softmax (F x) j * (1 - softmax (F x) j)) := by sorry
end AddLogReg.Multiclass
Source
Friedman, Hastie and Tibshirani, Additive logistic regression: a statistical view of boosting, Ann. Statist. 28 (2000), p. 357, Derivation of Result 6, step 2
Human review
Confirmed by the mission captain (proposal self-audit).
Confirmed by the moderator at approval.