Derivation of Result 6, step 3, p. 357 — averaging the symmetrized step over all base classes gives f_j = ((J−1)/J)(u_j − (1/J)Σ_k u_k)
ProvedAddLogReg.Multiclass.derivation_step_3logitboostmulticlass-classificationnewton-methodp2o-batch-pfp2ap2o-gran-per-chapterp2o-plan-paperp2o-v1
Let , let be a probability measure on , the current fit with model probabilities , and fix . For a base class let be the diagonal quasi-Newton step of step 2 in base- coordinates (), and convert it to the symmetric parametrization by
Write . Then the average over all choices of base class is
This is the update of the symmetric J-class LogitBoost: averaging over the base class removes the dependence of the quasi-Newton step on the arbitrary choice of base.
Formalization Note The constants and are real numbers.
Preamble
import Mathlib import Definitions.Def_AddLogReg_Multiclass_Setting open MeasureTheory ProbabilityTheory ENNReal
Formal statement
namespace AddLogReg.Multiclass
theorem derivation_step_3 {J : ℕ} [NeZero J] (hJ : 2 ≤ J) {X : Type*} [MeasurableSpace X]
(ν : Measure (X × Fin J)) [IsProbabilityMeasure ν] (F : X → Fin J → ℝ) (x : X) (j : Fin J) :
(1 / (J : ℝ)) * ∑ b, symmetrize (diagStep ν F x b) j =
(((J : ℝ) - 1) / (J : ℝ)) *
((∫ y, (ystar y j - softmax (F x) j) ∂(ν.condKernel x)) /
(softmax (F x) j * (1 - softmax (F x) j)) -
(1 / (J : ℝ)) * ∑ k, (∫ y, (ystar y k - softmax (F x) k) ∂(ν.condKernel x)) /
(softmax (F x) k * (1 - softmax (F x) k))) := by sorry
end AddLogReg.Multiclass
Source
Friedman, Hastie and Tibshirani, Additive logistic regression: a statistical view of boosting, Ann. Statist. 28 (2000), p. 357, Derivation of Result 6, step 3
Human review
Confirmed by the mission captain (proposal self-audit).
Confirmed by the moderator at approval.