Derivation of Result 6, step 1, p. 357 — multilogit coordinates G_j = F_j − F_b with base class b give the probabilities (40) and G_j = log(p_j/p_b)
ProvedAddLogReg.Multiclass.base_probsmulticlass-classificationmultinomial-logistic-regressionp2o-batch-pfp2ap2o-gran-per-chapterp2o-plan-paperp2o-v1softmax
Let be a real vector, let be its probabilities (40), and fix a base class . Put , so that . Then for every ,
So the multilogit parametrization , , used in step 1 of the Derivation of Result 6, describes the same probabilities as the symmetric parametrization (40); this is what allows the score and Hessian of step 1 to be written in terms of .
Formalization Note The page takes and remarks that the choice of base class is arbitrary; here is any class. The statement is about the values at a fixed .
Preamble
import Mathlib import Definitions.Def_AddLogReg_Multiclass_Setting open MeasureTheory ProbabilityTheory ENNReal
Formal statement
namespace AddLogReg.Multiclass
theorem base_probs {J : ℕ} (F : Fin J → ℝ) (b j : Fin J) (hj : j ≠ b) :
Real.exp (F j - F b) / (1 + ∑ k ∈ Finset.univ.erase b, Real.exp (F k - F b)) = softmax F j ∧
1 / (1 + ∑ k ∈ Finset.univ.erase b, Real.exp (F k - F b)) = softmax F b ∧
Real.log (softmax F j / softmax F b) = F j - F b := by sorry
end AddLogReg.Multiclass
Source
Friedman, Hastie and Tibshirani, Additive logistic regression: a statistical view of boosting, Ann. Statist. 28 (2000), p. 357, Derivation of Result 6, step 1, first display (multilogit parametrization G_j(x))
Human review
Confirmed by the mission captain (proposal self-audit).
Confirmed by the moderator at approval.