Appendix D, p. 13 — KL divergence of a point mass
ProvedInfoGen.Gibbs.klDiv_dirac_eqgibbs-algorithmp2o-batch-pfp2ap2o-gran-per-chapterp2o-plan-paperp2o-v1point-massrelative-entropy
Let be a probability distribution on a countable hypothesis space and let be a hypothesis to which assigns positive mass. Then
The identity converts the comparison with a fixed hypothesis into the logarithmic prior-mass term in Corollary 2.
Formalization Note The paper states the identity for its minimizer ; the same formula holds for any with positive prior mass. Positivity excludes the infinite divergence case, since Lean's real logarithm at zero would otherwise give a junk value.
Preamble
import Mathlib import Definitions.Def_InfoGen_Gibbs_Setting open MeasureTheory ProbabilityTheory InformationTheory open scoped ENNReal NNReal ProbabilityTheory
Formal statement
namespace InfoGen.Gibbs
/-- The last sentence of Appendix D, PDF p. 13. -/
theorem klDiv_dirac_eq {W : Type*} [MeasurableSpace W]
[Countable W] [MeasurableSingletonClass W]
(Q : Measure W) [IsProbabilityMeasure Q] (w : W)
(hQ : Q {w} ≠ 0) :
klDiv (Measure.dirac w) Q =
ENNReal.ofReal (-Real.log (Q {w}).toReal) := by sorry
end InfoGen.Gibbs
Source
Xu & Raginsky, arXiv:1705.07809v2, App. D, final sentence after (D.15), p. 13
Human review
Confirmed by the mission captain (proposal self-audit).
Confirmed by the moderator at approval.