Additivity of relative entropy over a measurable partition
ProvedInformationTheory.klDiv_restrict_add_klDiv_restrict_complRelative entropy is additive over a measurable partition of the underlying space.
Let and be finite measures on a measurable space , and let . Writing for the restriction of to and for the Kullback-Leibler divergence,
Both sides are equal in : if is not absolutely continuous with respect to , or the log-likelihood ratio fails to be integrable, then the failure is inherited by at least one of the two halves and both sides are infinite.
This is the localisation step used whenever one compares the divergences carried by two nested -algebras that differ only outside a distinguished event: splitting along that event isolates the part where the two -algebras agree from the part where new information arrives. It is the measure-theoretic content behind the stopped chain rule for relative entropy.
Formalization Note The statement is for arbitrary finite measures, not only probability measures. This matters: the restriction of a probability measure to a proper subset is a sub-probability measure, so the additivity can only be stated in a form valid for finite measures. Mathlib's klDiv uses the mass-corrected definition , which is precisely the form that makes the identity above hold exactly.
import Mathlib.InformationTheory.KullbackLeibler.Basic open MeasureTheory InformationTheory Set open scoped ENNReal
theorem InformationTheory.klDiv_restrict_add_klDiv_restrict_compl {α : Type*}
{mα : MeasurableSpace α} (μ ν : Measure α) [IsFiniteMeasure μ] [IsFiniteMeasure ν]
{s : Set α} (hs : MeasurableSet s) :
klDiv μ ν = klDiv (μ.restrict s) (ν.restrict s) + klDiv (μ.restrict sᶜ) (ν.restrict sᶜ) := by
sorry