KL divergence decreases under restriction to a sub--algebra
ProvedInformationTheory.klDiv_trim_ledata-processinginformation-theorymeasure-theoryrelative-entropy
Let and be probability measures on a measurable space , and let be a sub--algebra. Restrict both measures to . Then
This is the data-processing inequality for forgetting information: observing only events in a smaller sigma-algebra cannot increase relative entropy. It is reusable for stopped experiments, sufficient statistics, coarse observations, and finite-horizon projections.
Formalization Note Restriction to a sub-sigma-algebra is represented by Measure.trim. Both divergences are extended nonnegative real numbers, so the statement includes singular and infinite-divergence cases.
Preamble
import Mathlib.InformationTheory.KullbackLeibler.Basic import Mathlib.MeasureTheory.Function.ConditionalExpectation.RadonNikodym import Mathlib.MeasureTheory.Function.ConditionalExpectation.CondJensen open MeasureTheory InformationTheory Set open scoped ENNReal
Formal statement
theorem InformationTheory.klDiv_trim_le {α : Type*} {m m₀ : MeasurableSpace α}
(hm : m ≤ m₀) (μ ν : @Measure α m₀)
[IsProbabilityMeasure μ] [IsProbabilityMeasure ν] :
@klDiv α m (μ.trim hm) (ν.trim hm) ≤ @klDiv α m₀ μ ν := by
sorrySource
Standard relative-entropy data-processing inequality; specialized to stopped bandit experiments in Lattimore and Szepesvári, Bandit Algorithms (CUP 2020), Exercise 15.7, printed p. 211. Formal ingredients: Mathlib `MeasureTheory.rnDeriv_trim` and conditional Jensen inequality `ConvexOn.map_condExp_le`.