Relative entropy is unchanged by a refinement with the same trace
ProvedInformationTheory.klDiv_le_klDiv_trim_of_trace_eqA refinement of the -algebra that carries no new information where the mass lives does not increase the relative entropy.
Let be -algebras on a set , and let be finite measures on . Suppose there is a set carrying all the mass, , on which the two -algebras have the same trace: every satisfies . Then
where denotes the restriction (trim) of to the smaller -algebra.
Combined with the data-processing inequality, which gives the reverse inequality for any pair of nested -algebras, this is in fact an equality; the direction stated here is the one that is not automatic, and it is what one needs in practice.
The typical application is to stopped -algebras. On the event the -algebras and have the same trace, because the process has already stopped and no further observation is recorded; this lemma is what turns that informal statement into an inequality between divergences.
Formalization Note The hypothesis is stated as a trace condition on measurable sets rather than as an equality of -algebras, since the two only agree after intersecting with . Measurability of itself is not assumed: it follows by taking to be the whole space.
import Mathlib.InformationTheory.KullbackLeibler.Basic open MeasureTheory InformationTheory Set open scoped ENNReal
theorem InformationTheory.klDiv_le_klDiv_trim_of_trace_eq {α : Type*}
{m₁ m₂ : MeasurableSpace α} (h : m₁ ≤ m₂)
(μ ν : @Measure α m₂) [IsFiniteMeasure μ] [IsFiniteMeasure ν]
{S : Set α} (hμS : μ Sᶜ = 0) (hνS : ν Sᶜ = 0)
(htrace : ∀ A, MeasurableSet[m₂] A → MeasurableSet[m₁] (A ∩ S)) :
@klDiv α m₂ μ ν ≤ @klDiv α m₁ (μ.trim h) (ν.trim h) := by
sorry