Conditional relative entropy as an average of fibrewise divergences
ProvedInformationTheory.klDiv_compProd_self_eq_lintegralThe conditional relative entropy of two composition-products with a common first marginal is the average of the fibrewise divergences.
Let be a finite measure on and let be finite kernels from to with for every . Then
In words: if two joint laws agree on the first coordinate and differ only in the conditional law of the second, the divergence between them is exactly the expected divergence between the conditionals.
This is the quantity usually written and called the conditional relative entropy. The chain rule for relative entropy is normally stated with this term on the right — as in Lattimore and Szepesvari's Exercise 14.12, — so this identity is what turns an abstract chain rule into a computation one round at a time.
Formalization Note Mathlib's chain rule klDiv_compProd_eq_add leaves the conditional term in the unevaluated form klDiv (μ ⊗ₘ κ) (μ ⊗ₘ η). The countability hypothesis on the pair of spaces is what makes the fibrewise Radon-Nikodym derivative jointly measurable, and is satisfied in the standard-Borel settings where the chain rule is applied.
import Mathlib.InformationTheory.KullbackLeibler.ChainRule import Mathlib.Probability.Kernel.RadonNikodym open MeasureTheory ProbabilityTheory InformationTheory open scoped ENNReal
theorem InformationTheory.klDiv_compProd_self_eq_lintegral {α β : Type*}
{mα : MeasurableSpace α} {mβ : MeasurableSpace β}
[MeasurableSpace.CountableOrCountablyGenerated α β]
(μ : Measure α) [IsFiniteMeasure μ] (κ η : Kernel α β)
[IsFiniteKernel κ] [IsFiniteKernel η] (hac : ∀ a, κ a ≪ η a) :
klDiv (μ ⊗ₘ κ) (μ ⊗ₘ η) = ∫⁻ a, klDiv (κ a) (η a) ∂μ := by
sorry