One-round divergence, with finiteness only on the policy's support
ProvedBanditAlgorithm.klDiv_banditStepKernel_eq_sum_of_supportOne round of the canonical bandit model costs the policy-weighted average of the arm divergences — with the finiteness hypothesis required only of the arms the policy can actually play.
Fix a policy , two environments , , and a history of completed rounds. Assume for every arm that gives positive probability. Then
Restricting the hypothesis to the support of the policy is what makes the identity applicable in general. An arm the learner never plays in this round may have — the two environments may even be mutually singular on that arm — without affecting either side: on the right its term is , and on the left the round's law never sees it. Demanding finiteness for all arms would rule out exactly these situations, which arise whenever a policy deterministically avoids an arm.
Formalization Note In the convention is what makes the sum on the right well behaved; no finiteness is asserted for unplayed arms. The proof needs absolute continuity of the reward distributions only for almost every arm drawn from , which is exactly the stated hypothesis, since on a finite arm set almost every drawn arm has positive probability.
import Mathlib.InformationTheory.KullbackLeibler.ChainRule import Definitions.Def_BanditPolicy open MeasureTheory ProbabilityTheory InformationTheory
theorem BanditAlgorithm.klDiv_banditStepKernel_eq_sum_of_support {k n : ℕ}
(nu nu' : StochasticBandit k)
(pi : BanditPolicy k) (h : BanditHistory k n)
(hKL : ∀ i, (pi.select n h) {i} ≠ 0 → klDiv (nu.P i) (nu'.P i) ≠ ⊤) :
klDiv (banditStepKernel nu pi n h) (banditStepKernel nu' pi n h) =
∑ i, ENNReal.ofReal ((pi.select n h).real {i}) *
klDiv (nu.P i) (nu'.P i) := by
sorry