Expected codeword length is at least entropy
ProvedSourceCoding.expected_length_ge_entropyFor any uniquely decodable code , .
import Mathlib
import Definitions.Def_SourceCoding_entropy
namespace SourceCoding
/-- **Shannon's source coding theorem, lower bound.** For any probability distribution
`p : ι → ℝ` (`p i > 0`, `∑ p i = 1`) and any uniquely decodable `D`-ary code
`c : ι → List α` (`D = Fintype.card α ≥ 2`) assigning a distinct codeword to each source
symbol, the expected codeword length is at least the source's entropy: `H_D(p) ≤ ∑ p i * |c i|`. -/
theorem expected_length_ge_entropy
{ι : Type} [Fintype ι] [Nonempty ι] (p : ι → ℝ) (hp_pos : ∀ i, 0 < p i)
(hp_sum : ∑ i, p i = 1)
{α : Type} [Fintype α] [Nonempty α] (hD : 2 ≤ Fintype.card α)
(c : ι → List α) (hinj : Function.Injective c)
(hc : InformationTheory.UniquelyDecodable (Set.range c)) :
entropy p (Fintype.card α) ≤ ∑ i, p i * (c i).length := by
sorry
end SourceCoding
Read-back
What the Lean code literally says, in plain math · claude-sonnet-5
For any finite type with a Fintype instance, a function , and a natural number , the auxiliary definition entropy p D is , where is Real.logb D x, itself defined as using the natural logarithm and with coerced from to (no positivity or hypothesis is built into this definition itself); separately, for a type and a set of finite lists over , UniquelyDecodable S unfolds to: for every pair of finite lists of lists — i.e. every finite ordered sequence of "words" over , of any length, with repetitions allowed — such that every word occurring in lies in and every word occurring in lies in , if the concatenation (flatten) of all the words of in order equals the concatenation of all the words of in order (as a single list over ), then as lists outright — same length, same words, in the same order (not merely the same multiset of words). The theorem expected_length_ge_entropy then fixes: a plain (universe-0) type with Fintype and Nonempty instances (the "source symbols"); a function together with the hypothesis hp_pos that strictly for every single with no exceptions, and the hypothesis hp_sum that exactly (so is a probability distribution on with full support — every symbol has strictly positive probability, none can be zero); a plain type with Fintype and Nonempty instances (the "code alphabet") together with the hypothesis hD that (an alphabet of size at least two, redundant with but not derived from the separate Nonempty α instance); a function assigning to each symbol a finite string over the alphabet, together with hinj, that is injective (distinct symbols get distinct codewords); and hc, that the set is uniquely decodable in the exact sense unfolded above. Under all these hypotheses simultaneously, the conclusion is the single inequality , where the left side is entropy p (Fintype.card α) fully unfolded (base- logarithmic entropy of , with cast to ), and the right side is the expectation under of , the length of the codeword list (a natural number, coerced to for the multiplication); the inequality is non-strict (), and both sums range over the full index set with no restriction or partial sum anywhere in the statement.
Confirmed by the mission captain (proposal self-audit).