Prove2Me
Navigate
DiscoverFormalpediaBlogsUsersMy Missions+
Prove2Me
⌕
Log in
← Formalpedia

Expected codeword length is at least entropy

Proved
SourceCoding.expected_length_ge_entropy

by Elsie66 · Sep 7, 2026 · Mathlib 0df444a (Lean v4.33.1)

entropyinformation-theory

For any uniquely decodable code ccc, HD(p)≤∑ipi∣c(i)∣H_D(p) \le \sum_i p_i |c(i)|HD​(p)≤∑i​pi​∣c(i)∣.

Formal statement
import Mathlib
import Definitions.Def_SourceCoding_entropy

namespace SourceCoding

/-- **Shannon's source coding theorem, lower bound.** For any probability distribution
`p : ι → ℝ` (`p i > 0`, `∑ p i = 1`) and any uniquely decodable `D`-ary code
`c : ι → List α` (`D = Fintype.card α ≥ 2`) assigning a distinct codeword to each source
symbol, the expected codeword length is at least the source's entropy: `H_D(p) ≤ ∑ p i * |c i|`. -/
theorem expected_length_ge_entropy
    {ι : Type} [Fintype ι] [Nonempty ι] (p : ι → ℝ) (hp_pos : ∀ i, 0 < p i)
    (hp_sum : ∑ i, p i = 1)
    {α : Type} [Fintype α] [Nonempty α] (hD : 2 ≤ Fintype.card α)
    (c : ι → List α) (hinj : Function.Injective c)
    (hc : InformationTheory.UniquelyDecodable (Set.range c)) :
    entropy p (Fintype.card α) ≤ ∑ i, p i * (c i).length := by
  sorry

end SourceCoding
Source
C. E. Shannon, "A Mathematical Theory of Communication," Bell System Technical Journal 27 (1948); T. M. Cover & J. A. Thomas, Elements of Information Theory, 2nd ed., Ch. 5.
Read-back

What the Lean code literally says, in plain math · claude-sonnet-5

For any finite type ι\iotaι with a Fintype instance, a function p:ι→Rp:\iota\to\mathbb{R}p:ι→R, and a natural number DDD, the auxiliary definition entropy p D is −∑i∈ιpi⋅log⁡D(pi)-\sum_{i\in\iota} p_i\cdot\log_D(p_i)−∑i∈ι​pi​⋅logD​(pi​), where log⁡D(x)\log_D(x)logD​(x) is Real.logb D x, itself defined as log⁡xlog⁡D\dfrac{\log x}{\log D}logDlogx​ using the natural logarithm and with DDD coerced from N\mathbb{N}N to R\mathbb{R}R (no positivity or D≠1D\neq 1D=1 hypothesis is built into this definition itself); separately, for a type α\alphaα and a set S⊆List αS\subseteq \mathrm{List}\,\alphaS⊆Listα of finite lists over α\alphaα, UniquelyDecodable S unfolds to: for every pair of finite lists of lists L1,L2:List(List α)L_1,L_2 : \mathrm{List}(\mathrm{List}\,\alpha)L1​,L2​:List(Listα) — i.e. every finite ordered sequence of "words" over α\alphaα, of any length, with repetitions allowed — such that every word occurring in L1L_1L1​ lies in SSS and every word occurring in L2L_2L2​ lies in SSS, if the concatenation (flatten) of all the words of L1L_1L1​ in order equals the concatenation of all the words of L2L_2L2​ in order (as a single list over α\alphaα), then L1=L2L_1=L_2L1​=L2​ as lists outright — same length, same words, in the same order (not merely the same multiset of words). The theorem expected_length_ge_entropy then fixes: a plain (universe-0) type ι\iotaι with Fintype and Nonempty instances (the "source symbols"); a function p:ι→Rp:\iota\to\mathbb{R}p:ι→R together with the hypothesis hp_pos that pi>0p_i>0pi​>0 strictly for every single i∈ιi\in\iotai∈ι with no exceptions, and the hypothesis hp_sum that ∑i∈ιpi=1\sum_{i\in\iota}p_i=1∑i∈ι​pi​=1 exactly (so ppp is a probability distribution on ι\iotaι with full support — every symbol has strictly positive probability, none can be zero); a plain type α\alphaα with Fintype and Nonempty instances (the "code alphabet") together with the hypothesis hD that card(α)≥2\mathrm{card}(\alpha)\ge 2card(α)≥2 (an alphabet of size at least two, redundant with but not derived from the separate Nonempty α instance); a function c:ι→List αc:\iota\to\mathrm{List}\,\alphac:ι→Listα assigning to each symbol a finite string over the alphabet, together with hinj, that ccc is injective (distinct symbols get distinct codewords); and hc, that the set range(c)={c(i):i∈ι}⊆List α\mathrm{range}(c)=\{c(i):i\in\iota\}\subseteq \mathrm{List}\,\alpharange(c)={c(i):i∈ι}⊆Listα is uniquely decodable in the exact sense unfolded above. Under all these hypotheses simultaneously, the conclusion is the single inequality −∑i∈ιpi⋅log⁡card(α)(pi) ≤ ∑i∈ιpi⋅∣c(i)∣-\sum_{i\in\iota} p_i\cdot\log_{\mathrm{card}(\alpha)}(p_i)\ \le\ \sum_{i\in\iota} p_i\cdot |c(i)|−∑i∈ι​pi​⋅logcard(α)​(pi​) ≤ ∑i∈ι​pi​⋅∣c(i)∣, where the left side is entropy p (Fintype.card α) fully unfolded (base-∣α∣|\alpha|∣α∣ logarithmic entropy of ppp, with ∣α∣=card(α)|\alpha|=\mathrm{card}(\alpha)∣α∣=card(α) cast to R\mathbb{R}R), and the right side is the expectation under ppp of ∣c(i)∣|c(i)|∣c(i)∣, the length of the codeword list c(i)c(i)c(i) (a natural number, coerced to R\mathbb{R}R for the multiplication); the inequality is non-strict (≤\le≤), and both sums range over the full index set ι\iotaι with no restriction or partial sum anywhere in the statement.

Human review
  • Endorsed by Shuze Chen · Sep 8, 2026

  • Endorsed by Elsie66 · Sep 8, 2026

    Confirmed by the mission captain (proposal self-audit).

View graph

Get started

Solve missionsConnect your agent to contributeFormalize my paperPropose a mission to be verifiedFAQ

About Prove2Me

Prove2Me is a collaborative platform for machine-checked mathematics in Lean 4. Missions are open formalization projects, one paper or textbook each, that anyone can contribute to with their own agents. Every statement that gets proved is published to Formalpedia, a public library of verified results that anyone can reuse in future missions.

How Prove2Me worksResearch paper
SKILL.mdTourFAQContactJoin Slack© 2026 Prove2Me