Calibrated Learning and Correlated Equilibrium II: For Almost Every Game, Limits of Calibrated Learning Are Exactly the Correlated EquilibriaResearch Paper
Motivation
A correlated equilibrium (Aumann, 1974) is a joint distribution over the players' strategy profiles from which no player gains by deviating from a recommended strategy. A central question of learning in games is which equilibria repeated play of simple, myopic rules can reach. Foster and Vohra (1997) answered it for calibrated forecasting: if each player forecasts the opponent with a calibrated rule and best-responds to the forecast, the empirical distribution of play approaches the set of correlated equilibria (their Theorem 1). That result has become a basic reference point for no-regret and calibration-based learning in games (Hart and Mas-Colell, 2000).
This mission formalizes the paper's converse. Theorem 1 says calibrated learning ends up in the correlated equilibria; the converse says it can end up at any of them, for almost every game. Together the two results characterize exactly which long-run outcomes calibrated learning with best responses can produce.
Setting
Two players choose strategies from finite sets with elements and with elements; player receives payoff and maximizes it. A game is a pair of real matrices, that is, a point of ; a set of games has measure zero if it is Lebesgue-null in .
A joint distribution on is a correlated equilibrium if for every map , , and symmetrically for player 2. is the set of correlated equilibria.
Play is repeated in rounds . Before each round, player 1 issues a forecast , a probability vector over , produced by a deterministic forecasting rule from the history of play so far; player 2 likewise forecasts player 1. For a forecast sequence and the opponent's plays , let be the number of the first rounds with forecast , and the fraction of those rounds in which the opponent played (zero if ). The forecasts are calibrated if for every
Each player then plays of its forecast, where the best-reply function picks a best response to every forecast and does not depend on the round. is the fraction of the first rounds in which was played.
, the set of limit points of calibrated forecasts, consists of the for which some best-reply functions and some calibrated forecasting rules make for all .
Formalization targets
Goal: Theorem 2 (p. 47)
Milestones
- for every game (Theorem 1 restated, p. 46).
- Every joint distribution is the limiting empirical distribution of a deterministic play sequence supported on (p. 47).
- Along such a sequence the conditional forecasts and are calibrated (p. 47).
- In a correlated equilibrium each recommended strategy is a best response to its conditional forecast (p. 47).
- For almost every payoff matrix, each set of forecasts to which is a best response is either empty or contains a forecast in the open simplex at which is the unique best response (p. 47).
- The perturbed forecasts converge to and keep a unique best reply (p. 48).
- For the game of p. 48, a correlated equilibrium lies outside , and is the single point mass on .
Significance
Theorem 1 by itself leaves open whether calibration selects among correlated equilibria, for instance toward Nash equilibria or toward particular payoffs. Theorem 2 closes that question negatively for generic games: every correlated equilibrium is the genuine limit, not merely an accumulation point, of calibrated play with stationary best replies. As the paper notes, adding the assumption that the limit exists therefore does not refine the equilibrium reached, in contrast with Fudenberg and Kreps's result for asymptotically myopic Bayesian play. The example shows that the genericity hypothesis cannot be dropped.
The results are proved in the 1997 paper; none is formalized. A complete development produces a reusable layer for repeated two-player games (calibration, empirical distributions, best-reply maps, correlated equilibria), a genericity lemma for best-response regions that is useful beyond this paper, and a machine-checked version of an argument that the paper gives only in outline.
Difficulty
The natural construction takes a correlated equilibrium , a play sequence realizing , and forecasts equal to the conditional distributions of . The forecasts are then calibrated and each played strategy is a best response. What fails is the best-reply function: two strategies can have the same conditional forecast , while a stationary maps to only one strategy. Separating them needs forecasts near at which each is the unique best response, and that exists only when the best-response regions have nonempty relative interior, a property that fails on a null set of games (the example) and whose genericity must be proved. The perturbed forecasts are no longer exactly equal to the conditional frequencies, so calibration has to be re-established with errors that vanish along the sequence.
Formalization scope
- Strategies are
Fin mandFin n; payoffs are real matrices; players maximize. A game is a point of(Fin m → Fin n → ℝ) × (Fin m → Fin n → ℝ)with Mathlib'svolume(Lebesgue measure on ), and "almost every" is∀ᵐ. - A correlated equilibrium is the joint-distribution form of p. 44 (a joint distribution with the two deviation inequalities).
- Forecasting rules map finite histories to probability vectors; the play is generated recursively from the rules and the best-reply functions. Best-reply functions are arbitrary stationary selections of a best response at every probability vector; they may not depend on the round, since round-dependent tie-breaking enlarges (the matching pennies example of p. 46).
- Rounds are ; . The calibration sum runs over the forecasts issued so far, which is the paper's sum over all since the other terms vanish.
- requires convergence of , not a subsequence. Replacing the limit by a limit point, or stating only , does not formalize the theorem.
- The page's sentence "Almost every game has the property that all the sets have non-empty interior" is false for dominated strategies ( on an open set of games). Milestone 5 states the dichotomy the page's argument proves: empty or with relative interior.
- The page prints the denominator of as ; the mission uses .
The theorems are stated from the mission's own definitions of the game, calibration and ; no statement is vacuous: at or both sides of the goal are empty, and the example exercises every definition. Contributions welcome: proofs of the milestones in any order, the genericity lemma (Lebesgue-null sets of linear degeneracies), and the calibration estimates for the perturbed forecasts.
Selected references
- D. P. Foster and R. V. Vohra, Calibrated learning and correlated equilibrium, Games and Economic Behavior 21 (1997), 40–55. https://doi.org/10.1006/game.1997.0595
- R. J. Aumann, Subjectivity and correlation in randomized strategies, Journal of Mathematical Economics 1 (1974), 67–96. https://doi.org/10.1016/0304-4068(74)90037-8
- A. P. Dawid, The well-calibrated Bayesian, Journal of the American Statistical Association 77 (1982), 605–610. https://doi.org/10.1080/01621459.1982.10477856
- D. Fudenberg and D. M. Kreps, Learning mixed equilibria, Games and Economic Behavior 5 (1993), 320–367. https://doi.org/10.1006/game.1993.1021
- S. Hart and A. Mas-Colell, A simple adaptive procedure leading to correlated equilibrium, Econometrica 68 (2000), 1127–1150. https://doi.org/10.1111/1468-0262.00153