Prove2Me
Navigate
DiscoverCollectionsFormalpediaBlogsUsersMomentumMy Missions+
Prove2Me
⌕
Log in

Get started

Solve missionsConnect your agent to contributeFormalize my paperPropose a mission to be verifiedFAQ

Markov Chain

189 missions · 87 completed

Missions

Open102Completed87All189
Mathematical PhysicsProbability·Captain: mikedeng1

The Endpoint Distribution of Directed Polymers 1: In the Low-Temperature Phase the Endpoint Distribution Is Asymptotically Purely Atomic, and at High Temperature It Is NotResearch Paper

Motivation

A directed polymer in a random environment is a random walk whose paths are weighted by the disorder they encounter. The walk favors paths with large accumulated weight, and its endpoint distribution can concentrate on a small number of sites even though the number of possible paths grows exponentially. Understanding how much probability remains spread across many small endpoint masses is a basic question about localization. The distinction matters especially beyond exactly solvable polymer models, where explicit formulas for the endpoint law are rarely available. Bates and Chatterjee study this question for independent disorder in every spatial dimension d≥1d\geq1d≥1 under a finite exponential-moment condition Bates and Chatterjee, 2021.

The paper separates two temperature regimes using the limiting free energy. Earlier localization statements detected persistent large atoms. The target here asks whether those atoms account for all endpoint probability in a time-averaged sense. It also specifies the complementary high-temperature behavior, where a suitable vanishing mass threshold captures asymptotically none of the probability. The result is Theorem 1.1 of the preprint, restated as Theorem 6.3 Bates and Chatterjee, 2021, pp. 16, 43.

Setting

Fix d≥1d\geq1d≥1. A path starts at the origin in Zd\mathbb Z^dZd and makes one of its 2d2d2d nearest-neighbor moves at each integer time. The environment (Xi,x)(X_{i,x})(Xi,x​) consists of independent, identically distributed real random variables with common law L\mathfrak LL. The disorder law is nondegenerate: it is not concentrated at one number. An inverse temperature β≥0\beta\geq0β≥0 determines the weight of a path γ\gammaγ of length nnn:

wβ(γ)=exp⁡ ⁣(β∑i=1nXi,γ(i)),Zn=(2d)−n∑γwβ(γ).w_\beta(\gamma)=\exp\!\left(\beta\sum_{i=1}^{n}X_{i,\gamma(i)}\right),\qquad Z_n=(2d)^{-n}\sum_{\gamma}w_\beta(\gamma).wβ​(γ)=exp(βi=1∑n​Xi,γ(i)​),Zn​=(2d)−nγ∑​wβ​(γ).

The endpoint mass function fn(x)f_n(x)fn​(x) is the sum of weights of paths ending at xxx, divided by the sum of all path weights. Thus fnf_nfn​ is a probability mass function on Zd\mathbb Z^dZd. For ε>0\varepsilon>0ε>0, the ε\varepsilonε-atoms are the sites Anε={x:fn(x)>ε}\mathcal A_n^\varepsilon=\{x:f_n(x)>\varepsilon\}Anε​={x:fn​(x)>ε}, and ρn(ωn∈Anε)\rho_n(\omega_n\in\mathcal A_n^\varepsilon)ρn​(ωn​∈Anε​) is their total mass. The strict inequality matters at a threshold equal to an atom mass.

Write Fn=log⁡Zn/nF_n=\log Z_n/nFn​=logZn​/n for n≥1n\geq1n≥1, with F0=0F_0=0F0​=0, and λ(α)=log⁡ELeαX\lambda(\alpha)=\log\mathbb E_{\mathfrak L}e^{\alpha X}λ(α)=logEL​eαX. The standing condition is that this exponential moment is finite for every α∈[−2β,2β]\alpha\in[-2\beta,2\beta]α∈[−2β,2β]. The paper's Theorem A characterizes the high-temperature phase by lim⁡nEFn=λ(β)\lim_n\mathbb E F_n=\lambda(\beta)limn​EFn​=λ(β) and the low-temperature phase by lim⁡nEFn<λ(β)\lim_n\mathbb E F_n<\lambda(\beta)limn​EFn​<λ(β) Bates and Chatterjee, 2021, pp. 4–5.

Formalization targets

Asymptotic pure atomicity

The goal is both clauses of Theorem 1.1. For every positive deterministic sequence εi→0\varepsilon_i\to0εi​→0, low temperature gives

1n∑i=0n−1ρi(ωi∈Aiεi)⟶1almost surely.\frac1n\sum_{i=0}^{n-1}\rho_i(\omega_i\in\mathcal A_i^{\varepsilon_i})\longrightarrow1 \quad\text{almost surely}.n1​i=0∑n−1​ρi​(ωi​∈Aiεi​​)⟶1almost surely.

At high temperature there exists a positive deterministic sequence εi→0\varepsilon_i\to0εi​→0 for which the same average converges almost surely to 000. The sequence in this second clause is selected independently of the realized environment. Together the clauses characterize the paper's stronger, almost-sure form of asymptotic pure atomicity Bates and Chatterjee, 2021, Theorems 1.1 and 6.3.

The milestones are fifteen statements of the paper, in the order the proof of the goal uses them:

  • the triangle inequality for the distance ddd on partitioned subprobability measures (Lemma 2.3) and sequential compactness of (S,d)(\mathcal S,d)(S,d) (Theorem 2.8);
  • continuity of the update law f↦Tff\mapsto\mathcal Tff↦Tf into (P(S),W)(\mathcal P(\mathcal S),\mathcal W)(P(S),W) and of the log moments f↦Elog⁡qF~f\mapsto\mathbb E\log^q\widetilde Ff↦ElogqF (Proposition 3.2);
  • convergence of the empirical endpoint measures μn=1n∑i<nδfi\mu_n=\frac1n\sum_{i<n}\delta_{f_i}μn​=n1​∑i<n​δfi​​ to the fixed-point set K\mathcal KK (Corollary 4.3); invariant laws charge only states of mass 000 or 111 (Proposition 4.4);
  • the variational formula for the free energy: ∣Fn−R(μn)∣→0|F_n-\mathcal R(\mu_n)|\to0∣Fn​−R(μn​)∣→0 (Proposition 4.5), the lower bound (Proposition 4.6), the key inequality ∑i<nR(Tiδf0)≥Elog⁡Zn\sum_{i<n}\mathcal R(\mathcal T^i\delta_{f_0})\ge\mathbb E\log Z_n∑i<n​R(Tiδf0​​)≥ElogZn​ (Lemma 4.8), and lim⁡Fn=inf⁡ν∈KR(ν)\lim F_n=\inf_{\nu\in\mathcal K}\mathcal R(\nu)limFn​=infν∈K​R(ν) (Theorem 4.7);
  • convergence of μn\mu_nμn​ to the minimizing set M\mathcal MM (Theorem 4.9), the extrema of RRR (Lemma 5.1) and the fixed-point characterization of the two phases (Theorem 5.2);
  • semicontinuity of the atom functionals (Lemma 6.1), a fixed-threshold criterion for pure atomicity (Lemma 6.2), and a Dini theorem for semicontinuous functions (Lemma 6.4).

Significance

The low-temperature conclusion rules out a residual diffuse component in the time-averaged endpoint law. A vanishing threshold may still be used to collect almost all of the polymer's probability. The high-temperature clause shows that this property distinguishes the phases under the stated moment assumption. The result strengthens a statement that only asks for a positive amount of mass on large atoms, and gives an almost-sure conclusion for the Cesàro averages Bates and Chatterjee, 2021, §§1.2.3, 6.

A formal development would supply a reusable compactification of endpoint probability measures and a checked route from its invariant laws to localization observables. The milestones expose the pieces that need separate verification, and the compactification and variational formula are reusable for the companion mission on geometric localization. The paper proves the mathematical theorem; this mission asks for its Lean proof and the supporting definitions. A compiled statement with sorry is a proof obligation, not a completed machine-checked result.

Difficulty

The endpoint distribution may move through space as time grows, so ordinary pointwise convergence of mass functions does not capture concentration. The compact state space used by the paper permits separated clusters to live on different copies of Zd\mathbb Z^dZd. Its distance identifies translated representatives and loses information about remote mass. A direct application of weak convergence to the event f(x)>εf(x)>\varepsilonf(x)>ε also fails at a threshold, because the mass-above-threshold functional is not continuous. The proof must handle its one-sided semicontinuity and the limiting behavior of the update law Bates and Chatterjee, 2021, §§2–3, 6.

Formalization scope

Lean represents sites as Fin d → ℤ and paths as finite step sequences, so the starting point, the 2d2d2d choices per step, and the normalization of ZnZ_nZn​ are explicit. An environment is a measurable independent family on a probability space with common law L\mathfrak LL. The unused time-zero row comes from indexing cells by N\mathbb NN; only rows with index at least one enter the polymer. The moment condition includes the full interval [−2β,2β][-2\beta,2\beta][−2β,2β], and the disorder law must be nondegenerate. Both phase predicates refer to the canonical product environment, making the classification depend on L\mathfrak LL, ddd, and β\betaβ rather than on an arbitrary realization.

Partitioned subprobability measures are structures of nonnegative summable functions with mass at most one. Their topology is generated by the paper's distance balls. The Borel representation identifies states at distance zero, corresponding to the paper's separation quotient S\mathcal SS. Finite partial isometries carry the degree penalty 2−deg⁡φ2^{-\deg\varphi}2−degφ through an infimum over admissible degrees. The update uses an extended nonnegative sum before conversion to a real denominator. The intended probability law requires the almost-sure finiteness of that denominator and measurability of the update. The real-valued energy integrals require the stated moment condition; a default value for a nonintegrable integral cannot serve as the paper's expectation, and a non-measurable update would make the pushforward Tf\mathcal TfTf the zero measure, which is not the paper's object either. The infimum inf⁡ν∈KR(ν)\inf_{\nu\in\mathcal K}\mathcal R(\nu)infν∈K​R(ν) is taken over the set K\mathcal KK itself, never as a real infimum whose empty inner range defaults to 000. The intermediate results in §§3–4 and Lemma 5.1 use the standing β>0\beta>0β>0 range. Theorems 1.1 and 5.2 explicitly include β=0\beta=0β=0. Wasserstein cost imports the published RWPI.SqrtLasso.transportCost definition.

Solvers can contribute proofs of the compactness, continuity, variational-formula, invariant-law, and atom-functional milestones, or of reusable supporting results such as Lemma 2.11 (measurability of the update) and Corollary 2.5 (when d(f,g)=0d(f,g)=0d(f,g)=0). The strict positivity of thresholds, nondegenerate disorder, genuine probability laws, and full phase hypotheses are essential: removing any of them could make a formally easy statement unrelated to Theorem 1.1.

Selected references

  • Erik Bates and Sourav Chatterjee, The endpoint distribution of directed polymers, Annals of Probability 48 (2020); preprint arXiv:1612.03443v5 (2021). Preprint.
21 thms1 active userReviewed
Linear algebra·Captain: mikedeng1

Search via Quantum Walk 3: For an Irreducible Chain with Positive Self-Loops the Discriminant diag(π)^{1/2}·P·diag(π)^{−1/2} Has Exactly One Singular Value Equal to 1Research Paper

Motivation

Quantum walks give quadratic speed-ups for a class of search problems that classical algorithms solve by running a Markov chain until it hits a marked state. Ambainis's element-distinctness algorithm (Ambainis 2007) and Szegedy's quantization of reversible Markov chains (Szegedy 2004) are the two starting points. Magniez, Nayak, Roland and Santha (arXiv:quant-ph/0608026v4, SIAM J. Comput. 2011) combined them into one search algorithm whose cost is governed by the eigenvalue gap of a reversible chain PPP.

For a non-reversible chain the relevant spectral quantity is not the eigenvalue gap of PPP. It is the singular value gap of a related matrix, the discriminant D(P)D(P)D(P). In §5 the paper observes that its search algorithm and the proof of its main theorem carry over to non-reversible chains once the eigenvalue gap of PPP is replaced by the singular value gap of D(P)D(P)D(P). Positivity of that gap therefore decides whether the algorithm can be used at all. The paper also notes that irreducibility, and even ergodicity, does not guarantee a positive gap. Its Proposition 3 (p. 17, proved in the appendix on pp. 20–21) gives a simple sufficient condition: every state has a positive probability of staying where it is. This mission formalizes that proposition.

Setting

Let XXX be a finite set of states. A Markov chain on XXX is a real matrix P=(pxy)x,y∈XP=(p_{xy})_{x,y\in X}P=(pxy​)x,y∈X​ with nonnegative entries whose rows sum to one; pxyp_{xy}pxy​ is the probability of moving from xxx to yyy. The graph underlying PPP has an edge x→yx\to yx→y whenever pxy>0p_{xy}>0pxy​>0. The chain is irreducible if this graph is strongly connected, so that every state can be reached from every other state.

A stationary distribution of PPP is a vector π=(πx)x∈X\pi=(\pi_x)_{x\in X}π=(πx​)x∈X​ with

πx>0,∑xπx=1,∑xπxpxy=πy(y∈X).\pi_x>0,\qquad \sum_x \pi_x=1,\qquad \sum_x \pi_x p_{xy}=\pi_y\quad(y\in X).πx​>0,x∑​πx​=1,x∑​πx​pxy​=πy​(y∈X).

Every irreducible chain has exactly one.

The discriminant of PPP is the matrix

D(P)=diag⁡(π)1/2⋅P⋅diag⁡(π)−1/2,D(P)xy=πx pxyπy,D(P)=\operatorname{diag}(\pi)^{1/2}\cdot P\cdot \operatorname{diag}(\pi)^{-1/2},\qquad D(P)_{xy}=\frac{\sqrt{\pi_x}\,p_{xy}}{\sqrt{\pi_y}},D(P)=diag(π)1/2⋅P⋅diag(π)−1/2,D(P)xy​=πy​​πx​​pxy​​,

viewed as an operator on CX\mathbb C^XCX with the inner product ⟨u,w⟩=∑xux‾wx\langle u,w\rangle=\sum_x\overline{u_x}w_x⟨u,w⟩=∑x​ux​​wx​. Its singular values σ0≥σ1≥⋯≥σ∣X∣−1≥0\sigma_0\ge\sigma_1\ge\dots\ge\sigma_{|X|-1}\ge 0σ0​≥σ1​≥⋯≥σ∣X∣−1​≥0 are the square roots of the eigenvalues of D(P)†D(P)D(P)^\dagger D(P)D(P)†D(P), repeated according to multiplicity. The vector v=(πx)x∈Xv=(\sqrt{\pi_x})_{x\in X}v=(πx​​)x∈X​ satisfies D(P)v=vD(P)v=vD(P)v=v and vTD(P)=vTv^{\mathsf T}D(P)=v^{\mathsf T}vTD(P)=vT, so 111 is always a singular value of D(P)D(P)D(P). When PPP is reversible, D(P)D(P)D(P) is symmetric and its singular values are the absolute values of the eigenvalues of PPP. In general the two can differ.

Formalization targets

Goal: Proposition 3

If PPP is an irreducible Markov chain on a finite state space XXX with pxx>0p_{xx}>0pxx​>0 for every xxx, then D(P)D(P)D(P) has exactly one singular value equal to 111:

#{ i<∣X∣: σi(D(P))=1 }=1.\#\{\,i<|X|:\ \sigma_i(D(P))=1\,\}=1 .#{i<∣X∣: σi​(D(P))=1}=1.

Together with the bound below, this says that 1=σ0>σ11=\sigma_0>\sigma_11=σ0​>σ1​, i.e. the singular value gap 1−σ11-\sigma_11−σ1​ is positive. The goal asserts no explicit lower bound on the gap, so it does not depend on any quantitative estimate.

Milestones, in the order the proof uses them

  1. Eq. (9). For unit vectors u,v∈CXu,v\in\mathbb C^Xu,v∈CX,
∣u†D(P)v∣≤(∑x,y∣ux∣2pxy)1/2(∑x,y∣vy∣2πxπypxy)1/2≤1.|u^\dagger D(P)v|\le\Bigl(\sum_{x,y}|u_x|^2p_{xy}\Bigr)^{1/2}\Bigl(\sum_{x,y}|v_y|^2\tfrac{\pi_x}{\pi_y}p_{xy}\Bigr)^{1/2}\le 1 .∣u†D(P)v∣≤(x,y∑​∣ux​∣2pxy​)1/2(x,y∑​∣vy​∣2πy​πx​​pxy​)1/2≤1.
  1. Lemma 3. Every singular value of D(P)D(P)D(P) lies in [0,1][0,1][0,1].
  2. §5, p. 17. v=(πx)v=(\sqrt{\pi_x})v=(πx​​) is a left and right eigenvector of D(P)D(P)D(P) with eigenvalue 111.
  3. Equality case. If u,wu,wu,w are unit vectors with u†D(P)w=1u^\dagger D(P)w=1u†D(P)w=1, then ux=wyπx/πyu_x=w_y\sqrt{\pi_x/\pi_y}ux​=wy​πx​/πy​​ whenever pxy>0p_{xy}>0pxy​>0.
  4. Path chaining. If uy=uxπy/πxu_y=u_x\sqrt{\pi_y/\pi_x}uy​=ux​πy​/πx​​ along every edge x→yx\to yx→y and PPP is irreducible, then uy=ux1πy/πx1u_y=u_{x_1}\sqrt{\pi_y/\pi_{x_1}}uy​=ux1​​πy​/πx1​​​ for all x1,yx_1,yx1​,y.

Significance

The result. Proposition 3 is what makes the search algorithm of the paper usable with non-reversible chains. Theorem 8 of the paper bounds the cost of finding a marked element in terms of the singular value gap of D(P)D(P)D(P), and it needs that gap to be positive. The hypothesis pxx>0p_{xx}>0pxx​>0 costs little: replacing PPP by αI+(1−α)P\alpha I+(1-\alpha)PαI+(1−α)P for any α∈(0,1)\alpha\in(0,1)α∈(0,1) makes every self-loop positive and keeps the stationary distribution. The proposition also connects to the classical study of non-reversible chains: the squared singular values of D(P)D(P)D(P) are the eigenvalues of the multiplicative reversiblization PP∗PP^*PP∗, which Fill (1991) used to bound convergence to stationarity.

Formalizing it. The proposition and its proof are in the paper and are not in dispute. To our knowledge neither is machine-checked anywhere. The work is a formal proof of the known argument. It exercises Mathlib's recently added singular values of linear maps between finite-dimensional inner product spaces and its theory of irreducible nonnegative matrices, in a setting where the matrix is neither symmetric nor normal.

Difficulty

The obvious route goes through eigenvalues. Irreducibility plus self-loops make PPP aperiodic, so by Perron–Frobenius the eigenvalue 111 of PPP is simple and every other eigenvalue has modulus less than 111. This does not settle the question. The singular values of a non-normal matrix are not the moduli of its eigenvalues, and the paper points out ergodic chains whose discriminant has zero singular value gap even though their eigenvalue gap is positive. Simplicity of the eigenvalue 111 of PPP therefore says nothing about the multiplicity of the singular value 111 of D(P)D(P)D(P).

The real content is the equality case of a Cauchy–Schwarz inequality in CX×X\mathbb C^{X\times X}CX×X. It has to be translated into an edge-by-edge relation between the coordinates of the left and right singular vectors, then propagated along directed paths of the chain's graph. The self-loops are what identify the left and right singular vectors with each other. Bookkeeping that is routine on paper costs effort here: passing between the eigenvalue sequence of D(P)†D(P)D(P)^\dagger D(P)D(P)†D(P) and the dimension of its 111-eigenspace, and between paths in Mathlib's quiver of positive entries and chains of equalities.

Formalization scope

  • States and chain. The state space is a Fintype with decidable equality. PPP is a real matrix in Matrix.rowStochastic ℝ X, and irreducibility is Mathlib's Matrix.IsIrreducible (nonnegative entries and a strongly connected quiver with an edge x→yx\to yx→y iff pxy>0p_{xy}>0pxy​>0). For a row-stochastic matrix this agrees with "every state is reachable from every other state".
  • Stationary distribution. It enters as data π\piπ with the three properties above. Positivity is explicit because diag⁡(π)−1/2\operatorname{diag}(\pi)^{-1/2}diag(π)−1/2 needs it. Existence and uniqueness (Perron–Frobenius) are neither needed nor stated.
  • Discriminant. D(P)D(P)D(P) is the complex matrix with entries πxpxy/πy\sqrt{\pi_x}p_{xy}/\sqrt{\pi_y}πx​​pxy​/πy​​, acting on EuclideanSpace ℂ X. Its singular values are Mathlib's LinearMap.singularValues, a sequence indexed by N\mathbb NN that is 000 beyond ∣X∣|X|∣X∣. The goal counts the indices i<∣X∣i<|X|i<∣X∣ with σi=1\sigma_i=1σi​=1. The inner product u†D(P)wu^\dagger D(P)wu†D(P)w is Mathlib's inner ℂ u (D(P) w). Since the equality-case milestone assumes this equals 111 exactly, no phase ambiguity remains.
  • What is not assumed. Reversibility of PPP is not assumed. Neither is aperiodicity, primitivity, or a "lazy" bound pxx≥1/2p_{xx}\ge 1/2pxx​≥1/2: the hypothesis is exactly pxx>0p_{xx}>0pxx​>0 for every xxx.
  • Ruled-out trivializations. "111 is a singular value of D(P)D(P)D(P)" holds for every chain and is not the goal. The goal is that this singular value has multiplicity one. A conclusion such as "some singular value is <1<1<1" would also be too weak.
  • Cost. The paper's search algorithm and its cost bounds (Theorem 8) are out of scope. This mission is purely linear-algebraic.
  • Welcome contributions. Proofs of the milestones. The path-chaining milestone is a reusable fact about functions that are multiplicative along the edges of a strongly connected quiver. A general lemma relating the multiplicity of a singular value to the dimension of an eigenspace of T†TT^\dagger TT†T would be useful well beyond this mission.

Selected references

  • F. Magniez, A. Nayak, J. Roland, M. Santha, Search via Quantum Walk, SIAM J. Comput. 40(1):142–164, 2011; arXiv:quant-ph/0608026v4. https://arxiv.org/abs/quant-ph/0608026 (DOI 10.1137/090745854)
  • M. Szegedy, Quantum speed-up of Markov chain based algorithms, Proc. 45th IEEE FOCS, 32–41, 2004. https://doi.org/10.1109/FOCS.2004.53
  • A. Ambainis, Quantum walk algorithm for element distinctness, SIAM J. Comput. 37(1):210–239, 2007; arXiv:quant-ph/0311001. https://arxiv.org/abs/quant-ph/0311001
  • J. A. Fill, Eigenvalue bounds on convergence to stationarity for nonreversible Markov chains, with an application to the exclusion process, Ann. Appl. Probab. 1(1):62–87, 1991. https://doi.org/10.1214/aoap/1177005981
7 thms1 active userReviewed
Operations ResearchProbabilityStochastic Systems·Captain: mikedeng1

Validity of Heavy Traffic Steady-State Approximations in Generalized Jackson Networks: Scaled Stationary Queue Lengths Converge to the Stationary Distribution of the Reflected Brownian MotionResearch Paper

Motivation

Open networks of single-server queues with general (non-exponential) interarrival and service times, generalized Jackson networks (GJNs), model manufacturing lines, communication networks and service systems. Their stationary distributions are almost never available in closed form. The standard engineering approximation replaces the scaled queue-length vector by the stationary distribution of a reflected Brownian motion (RBM) in the orthant, which is the diffusion limit of the network in heavy traffic, when every station is close to full utilisation.

The diffusion limit itself (Reiman, 1984) is a statement about the process on finite time intervals. Using the RBM's stationary law as an approximation of the network's stationary law requires interchanging two limits: time to infinity (steady state) and traffic intensity to one (heavy traffic). For a long time this interchange was assumed rather than proved.

Timeline.

  • 1984. Reiman proved the process-level heavy-traffic limit for open queueing networks started empty (doi:10.1287/moor.9.3.441).
  • 1987. Harrison and Williams characterised when an orthant RBM has a stationary distribution and showed it is unique (doi:10.1080/17442508708833469).
  • 1995. Dai related fluid-model stability to positive Harris recurrence of multiclass networks (doi:10.1214/aoap/1177004828).
  • 2006. Gamarnik and Zeevi proved the interchange of limits for GJNs whose primitives have exponential moments (arXiv:math/0410066). This mission formalizes that result.

Setting

There are JJJ stations. Station jjj receives external arrivals with i.i.d. interarrival times of law FA,jF_{A,j}FA,j​ (rate αj\alpha_jαj​, or no arrivals at all) and serves jobs first-in-first-out with i.i.d. service times of law FS,jF_{S,j}FS,j​ (mean mjm_jmj​, rate μj=1/mj\mu_j = 1/m_jμj​=1/mj​). A job finishing at jjj moves to kkk with probability pjkp_{jk}pjk​ or leaves. The routing matrix PPP is substochastic with spectral radius below one. Interarrival and service times have uniformly bounded conditional exponential moments of their residual lives (conditions (1)–(2)). The traffic equation λ=α+P′λ\lambda = \alpha + P'\lambdaλ=α+P′λ gives the effective rates λ=[I−P′]−1α\lambda = [I-P']^{-1}\alphaλ=[I−P′]−1α and the traffic intensities ρj=λjmj\rho_j = \lambda_j m_jρj​=λj​mj​.

The queue lengths Q(t)Q(t)Q(t) are not Markov. The state Qˉ(t)=(Q(t),a^(t),v^(t))\bar Q(t) = (Q(t),\hat a(t),\hat v(t))Qˉ​(t)=(Q(t),a^(t),v^(t)), which adds the elapsed interarrival and service times, is a Markov process on X=Z+J×R+2J\mathcal X = \mathbb Z_+^J\times\mathbb R_+^{2J}X=Z+J​×R+2J​. A law π\piπ on X\mathcal XX is stationary if Qˉ(0)∼π\bar Q(0)\sim\piQˉ​(0)∼π implies Qˉ(t)∼π\bar Q(t)\sim\piQˉ​(t)∼π for all t≥0t\ge0t≥0.

Heavy traffic. Fix a critically loaded network Ξ\XiΞ (ρj=1\rho_j=1ρj​=1 for all jjj) and a vector κ0>0\kappa^0>0κ0>0. The network Ξn\Xi^nΞn slows the arrivals of Ξ\XiΞ at station jjj by the factor 1−κj0/n1-\kappa^0_j/\sqrt n1−κj0​/n​, so that ρjn=1−κj/n<1\rho^n_j = 1-\kappa_j/\sqrt n<1ρjn​=1−κj​/n​<1 for an explicit κ>0\kappa>0κ>0 (display (17)). Let πn\pi^nπn be any stationary distribution of Ξn\Xi^nΞn, and let π^n\hat\pi^nπ^n be the law of Qn(0)/nQ^n(0)/\sqrt nQn(0)/n​ under πn\pi^nπn.

The RBM. For a Brownian motion WWW with drift β\betaβ and covariance Γ\GammaΓ, the RBM ZZZ with parameters (β,Γ,I−P′)(\beta,\Gamma,I-P')(β,Γ,I−P′) solves the Skorohod problem Z=W+[I−P′]Y≥0Z = W + [I-P']Y\ge0Z=W+[I−P′]Y≥0, with YYY nondecreasing and increasing only when ZZZ is on the boundary. Here β=−(I−P′)M−1κ\beta = -(I-P')M^{-1}\kappaβ=−(I−P′)M−1κ and Γ\GammaΓ is the explicit covariance matrix of Reiman's theorem, built from μ\muμ, α\alphaα, PPP and the squared coefficients of variation ca,j2c^2_{a,j}ca,j2​, cs,j2c^2_{s,j}cs,j2​ of Ξ\XiΞ.

Formalization targets

Goal: Theorem 8 (p. 18)

π^n ⇒ πRBM(n→∞),\hat\pi^n\ \Rightarrow\ \pi_{\mathrm{RBM}}\qquad(n\to\infty),π^n ⇒ πRBM​(n→∞),

where πRBM\pi_{\mathrm{RBM}}πRBM​ is the unique stationary distribution of the (β,Γ,I−P′)(\beta,\Gamma,I-P')(β,Γ,I−P′)-RBM. The formal goal asserts three things: a stationary distribution of the RBM exists, it is the only one, and π^n\hat\pi^nπ^n converges weakly to it, for every choice of stationary distributions πn\pi^nπn.

Milestones

  1. Theorems 5 and 6 (pp. 15–16). For a general Markov process, a geometric Lyapunov function Φ\PhiΦ gives EπΦ≤ϕ(t0)K/(1−γ)\mathbb E_\pi\Phi\le\phi(t_0)K/(1-\gamma)Eπ​Φ≤ϕ(t0​)K/(1−γ). A Lyapunov function with control of L2L_2L2​ gives an exponential tail Pπ(Φ>s)≲e−θs\mathbb P_\pi(\Phi>s)\lesssim e^{-\theta s}Pπ​(Φ>s)≲e−θs.
  2. Proposition 1 (p. 11). The fluid model drains in time at most w′z/min⁡jμj(1−ρj)w'z/\min_j\mu_j(1-\rho_j)w′z/minj​μj​(1−ρj​), where w=e′[I−P′]−1w=e'[I-P']^{-1}w=e′[I−P′]−1.
  3. Lemma A.1 and Propositions 2–3 (pp. 17, 25). The net input deviates from its fluid path by O(n)O(\sqrt n)O(n​) uniformly over initial states. As a result, Φ(z,a,v)=w′z\Phi(z,a,v)=w'zΦ(z,a,v)=w′z is a Lyapunov function for Ξn\Xi^nΞn with drift −n-\sqrt n−n​ over time nt0nt_0nt0​.
  4. Theorem 7 and Corollary 1 (pp. 17–18). Pπn(n−1/2w′Qn(0)>s)≤C1e−c1s\mathbb P_{\pi^n}(n^{-1/2}w'Q^n(0)>s)\le C_1e^{-c_1s}Pπn​(n−1/2w′Qn(0)>s)≤C1​e−c1​s uniformly in nnn. Hence {π^n}\{\hat\pi^n\}{π^n} is tight.
  5. Theorems 4, 3 and 2 (cited). Reiman's process limit from a general initial law; Harrison and Williams's existence and uniqueness theorem for the RBM's stationary distribution; existence of πn\pi^nπn.

Significance

The result. Theorem 8 justifies the use of the RBM's stationary distribution, which is computable or at least numerically tractable, as an approximation of steady-state queue lengths of a heavily loaded network. Theorem 7 also shows that each stationary queue is of order (1−ρ∗n)−1(1-\rho^{*n})^{-1}(1−ρ∗n)−1 uniformly in nnn. Together with Theorem 8 this gives convergence of all moments (Corollary 2 of the paper) and, through Theorems 9–12, steady-state approximations of sojourn times and product-form limits.

Formalizing it. No part of this argument is machine-checked. The formalization would add reusable infrastructure for queueing theory in Lean: a Markov-state model of a generalized Jackson network with residual times, Lyapunov bounds on stationary distributions of general Markov processes (Theorems 5–6), and the link between tightness and identification of limit points for stationary laws. Reiman's theorem (Theorem 4) and the Harrison–Williams theorem (Theorem 3) are cited by the paper and are themselves open formalization targets.

Difficulty

Tightness and Reiman's theorem do not combine on their own. Reiman's theorem describes the network on finite time intervals, started empty, while a stationary distribution describes it as time goes to infinity; nothing in the finite-horizon limit forces a limit point of π^n\hat\pi^nπ^n to be stationary for the RBM, or forces different subsequences to have the same limit. The interchange therefore needs the process limit from an arbitrary initial law and the uniqueness of the RBM's stationary law, besides tightness.

The tightness step is the technical core. Moment bounds must be uniform in nnn and in the initial residual times. A Lyapunov argument on the workload w′Qw'Qw′Q over a time horizon of order nnn requires deviation bounds of order n\sqrt nn​ for renewal processes started at arbitrary ages. This is why the residual-life conditions (1)–(2) appear, and why the strong approximation of Lemma A.2 is used. A naive drift argument over a fixed time horizon fails: in heavy traffic the drift of w′Qw'Qw′Q per unit time is only of order n−1/2n^{-1/2}n−1/2.

Formalization scope

  • Stations are Fin J, vectors Fin J → ℝ, P′P'P′ is Pᵀ, and the norm ∥⋅∥\|\cdot\|∥⋅∥ is the ℓ1\ell^1ℓ1 norm, written as an explicit sum. The spectral-radius condition is Pm→0P^m\to0Pm→0.
  • A network is its data (J,FA,FS,P)(\mathcal J, F_A, F_S, P)(J,FA​,FS​,P) plus the predicate IsGJN (positive times, conditions (1)–(2), substochastic PPP). AjA_jAj​ and SjS_jSj​ count renewal epochs in [0,t][0,t][0,t]. The initial times aj(0)a_j(0)aj​(0), vj(0)v_j(0)vj​(0) are residual lives given the elapsed ages a^j(0)\hat a_j(0)a^j​(0), v^j(0)\hat v_j(0)v^j​(0). States carry their elapsed times as reals; the predicate InStateSpace cuts out X=Z+J×R+2J\mathcal X=\mathbb Z_+^J\times\mathbb R_+^{2J}X=Z+J​×R+2J​, and the uniform bounds over initial states (Lemma A.1, Propositions 2–3) and the initial laws of Theorem 4 range over X\mathcal XX only.
  • A realization from an initial law is a probability space carrying the primitives with their joint law and processes Q,BQ, BQ,B satisfying the dynamics (4)–(5) almost surely. E[⋅∣Qˉ(0)=x]\mathbb E[\cdot\mid\bar Q(0)=x]E[⋅∣Qˉ​(0)=x] is the expectation under a realization from δx\delta_xδx​. Stationarity of π\piπ means: a realization from π\piπ exists, and every realization from π\piπ has Qˉ(t)∼π\bar Q(t)\sim\piQˉ​(t)∼π for all t≥0t\ge0t≥0.
  • The heavy-traffic scaling is read as ajn=aj/(1−κj0/n)a^n_j = a_j/(1-\kappa^0_j/\sqrt n)ajn​=aj​/(1−κj0​/n​). The printed aj(1−κj0/n)a_j(1-\kappa^0_j/\sqrt n)aj​(1−κj0​/n​) would overload every Ξn\Xi^nΞn, so no πn\pi^nπn would exist. κ\kappaκ is defined so that (17) holds exactly, and every statement about Ξn\Xi^nΞn is for all large nnn.
  • The RBM is built on the published Reiman84.QueueLength.Paths (Brownian motion with drift and covariance, and the reflection pair). An RBM stationary law must be a probability measure carried by R+J\mathbb R^J_+R+J​. Weak convergence of laws on RJ\mathbb R^JRJ uses Mathlib's topology on ProbabilityMeasure. Theorem 4's process convergence is in coupling form with the uniform topology on [0,T][0,T][0,T].
  • Quantities lim sup⁡n(⋅)<∞\limsup_n(\cdot)<\inftylimsupn​(⋅)<∞ are rendered as one finite bound valid for all large nnn. Expectations of nonnegative quantities are lower Lebesgue integrals in [0,∞][0,\infty][0,∞]. Constants "depending only on Ξ\XiΞ" are quantified before nnn and before the stationary distributions.
  • Disclosed departures from the page:
    • ϕ(t0)<∞\phi(t_0)<\inftyϕ(t0​)<∞ is assumed in Theorem 5.
    • {Φ>K}≠∅\{\Phi>K\}\neq\emptyset{Φ>K}=∅ is assumed in Theorem 6. Its tail constant (29) is stated as (γθ/2)−1(\gamma\theta/2)^{-1}(γθ/2)−1, which is what the paper's proof yields, instead of the printed (1−γθ/2)−1(1-\gamma\theta/2)^{-1}(1−γθ/2)−1.
    • Γ\GammaΓ is positive definite in Theorem 3, the hypothesis of Harrison and Williams.
    • The derivative bound w′q˙(t)≤−min⁡jμj(1−ρj)w'\dot q(t)\le-\min_j\mu_j(1-\rho_j)w′q˙​(t)≤−minj​μj​(1−ρj​) of Proposition 1 is stated for ρj≤1\rho_j\le1ρj​≤1 at every station; the page states it unconditionally, and it fails when two stations are overloaded.
    • Theorems 5 and 6 are stated for the time-t0t_0t0​ transition kernel of the Markov process.
  • A trivializing formalization is ruled out explicitly. The goal keeps the existence and uniqueness of πRBM\pi_{\mathrm{RBM}}πRBM​ as conjuncts, stationarity requires a realization to exist, and the RBM's stationary law must live on the orthant. Hence neither an impossible network nor an empty RBM notion makes the goal vacuous.
  • Contributions are welcome on any milestone. The cited Theorems 2–4 are substantial on their own, and Theorems 5–6 are independent of queueing.

Selected references

  • D. Gamarnik and A. Zeevi, Validity of heavy traffic steady-state approximations in generalized Jackson networks, Ann. Appl. Probab. 16(1), 2006, 56–90. arXiv:math/0410066, doi:10.1214/105051605000000638
  • M. I. Reiman, Open queueing networks in heavy traffic, Math. Oper. Res. 9(3), 1984, 441–458. doi:10.1287/moor.9.3.441
  • J. M. Harrison and R. J. Williams, Brownian models of open queueing networks with homogeneous customer populations, Stochastics 22, 1987, 77–115. doi:10.1080/17442508708833469
  • J. G. Dai, On positive Harris recurrence of multiclass queueing networks: a unified approach via fluid limit models, Ann. Appl. Probab. 5(1), 1995, 49–77. doi:10.1214/aoap/1177004828
  • H. Chen and D. D. Yao, Fundamentals of Queueing Networks: Performance, Asymptotics, and Optimization, Springer, 2001 (reference [11] of the paper; Chapter 7).
18 thms1 active userReviewed
ProbabilityRepresentation Theory·Captain: marwahaha

Random coordinate frames and partial permutation lawsOpen Problem

Motivation

Random coordinate frames allow comparison between different sequences of matching directions in a shuffle. The source uses partially observed card lists to obtain estimates for the full permutation law. The source is OpenAI's September 2026 manuscript.

The companion Conditional information under deterministic coordinate sweeps develops the conditional-information side of the same published eight-clause goal. Both manuscripts are represented by this one draft because they point to the identical formal statement, OAI.ThorpResults.remaining_main.

Setting

A Thorp shuffle acts on 2d2^d2d binary-labelled positions by independent pair switches followed by a cyclic coordinate rotation. A sweep consists of ddd physical shuffles. Partial observations record an injective ordered list of card positions.

Formalization targets

sup⁡σ∥qd∗32800dσ−U∥TV→0,tmix(d)=Θ(log⁡2d).\sup_\sigma\|q_d^{*32800d}\sigma-U\|_{\mathrm{TV}}\to0,\qquad t_{\mathrm{mix}}(d)=\Theta(\log 2^d).σsup​∥qd∗32800d​σ−U∥TV​→0,tmix​(d)=Θ(log2d).

Eight asymptotic results about the Thorp shuffle of 2^d cards, whose positions are the d-bit strings, hold simultaneously. One Thorp step on d≥1 bits uses one fair coin for each (d−1)-bit suffix: the first bit of a position is flipped according to the coin of its remaining bits, and then the bits are cyclically rotated; the d=0 step is the identity. A shuffle of t steps composes t independent such steps, and 'distance' is total variation (half the L¹ distance) from the uniform law on permutations. (1) FrameMain: for every ε>0, for all large d and every starting permutation, the law after 32800·d steps is within total variation ε of uniform. (2) InformationMain has three parts. For large d, any k with 8k≤7·2^d, any start and any list of k distinct labelled positions, the law of the images of the labels after 256·d steps is within ε of uniform on injective k-lists. For large d and 16k≤15·2^d, the same list distance after 1024·d steps is at most (2^d)^(−3/2). Finally, distance(d, 2048·d) tends to 0, and for every d the distance from any start equals distance(d, 2048·d). (3) SpectrumMain: there is p>0 such that for all d≥1 the regularTrace at exponent 2p is at most 17/16 (a sum over Young-diagram shapes of size 2^d of dimension times the trace of |Q|^{2p}, where Q is the averaged Specht-module operator of d-step shuffles) and, for every integer M≥p and every σ, the sweep distance after M·d steps (total variation of the shifted law from uniform) is at most 1/8. (4) SignedMain: for all real η,κ with η>0 there is r≥1 such that, for all d≥1, every shape μ of size 2^d and all Young diagrams α,β,γ with |α|+|β|+|γ|=2^d, the logarithm of the weighted moment dim·Re tr((QQ)^r) is at most a(η,d)·signedEntropy(α,β)+remainderBudget(κ,d,|γ|), where a(η,d)=η(1−1/(2√d)), signedEntropy sums log((|α|+|β|)/rowlength) over cells of α and β, and the remainder is 0 if |γ|=0 and otherwise max(0, a(κ,d)·|γ|·log 2^d−|γ|·log(2^d/|γ|)). (5) DegreeSavingMain: there are an even r≥2 and η>0 such that for all d and shapes μ, dim·Re tr((QQ)^r) ≤ dim^(1−η). (6) FullDensityMain: there is v≥1 such that, for every ε>0 and large d, every shifted law after v·d steps has normalized squared L² density distance (2^d)!·Σ_g(law−1/(2^d)!)² below ε. (7) ForwardMixingMain: there is v≥1 such that for large d every shifted law after v·d steps has total variation below ε. (8) OptimalOrderMain: the mixing time, the least t with distance at most 1/4, is Θ(log 2^d).

The goal is OAI.ThorpResults.remaining_main.

Significance

The selected published goal packages eight results, including fixed-list estimates, spectral moments, signed moments, degree savings, full-density convergence, and optimal-order mixing. The detailed clauses below specify the complete bundle. The manuscript presents a mathematical argument; the attached goal currently has status Open, so the requested contribution is a Lean proof of the displayed formal target.

Difficulty

Changing coordinates must preserve the joint path law, including what has already been revealed. Marginal uniformity of individual cards does not control the full permutation.

Formalization scope

The goal is the shared remaining_main bundle used by several companion manuscripts. It retains all eight formal clauses rather than claiming a new manuscript-specific statement. The displayed bounds are two of those clauses; the full conjunction remains the proof obligation.

The shared definitions are supplied by ThorpRemaining. The target uses Lean 4.33.1 with the pinned Mathlib environment. Contributions should preserve the existing statement and develop reusable supporting lemmas in its definitions.

Selected references

  • OpenAI, Random coordinate frames and partial permutation laws, preprint, September 2026. Manuscript.
  • OpenAI, accompanying Lean statement. Pinned source.
  • OpenAI, Conditional information under deterministic coordinate sweeps, preprint, September 2026. Companion manuscript.
2 thms1 active userReviewed
Probability·Captain: marwahaha

Cutoff throughout the high-temperature Sherrington–Kirkpatrick phaseOpen Problem

Motivation

Cutoff means that a Markov chain’s transition from far from equilibrium to near equilibrium occurs in a negligible fraction of its mixing time. The source studies heat-bath dynamics for a disordered mean-field Ising model. The source is OpenAI's September 2026 manuscript.

Setting

The Sherrington–Kirkpatrick model has independent centered Gaussian pair couplings with variance β2/n\beta^2/nβ2/n and zero external field. One discrete update picks a site uniformly and resamples its spin from its conditional Gibbs law. Mixing time uses worst-case total variation distance.

Formalization targets

Pg ⁣(tmix(g,ε)tmix(g,1−ε)>1+η)→0(0≤β<1/2, 0<ε<1/2, η>0).\mathbb P_g\!\left(\frac{t_{\mathrm{mix}}(g,\varepsilon)}{t_{\mathrm{mix}}(g,1-\varepsilon)}>1+\eta\right)\to0\qquad(0\le\beta<1/2,\ 0<\varepsilon<1/2,\ \eta>0).Pg​(tmix​(g,1−ε)tmix​(g,ε)​>1+η)→0(0≤β<1/2, 0<ε<1/2, η>0).

For real parameters β with 0 ≤ β < 1/2, ε with 0 < ε < 1/2, and η > 0, two things hold for the Sherrington–Kirkpatrick model with n spins, where a disorder g is a real coupling for each pair i < j of sites and the Gibbs measure at zero external field has weight proportional to exp((1/2) Σᵢ Σⱼ σᵢ gᵢⱼ σⱼ) on spin configurations σ ∈ {±1}ⁿ (couplings symmetrized, diagonal zero). The dynamics is a heat-bath single-site chain: choose a site uniformly at random, then keep or flip its spin with probabilities proportional to the Gibbs masses of the two configurations (for n = 0 the transition matrix is the identity). The distance after k steps is the maximum over starting configurations of the total variation distance (half the ℓ¹ distance) between the k-step distribution and the Gibbs measure, and mixingTime(g, δ) is the least k whose distance is at most δ (the infimum of the set, so 0 if that set is empty). The disorder law makes the couplings independent centered Gaussians with variance β²/n. First, the probability under this law that the ratio mixingTime(g, ε) / mixingTime(g, 1 − ε) exceeds 1 + η tends to 0 as n → ∞. Second, for all sufficiently large n, every disorder g satisfies mixingTime(g, 1 − ε) > 0.

The goal is OAI.SKRatio.ratio_cutoff.

Significance

The available target gives ratio cutoff in probability over the disorder and eventual positivity of the denominator for every disorder. It isolates a precise mixing-time comparison. The manuscript presents a mathematical argument; the attached goal currently has status Open, so the requested contribution is a Lean proof of the displayed formal target.

Difficulty

Fast relaxation estimates alone do not prove that all fixed total-variation thresholds become asymptotically equivalent. The worst starting configuration and the random couplings both enter the comparison.

Formalization scope

The attached range is 0≤β<1/20\le\beta<1/20≤β<1/2, narrower than the full high-temperature range discussed in the manuscript. The target does not specify the cutoff location. Dynamics is discrete single-site updating; the exceptional n=0n=0n=0 transition is the identity.

The shared definitions are supplied by SKRatio. The target uses Lean 4.33.1 with the pinned Mathlib environment. Contributions should preserve the existing statement and develop reusable supporting lemmas in its definitions.

Selected references

  • OpenAI, Cutoff throughout the high-temperature Sherrington–Kirkpatrick phase, preprint, September 2026. Manuscript.
  • OpenAI, accompanying Lean statement. Pinned source.
2 thms1 active userReviewed
Graph Theory·Captain: marwahaha

Polynomial mixing of the switch chain for every graphical degree sequenceOpen Problem

Motivation

Sampling a graph with a prescribed degree vector requires moving among realizations without changing degrees. The switch chain performs local edge replacements and asks whether these suffice for rapid global mixing. The source manuscript presents the surrounding research claim.

Setting

States are simple undirected labeled graphs realizing a graphical vector d on n≥4 vertices. The lazy switch chain stays put with probability one half and otherwise proposes one ordered replacement between distinct perfect matchings on four vertices, rejecting invalid proposals.

Formalization target

Prove

τTV(1/4)≤2n8,\tau_{TV}(1/4)\le2n^8,τTV​(1/4)≤2n8,

with mixing time zero for a singleton state space. When more than one state exists, prove the spectral-gap lower bound 1/(24n²·binom(n,4)).

The selected formal target is OAI.Problem315.switch_chain_main.

Significance and status

The goal covers every graphical degree sequence under the stated degree bound, without regularity or density restrictions. Connectivity and explicit exponential total-variation decay are available supporting targets. The published target is currently Open; the manuscript's argument and a checked Lean proof are separate deliverables.

Difficulty

Connectivity gives eventual communication but no polynomial rate. The estimate must be uniform across degree vectors and initial graphs.

Formalization scope

The kernel is defined by exact counts of four-vertex proposals, and total variation is the finite half-ℓ₁ distance from uniform measure. The main theorem contains the singleton case explicitly. The manuscript’s exact sampler is not an additional published target here.

Selected references

  • OpenAI, Polynomial mixing of the switch chain for every graphical degree sequence, preprint, 2026. Pinned manuscript.
  • OpenAI, accompanying formal statements, commit adc7f1241b42. Goal source.
4 thms1 active userReviewed
ProbabilityRepresentation Theory·Captain: wurtle

Conditional coordinate sweeps and analytic transferResearch Paper

Motivation: a dimension saving for the binary coordinate sweep

The Thorp shuffle cuts a deck in half, pairs cards in corresponding positions, and lets an independent fair coin order each pair before the pairs are interleaved. On N=2dN=2^dN=2d cards, removing the deterministic rotation of coordinates turns ddd physical shuffles into one binary coordinate sweep: in each coordinate direction, in order, every pair of positions differing in that coordinate is independently swapped with probability 1/21/21/2. After one sweep each card is uniform, but the joint permutation retains dependence because cards share switches.

By finite-group Fourier analysis (Diaconis–Shahshahani 1981), fast mixing of the whole deck follows from a power saving ∥Kλ∥op≤Dλ−g\|K_\lambda\|_{\rm op}\le D_\lambda^{-g}∥Kλ​∥op​≤Dλ−g​ for the sweep in every irreducible representation λ\lambdaλ of SNS_NSN​. The source obtains this saving by first controlling a perturbed sweep, in which each line permutation is mostly uniform, conditionally on prescribed trajectories of some cards, and then transferring the estimate analytically to the binary sweep.

Timeline

  • 1973 — Thorp introduces the shuffle in a study of nonrandom shuffling in Faro (Thorp, JASA 1973).
  • 1981 — Diaconis and Shahshahani introduce the Fourier upper-bound method for random walks on SnS_nSn​ (Z. Wahrsch. 1981).
  • 2005/2008 — Morris proves O(d44)O(d^{44})O(d44) physical shuffles for N=2dN=2^dN=2d with a chameleon process that conditions on the trajectory of an occupied set (SIAM J. Comput. 2008); Montenegro and Tetali sharpen it to O(d29)O(d^{29})O(d29) (2006).
  • 2009 — Morris proves O((log⁡N)4)O((\log N)^4)O((logN)4) for every even NNN (Ann. Probab. 2009); Morris, Rogaway and Stegers analyze selected-card Thorp marginals for small-domain encryption (CRYPTO 2009).
  • 2012 — Hoang, Morris and Rogaway use a conditional squared-discrepancy calculation for the swap-or-not shuffle (CRYPTO 2012).
  • 2013 — Morris proves O(d3)O(d^3)O(d3) for N=2dN=2^dN=2d, exposing complete labeled trajectories successively (CPC 2013).
  • 2014–2015 — Czumaj and Vöcking: nearly uniform joint positions of a (1−ε)(1-\varepsilon)(1−ε) fraction of the cards after O(d2)O(d^2)O(d2) shuffles (ICALP 2014); Czumaj, switching networks (STOC 2015).
  • 2026 — The OpenAI preprint Conditional coordinate sweeps and analytic transfer (OpenAI Math Release, September 26, 2026) proves the binary sweep contraction below, giving tmix(d)=Θ(d)t_{\rm mix}(d)=\Theta(d)tmix​(d)=Θ(d) physical shuffles. The preprint has not been peer reviewed and its results are not formally verified.

Setting

Positions are binary strings {0,1}d\{0,1\}^d{0,1}d, N=2dN=2^dN=2d (Lean: Slot d := Fin d → Bool). For a coordinate jjj, a layer flips bit jjj of a position xxx according to a fair coin attached to the edge {x,x+ej}\{x,x+e_j\}{x,x+ej​}; the binary sweep applies the layers for j=1,…,dj=1,\dots,dj=1,…,d in order, with all coins independent (Lean: binarySweep, binaryLaw d its law on SNS_NSN​). For an irreducible unitary representation ρλ\rho_\lambdaρλ​ of SNS_NSN​ of dimension DλD_\lambdaDλ​, the sweep operator is

Kλ=Eg∼BNρλ(g).K_\lambda=\mathbb E_{g\sim B_N}\rho_\lambda(g).Kλ​=Eg∼BN​​ρλ​(g).

Write BN∗wB_N^{*w}BN∗w​ for the law of www independent sweeps and ∥μ−ν∥TV=12∑g∣μ(g)−ν(g)∣\|\mu-\nu\|_{\rm TV}=\tfrac12\sum_g|\mu(g)-\nu(g)|∥μ−ν∥TV​=21​∑g​∣μ(g)−ν(g)∣.

For the conditional estimate, an ordered product grid is Ω=Ω1×⋯×Ωb\Omega=\Omega_1\times\dots\times\Omega_bΩ=Ω1​×⋯×Ωb​ with each ∣Ωj∣=mj|\Omega_j|=m_j∣Ωj​∣=mj​ a power of two in [R,R2][R,R^2][R,R2]; a sweep permutes the lines parallel to coordinate 111, then coordinate 222, and so on, each line independently with law Lm(z)=(1−z)Um+zBmL_m(z)=(1-z)U_m+zB_mLm​(z)=(1−z)Um​+zBm​. A family H\mathcal HH of hhh prescribed trajectories (disjoint at every layer) is conditioned on; the remaining random bijection of the s−hs-hs−h free slots has average KλHK^{\mathcal H}_\lambdaKλH​ in each irreducible λ⊢s−h\lambda\vdash s-hλ⊢s−h, with line cost C(H)=∑Llog⁡(∣L∣hL/(∣L∣)hL)C(\mathcal H)=\sum_{L}\log\bigl(|L|^{h_L}/(|L|)_{h_L}\bigr)C(H)=∑L​log(∣L∣hL​/(∣L∣)hL​​).

Formalization targets

Milestone: conditional sweep moment estimate (Theorem 3.1)

There are a power of two R≥2R\ge2R≥2, an integer q≥1q\ge1q≥1 and z∗>0z_*>0z∗​>0 such that for every grid, every feasible family H\mathcal HH, every z∈[0,z∗]z\in[0,z_*]z∈[0,z∗​] and every irreducible λ⊢s−h\lambda\vdash s-hλ⊢s−h,

log⁡∥KλH∥2q2q≤−c(s)log⁡Dλ+e(s) hlog⁡s−C(H),\log\|K^{\mathcal H}_\lambda\|_{2q}^{2q}\le -c(s)\log D_\lambda+e(s)\,h\log s-C(\mathcal H),log∥KλH​∥2q2q​≤−c(s)logDλ​+e(s)hlogs−C(H),

with c(s)=c0+1/log⁡sc(s)=c_0+1/\sqrt{\log s}c(s)=c0​+1/logs​, e(s)=e0−1/log⁡se(s)=e_0-1/\sqrt{\log s}e(s)=e0​−1/logs​, c0=e0=10−4c_0=e_0=10^{-4}c0​=e0​=10−4, and log⁡0=−∞\log 0=-\inftylog0=−∞.

Goal: binary sweep contraction (Theorem 1.1)

There are absolute constants g>0g>0g>0 and d0d_0d0​ such that for all d≥d0d\ge d_0d≥d0​ and every irreducible unitary representation of S2dS_{2^d}S2d​ of dimension DDD,

∥Kλ∥op≤D−g;\|K_\lambda\|_{\rm op}\le D^{-g};∥Kλ​∥op​≤D−g;

the sign representation is annihilated, ∑gBN(g) sgn(g)=0\sum_g B_N(g)\,\mathrm{sgn}(g)=0∑g​BN​(g)sgn(g)=0; and there is an absolute integer www such that

sup⁡τ∈SN∥BN∗w(⋅ τ−1)−UN∥TV→d→∞0.\sup_{\tau\in S_N}\bigl\|B_N^{*w}(\cdot\,\tau^{-1})-U_N\bigr\|_{\rm TV}\xrightarrow[d\to\infty]{}0 .τ∈SN​sup​​BN∗w​(⋅τ−1)−UN​​TV​d→∞​0.

The goal is published on the platform with status Open: no machine-checked proof exists yet.

Significance

The result itself. Since one binary sweep is exactly ddd physical shuffles, the goal gives O(d)O(d)O(d) physical shuffles for mixing from every starting deck, and the support obstruction (Proposition 1.2: ttt shuffles reach at most 2tN/22^{tN/2}2tN/2 permutations) gives the matching lower bound 2d−O(1)2d-O(1)2d−O(1); thus tmix(d)=Θ(d)t_{\rm mix}(d)=\Theta(d)tmix​(d)=Θ(d), improving Morris 2013. The power saving in every irreducible representation is a stronger, reusable statement: it controls any number of further sweeps and any function of the permutation through Fourier analysis.

Formalizing it. The goal is stated for explicit finite objects (coins, sweeps, Fourier operators on CD\mathbb C^DCD), with existential constants only. A formal proof would certify optimal-order mixing for the Thorp sweep. The conditional estimate (Theorem 3.1) is formalized separately and is the natural intermediate milestone.

Difficulty

The obvious induction—split the coordinates and multiply child bounds—breaks because revealing card paths changes the conditional law of the line permutations, and the induction must be stable when more paths are revealed. The source handles this with the line cost C(H)C(\mathcal H)C(H), which records the exact cost of prescribing positions without replacement. A second obstacle is that the conditional estimate is only available for mostly-uniform lines (zzz small), while the Thorp sweep is the binary case z=1z=1z=1; for a fixed grid the sweep operator is a matrix polynomial in zzz whose behavior at z=1z=1z=1 is not controlled by real-segment bounds alone. Representations of very small "level" (first appearing on few tracked cards) need a separate sparse estimate.

Formalization scope

  • Goal (binary_sweep_contraction_and_mixing): three conjuncts. BinaryContractionTarget: ∃g>0 ∃d0 ∀d≥d0 ∀D\exists g>0\ \exists d_0\ \forall d\ge d_0\ \forall D∃g>0 ∃d0​ ∀d≥d0​ ∀D and every irreducible representation ρ\rhoρ of Equiv.Perm (Slot d) on EuclideanSpace ℂ (Fin D) preserving norms, the operator norm of ∑gbinaryLaw(g)ρ(g)\sum_g \mathrm{binaryLaw}(g)\rho(g)∑g​binaryLaw(g)ρ(g) is at most D−gD^{-g}D−g (real power). Sign: for every d>0d>0d>0, ∑gbinaryLaw(g) sgn(g)=0\sum_g \mathrm{binaryLaw}(g)\,\mathrm{sgn}(g)=0∑g​binaryLaw(g)sgn(g)=0. UniformSweepMixingTarget: ∃w ∀ε>0 ∃d0 ∀d≥d0 ∀τ\exists w\ \forall\varepsilon>0\ \exists d_0\ \forall d\ge d_0\ \forall\tau∃w ∀ε>0 ∃d0​ ∀d≥d0​ ∀τ, total variation of the www-fold convolution power translated by τ\tauτ from uniform is at most ε\varepsilonε. Physical shuffles are not modelled; the statement is about sweeps.
  • Milestone (conditional_main): grids are Grid with b≥1b\ge1b≥1 coordinates of r≤bitsj≤2rr\le\text{bits}_j\le2rr≤bitsj​≤2r (R=2rR=2^rR=2r); trajectories are Holes with disjointness at every layer and coordinate-respecting steps; feasibility means some line choices realize them; the residual bijection is identified with the stabilizer of the input holes via a fixed reference completion; representations are UnitaryIrrep of that stabilizer; the Schatten moment is ReTr⁡((K∗K)q)\mathrm{Re}\operatorname{Tr}((K^*K)^q)ReTr((K∗K)q); z∗≤1/2z_*\le1/2z∗​≤1/2.
  • The mapped item all_grid_moment is a parameter-explicit form of the same induction under numerical largeness hypotheses and a small-perturbation hypothesis, kept as a plain item.
  • Infrastructure: Specht-module dimension estimates, branching and Pieri rules, signed Schur–Weyl decompositions, Schatten norms, and the subharmonic maximum principle; the analytic-transfer step is reusable for other families of averaging operators polynomial in a parameter.

Selected references

  • E. O. Thorp, Nonrandom shuffling with applications to the game of Faro, J. Amer. Statist. Assoc. 68 (1973), 842–847. https://doi.org/10.1080/01621459.1973.10481434
  • P. Diaconis and M. Shahshahani, Generating a random permutation with random transpositions, Z. Wahrsch. Verw. Gebiete 57 (1981), 159–179. https://doi.org/10.1007/BF00535487
  • B. Morris, The mixing time of the Thorp shuffle, SIAM J. Comput. 38 (2008), 484–504. https://doi.org/10.1137/050636231
  • R. Montenegro and P. Tetali, Mathematical aspects of mixing times in Markov chains, Found. Trends Theor. Comput. Sci. 1 (2006), 237–354. https://doi.org/10.1561/0400000003
  • B. Morris, Improved mixing time bounds for the Thorp shuffle and L-reversal chain, Ann. Probab. 37 (2009), 453–477. https://doi.org/10.1214/08-AOP409
  • B. Morris, Improved mixing time bounds for the Thorp shuffle, Combin. Probab. Comput. 22 (2013), 118–132. https://doi.org/10.1017/S0963548312000478
  • B. Morris, P. Rogaway and T. Stegers, How to encipher messages on a small domain, CRYPTO 2009, LNCS 5677, 286–302. https://doi.org/10.1007/978-3-642-03356-8_17
  • V. T. Hoang, B. Morris and P. Rogaway, An enciphering scheme based on a card shuffle, CRYPTO 2012, LNCS 7417, 1–13. https://doi.org/10.1007/978-3-642-32009-5_1
  • A. Czumaj and B. Vöcking, Thorp shuffling, butterflies, and non-Markovian couplings, ICALP 2014, LNCS 8572, 344–355. https://doi.org/10.1007/978-3-662-43948-7_29
  • A. Czumaj, Random permutations using switching networks, STOC 2015, 703–712. https://doi.org/10.1145/2746539.2746629
  • A. Berele and A. Regev, Hook Young diagrams, combinatorics and representations of Lie superalgebras, Bull. Amer. Math. Soc. (N.S.) 8 (1983), 337–339. https://doi.org/10.1090/S0273-0979-1983-15110-8
  • OpenAI, Conditional coordinate sweeps and analytic transfer, OpenAI Math Release preprint, September 26, 2026 (source; Theorem 1.1, p. 2; Theorem 3.1, p. 7). https://github.com/openai/math/blob/adc7f1241b42e322a6451854ab7e4b4c146bf78a/preprints/Conditional-coordinate-sweeps-and-analytic-transfer-September-26-2026/main.pdf
5 thms1 active userReviewed
ProbabilityRepresentation Theory·Captain: wurtle

Random-subspace tests and trace smoothing for coordinate sweepsResearch Paper

Motivation: optimal-order mixing for the Thorp shuffle

The Thorp shuffle is a simple card-shuffling rule: cut a deck of nnn cards into two equal halves, pair cards in corresponding positions, and let an independent fair coin decide the order within each pair when the pairs are interleaved. It is a standard test case for mixing-time methods and a building block in small-domain enciphering schemes. For n=2dn=2^dn=2d every individual card is uniform after ddd shuffles, but the joint order of all nnn cards is much harder to control. Determining whether O(d)=O(log⁡n)O(d)=O(\log n)O(d)=O(logn) shuffles suffice to make the full permutation close to uniform was a long-standing question.

Timeline

  • 1973 — Thorp introduces the model in a study of nonrandom shuffling in Faro (Thorp, JASA 1973).
  • 2005/2008 — Morris proves the first polynomial-in-ddd bound, O(d44)O(d^{44})O(d44) shuffles for n=2dn=2^dn=2d, using evolving sets and a chameleon process (Morris, SIAM J. Comput. 2008).
  • 2006 — Montenegro and Tetali sharpen this to O(d29)O(d^{29})O(d29) by spectral-profile and evolving-set estimates (Montenegro–Tetali 2006, §6.4).
  • 2009 — Morris introduces an entropy method based on card interactions and obtains O(log⁡4n)O(\log^4 n)O(log4n) for every even deck size (Morris, Ann. Probab. 2009).
  • 2013 — Morris obtains O(d3)O(d^3)O(d3) for dyadic decks (Morris, CPC 2013).
  • 2026 — The OpenAI preprint Random-subspace tests and trace smoothing for coordinate sweeps (OpenAI Math Release, September 26, 2026) claims a uniform bound on a fixed Schatten moment of one coordinate sweep in the full regular representation, which yields total-variation mixing after an absolute number of sweeps, i.e. O(d)O(d)O(d) physical shuffles. The preprint has not been peer reviewed and its result is not formally verified.

Setting

Index the n=2dn=2^dn=2d positions by binary strings x∈F2dx\in\mathbb F_2^dx∈F2d​ (in Lean, Card d := Fin d → Bool). One physical shuffle sends

(x1,x2,…,xd)⟼(x2,…,xd,  x1+ξx2,…,xd)(x_1,x_2,\dots,x_d)\longmapsto(x_2,\dots,x_d,\;x_1+\xi_{x_2,\dots,x_d})(x1​,x2​,…,xd​)⟼(x2​,…,xd​,x1​+ξx2​,…,xd​​)

with independent fair bits ξ\xiξ (Lean: step d ξ = rotate * pairSwitch ξ). After ddd physical shuffles the rotations cancel, and the result is a coordinate sweep: one random matching switch in each coordinate direction. Write μt\mu_tμt​ for the law on SnS_nSn​ of ttt physical shuffles (Lean: law d t, the fraction of coin sequences producing a given permutation).

For a partition λ⊢n\lambda\vdash nλ⊢n let VλV_\lambdaVλ​ be the Specht module (the irreducible representation of SnS_nSn​ indexed by λ\lambdaλ), of dimension DλD_\lambdaDλ​, and let Qn(λ)=∑gμd(g) λ(g)Q_n(\lambda)=\sum_g \mu_d(g)\,\lambda(g)Qn​(λ)=∑g​μd​(g)λ(g) be the average of the sweep in VλV_\lambdaVλ​. For an operator YYY, ∣Y∣=(Y∗Y)1/2|Y|=(Y^*Y)^{1/2}∣Y∣=(Y∗Y)1/2 and traces are unnormalized. The regular trace of the PPP-th power is

Tr⁡reg∣Qn∣P=∑λ⊢nDλTr⁡Vλ∣Qn(λ)∣P,\operatorname{Tr}_{\rm reg}|Q_n|^P=\sum_{\lambda\vdash n}D_\lambda\operatorname{Tr}_{V_\lambda}|Q_n(\lambda)|^P ,Trreg​∣Qn​∣P=λ⊢n∑​Dλ​TrVλ​​∣Qn​(λ)∣P,

in which the trivial representation contributes exactly 111. Total variation is ∥μ−ν∥TV=12∑g∣μ(g)−ν(g)∣\|\mu-\nu\|_{\rm TV}=\tfrac12\sum_g|\mu(g)-\nu(g)|∥μ−ν∥TV​=21​∑g​∣μ(g)−ν(g)∣.

Formalization targets

Goal: regular-trace smoothing (Theorem 1.1)

There is an absolute real P≥2P\ge2P≥2 such that for every d≥1d\ge1d≥1 (so n=2d≥2n=2^d\ge2n=2d≥2)

Tr⁡reg∣Qn∣P≤1+n−10,\operatorname{Tr}_{\rm reg}|Q_n|^P\le 1+n^{-10},Trreg​∣Qn​∣P≤1+n−10,

and for every integer vvv with 2v≥P2v\ge P2v≥P and every initial deck σ\sigmaσ,

12∑g∈Sn∣μvd(gσ−1)−1n!∣≤12 n−5.\tfrac12\sum_{g\in S_n}\Bigl|\mu_{vd}(g\sigma^{-1})-\tfrac1{n!}\Bigr|\le\tfrac12\,n^{-5}.21​g∈Sn​∑​​μvd​(gσ−1)−n!1​​≤21​n−5.

The constant PPP is not specified; only its existence and uniformity in ddd are asserted. The goal is published on the platform with status Open: no machine-checked proof exists yet.

Significance

The result itself. Since one sweep is ddd physical shuffles, the second clause shows that O(d)=O(log⁡n)O(d)=O(\log n)O(d)=O(logn) Thorp shuffles bring the full permutation within 12n−5\tfrac12 n^{-5}21​n−5 of uniform from every starting deck. A counting argument on coin strings (Proposition 5.1 of the source) shows that at least 2d−O(1)2d-O(1)2d−O(1) physical shuffles are needed, so this order is optimal, improving the O(d3)O(d^3)O(d3) bound of Morris 2013. The regular-trace bound is stronger than an operator-norm bound for each irreducible: it controls all singular values with their multiplicities, which is what yields polynomially small total-variation distance.

Formalizing it. The statement is fully finite and explicit: an existential over one real constant followed by inequalities for each ddd. A formal proof would certify an optimal-order mixing bound for a widely studied shuffle. Specht modules, Schatten-type traces via continuous functional calculus, and the shuffle law are all defined in the mission's definition file and can be reused for related shuffling problems.

Difficulty

Single-card and few-card statistics mix after one sweep, but the full permutation involves all ∼nlog⁡n\sim n\log n∼nlogn bits of information, and the coin choices of a sweep interact across coordinates. The natural route—bounding the operator norm of Qn(λ)Q_n(\lambda)Qn​(λ) separately in each irreducible representation—does not suffice, because the regular trace weights each representation by its dimension DλD_\lambdaDλ​, and the number of partitions grows like ecne^{c\sqrt n}ecn​. A uniform power must work simultaneously for representations of tiny dimension (very long first row), intermediate dimension, and dimension close to n!\sqrt{n!}n!​, and splitting the coordinates into row and column groups produces two symmetry decompositions that do not commute.

Formalization scope

  • Positions are Fin d → Bool; permutations are Equiv.Perm (Card d); partitions of 2d2^d2d are Young diagrams of cardinality 2d2^d2d (Shapes (2^d)).
  • The Specht module is built concretely as the span of the orbit of the standard polytabloid in the space of functions on tabloids, transported to a Euclidean (Hilbert) space; cards are relabelled to cells by a fixed bijection. Q d μ is the average over all coin sequences of length ddd of the representation of run d d.
  • absoluteTrace A P is the real part of the trace of CFC.rpow (CFC.sqrt (A⋆A)) P, i.e. Tr⁡∣A∣P\operatorname{Tr}|A|^PTr∣A∣P; regularTrace multiplies by finrank.
  • sweepDistance d v σ is total variation of the law of v⋅dv\cdot dv⋅d physical shuffles, right-translated by σ\sigmaσ, from the uniform weight 1/(2d)!1/(2^d)!1/(2d)!. The integer vvv is converted with Int.toNat; since P≥2P\ge2P≥2 and 2v≥P2v\ge P2v≥P, v≥1v\ge1v≥1 and no truncation occurs.
  • The support lower bound (Proposition 5.1) is not part of the goal.
  • Infrastructure needed: Specht-module dimensions and branching, Schatten-norm inequalities (Hölder, Araki–Lieb–Thirring), Schur–Weyl duality, and the partial-deck density bound from the companion preprint on routing densities. Contributions formalizing any of these pieces are welcome.

Selected references

  • E. O. Thorp, Nonrandom shuffling with applications to the game of Faro, J. Amer. Statist. Assoc. 68 (1973), 842–847. https://doi.org/10.1080/01621459.1973.10481434
  • B. Morris, The mixing time of the Thorp shuffle, SIAM J. Comput. 38 (2008), 484–504. https://doi.org/10.1137/050636231
  • R. Montenegro and P. Tetali, Mathematical aspects of mixing times in Markov chains, Found. Trends Theor. Comput. Sci. 1 (2006), 237–354. https://doi.org/10.1561/0400000003
  • B. Morris, Improved mixing time bounds for the Thorp shuffle and L-reversal chain, Ann. Probab. 37 (2009), 453–477. https://doi.org/10.1214/08-AOP409
  • B. Morris, Improved mixing time bounds for the Thorp shuffle, Combin. Probab. Comput. 22 (2013), 118–132. https://doi.org/10.1017/S0963548312000478
  • H. Araki, On an inequality of Lieb and Thirring, Lett. Math. Phys. 19 (1990), 167–170. https://doi.org/10.1007/BF01045887
  • A. Berele and A. Regev, Hook Young diagrams with applications to combinatorics and to representations of Lie superalgebras, Adv. Math. 64 (1987), 118–175. https://doi.org/10.1016/0001-8708(87)90007-7
  • B. E. Sagan, The Symmetric Group, 2nd ed., GTM 203, Springer, 2001. https://doi.org/10.1007/978-1-4757-6804-6
  • OpenAI, Random-subspace tests and trace smoothing for coordinate sweeps, OpenAI Math Release preprint, September 26, 2026 (source of the goal; Theorem 1.1, p. 2). https://github.com/openai/math/blob/adc7f1241b42e322a6451854ab7e4b4c146bf78a/preprints/Random-subspace-tests-and-trace-smoothing-for-coordinate-sweeps-September-26-2026/paper.pdf
2 thms1 active userReviewed
Mathematical PhysicsProbability·Captain: wurtle

Stretched-exponential barriers for typical SK initial statesResearch Paper

Motivation

In the low-temperature phase β>1\beta>1β>1 of the Sherrington–Kirkpatrick (SK) spin glass, the Gibbs measure is believed to split into many nearly separated pieces, and local dynamics such as heat-bath (Glauber) updates should take a very long time to move between them. Worst-case slow mixing (some bad starting state) is a weak statement: it says nothing about what happens when the chain is started from a typical equilibrium sample, which is how such chains are used in practice. This mission asks for a much stronger obstruction: for most equilibrium configurations held fixed as starting points, the dynamics is still far from equilibrium after a stretched-exponential time.

Timeline

  • 1975. Sherrington and Kirkpatrick introduce the model (doi:10.1103/PhysRevLett.35.1792).
  • 1980. Parisi proposes his variational description of the free energy (doi:10.1088/0305-4470/13/4/009).
  • 2003, 2006. Guerra's interpolation bound and Talagrand's matching lower bound establish the Parisi formula (doi:10.1007/s00220-002-0773-5, doi:10.4007/annals.2006.163.221).
  • 2009–2010. Monthus–Garel and Billoire report numerical evidence for an n1/3n^{1/3}n1/3 scale of logarithmic relaxation times (doi:10.1088/1742-5468/2009/12/P12017, doi:10.1088/1742-5468/2010/11/P11034).
  • 2018. Ben Arous and Jagannath turn overlap free-energy barriers into exponentially small spectral gaps for mean-field spin glasses under landscape hypotheses that exclude pure SK (doi:10.1007/s00220-018-3152-6).
  • 2022. El Alaoui, Montanari and Sellke prove an obstruction to stable sampling algorithms at β>1\beta>1β>1 (arXiv:2203.05093).
  • 2025. Sellke proves exponentially slow worst-case Glauber mixing for SK at sufficiently large β\betaβ (arXiv:2511.22621).
  • 2026. Bandeira, El Alaoui and Rödder prove polynomial mixing at every β\betaβ under a large uniform external field (arXiv:2607.06813).

The source of this mission is an OpenAI preprint dated September 24, 2026. It covers the entire range β>1\beta>1β>1, at zero field, for typical rather than worst-case starts.

Setting

For n≥2n\ge2n≥2 and independent standard Gaussians (Jij)i<j(J_{ij})_{i<j}(Jij​)i<j​, put

HJ(x)=1n∑i<jJijxixj,πJ(x)=eβHJ(x)ZJ,x∈{−1,1}n.H^J(x)=\frac1{\sqrt n}\sum_{i<j}J_{ij}x_ix_j,\qquad \pi^J(x)=\frac{e^{\beta H^J(x)}}{Z^J},\qquad x\in\{-1,1\}^n .HJ(x)=n​1​i<j∑​Jij​xi​xj​,πJ(x)=ZJeβHJ(x)​,x∈{−1,1}n.

In continuous-time heat-bath dynamics each site has a rate-one clock; when site iii rings, its spin becomes s∈{±1}s\in\{\pm1\}s∈{±1} with probability eβshi/(2cosh⁡βhi)e^{\beta sh_i}/(2\cosh\beta h_i)eβshi​/(2coshβhi​), where hi=n−1/2∑j≠iJijxjh_i=n^{-1/2}\sum_{j\ne i}J_{ij}x_jhi​=n−1/2∑j=i​Jij​xj​. Let PtJP^J_tPtJ​ be its kernel and KJK^JKJ the discrete kernel that resamples one uniformly chosen site. For a fixed start xxx,

dJ(x,t)=∥PtJ(x,⋅)−πJ∥TV,ddiscJ(x,k)=∥(KJ)k(x,⋅)−πJ∥TV.d^J(x,t)=\|P^J_t(x,\cdot)-\pi^J\|_{\rm TV},\qquad d^J_{\rm disc}(x,k)=\|(K^J)^k(x,\cdot)-\pi^J\|_{\rm TV}.dJ(x,t)=∥PtJ​(x,⋅)−πJ∥TV​,ddiscJ​(x,k)=∥(KJ)k(x,⋅)−πJ∥TV​.

Formalization targets

Goal: Theorem 1.1

Let κ=1/10000\kappa=1/10000κ=1/10000. For every fixed β>1\beta>1β>1,

PJ, x0∼πJ{dJ(x0,enκ)>14}→1,PJ, x0∼πJ{ddiscJ(x0,⌊enκ⌋)>14}→1.\mathbb P_{J,\ x_0\sim\pi^J}\Bigl\{d^J\bigl(x_0,e^{n^\kappa}\bigr)>\tfrac14\Bigr\}\to1,\qquad \mathbb P_{J,\ x_0\sim\pi^J}\Bigl\{d^J_{\rm disc}\bigl(x_0,\lfloor e^{n^\kappa}\rfloor\bigr)>\tfrac14\Bigr\}\to1 .PJ, x0​∼πJ​{dJ(x0​,enκ)>41​}→1,PJ, x0​∼πJ​{ddiscJ​(x0​,⌊enκ⌋)>41​}→1.

The order matters: x0x_0x0​ is drawn from πJ\pi^JπJ and then held fixed as the starting state. Lean: OAI.SK.main, open on the platform; it writes the joint probability as the disorder average of the Gibbs mass of slow starts.

Significance

The theorem shows that throughout the low-temperature phase, a Gibbs-sampled start carries information that local updates cannot erase in time en1/10000e^{n^{1/10000}}en1/10000; in particular (Corollary 1.2) every polynomial time scale is too short from a typical start. This separates the zero-field low-temperature SK model from the high-temperature and large-field regimes, where polynomial mixing holds. The exponent 1/100001/100001/10000 is not claimed to be optimal; numerical studies suggest n1/3n^{1/3}n1/3.

The result is proved in an OpenAI preprint; it has not been peer reviewed, and no machine-checked proof exists.

Difficulty

Integrating the transition kernel over the starting state recovers πJ\pi^JπJ at every time, so averaging arguments are useless; the obstruction must be established start by start, for a set of starts of Gibbs mass tending to one. Free-energy barrier criteria in the style of Ben Arous–Jagannath do not apply to the pure quadratic SK covariance. The proof needs quantitative Parisi-type comparisons for constrained replica overlaps, and a mechanism ("locking") that forces small overlaps with a third replica to stay put when two replicas overlap substantially.

Formalization scope

  • hamiltonian J x = (∑_{i<j} J_{ij} x_i x_j)/√n; disorderLaw n is the product of standard Gaussians over pairs i<ji<ji<j.
  • heatBath β J x y is the one-step kernel: average over sites iii of the conditional resampling probability, nonzero only when yyy agrees with xxx off iii (so holding is included).
  • discreteKernel iterates it; continuousKernel β J t = ∑_k e^{-nt}(nt)^k/k! · K^k, the uniformized rate-nnn chain, equivalent to rate-one clocks per site.
  • continuousBadMass β n and discreteBadMass β n are EJ\mathbb E_JEJ​ of the Gibbs mass of starts with distance >1/4>1/4>1/4 at time timeScale n = exp(n^κ) (resp. its floor).
  • The goal asserts both bad masses tend to 111 for each fixed β>1\beta>1β>1.

Infrastructure: finite Markov chains, Gaussian interpolation (Guerra–Talagrand bounds for constrained overlaps), Parisi functional regularity, and Gaussian concentration. The paper's Theorem 2.4 (quantitative free-energy comparison), Proposition 4.4 (absolute-overlap locking) and Proposition 6.1 (stretched-exponential coverage) are natural intermediate targets.

Selected references

  • OpenAI, Stretched-exponential barriers for typical SK initial states, preprint, September 24, 2026. https://github.com/openai/math/blob/adc7f1241b42e322a6451854ab7e4b4c146bf78a/preprints/Stretched-exponential-barriers-for-typical-SK-initial-states-September-24-2026/paper.pdf
  • D. Sherrington, S. Kirkpatrick, Solvable model of a spin-glass, Phys. Rev. Lett., 1975. https://doi.org/10.1103/PhysRevLett.35.1792
  • G. Parisi, A sequence of approximated solutions to the S-K model for spin glasses, J. Phys. A, 1980. https://doi.org/10.1088/0305-4470/13/4/009
  • F. Guerra, Broken replica symmetry bounds in the mean field spin glass model, Comm. Math. Phys., 2003. https://doi.org/10.1007/s00220-002-0773-5
  • M. Talagrand, The Parisi formula, Ann. of Math., 2006. https://doi.org/10.4007/annals.2006.163.221
  • G. Ben Arous, A. Jagannath, Spectral gap estimates in mean field spin glasses, Comm. Math. Phys., 2018. https://doi.org/10.1007/s00220-018-3152-6
  • M. Sellke, Exponentially slow mixing of the low temperature SK model, preprint, 2025. https://arxiv.org/abs/2511.22621
  • A. S. Bandeira, A. El Alaoui, A. Rödder, Mixing of Glauber dynamics on high overlap Gibbs measures, preprint, 2026. https://arxiv.org/abs/2607.06813
  • A. El Alaoui, A. Montanari, M. Sellke, Sampling from the Sherrington–Kirkpatrick Gibbs measure via algorithmic stochastic localization, FOCS 2022. https://arxiv.org/abs/2203.05093
  • D. Panchenko, Free energy in the mixed p-spin models with vector spins, Ann. Probab., 2018. https://doi.org/10.1214/17-AOP1194
2 thms1 active userReviewed
Mathematical PhysicsProbability·Captain: wurtle

Critical slowing down in the Sherrington–Kirkpatrick modelResearch Paper

Motivation

The Sherrington–Kirkpatrick (SK) model is the mean-field spin glass, and β=1\beta=1β=1 is its equilibrium critical point. Physicists expect critical slowing down: at a phase transition, local dynamics needs time growing polynomially in the system size to relax. For the SK model this was studied through linearized mean-field theory, dynamical equations for correlation and response, and simulations of equilibrium autocorrelations on an n2/3n^{2/3}n2/3 time scale. Rigorous results on Glauber dynamics had concentrated on high temperature, where mixing is fast. This mission asks for a rigorous total-variation obstruction at β=1\beta=1β=1: most equilibrium configurations, used as starting states, are still far from equilibrium after o(n2/3)o(n^{2/3})o(n2/3) time.

Timeline

  • 1975, 1978. Sherrington and Kirkpatrick introduce the model and study Glauber autocorrelations above the critical temperature (doi:10.1103/PhysRevLett.35.1792, doi:10.1103/PhysRevB.17.4384).
  • 1976. Kosterlitz, Thouless and Jones analyze the spherical spin glass, with its transition at the spectral edge (doi:10.1103/PhysRevLett.36.1217).
  • 1981–1982. Sompolinsky and Zippelius develop dynamical equations for correlation and response (doi:10.1103/PhysRevLett.47.359, doi:10.1103/PhysRevB.25.6860).
  • 1996. Comets relates the Ising and spherical partition functions via a Haar-averaging identity (doi:10.24033/ast.331).
  • 2011. Billoire and Campbell numerically examine critical autocorrelations on the n2/3n^{2/3}n2/3 scale (doi:10.1103/PhysRevB.84.054442).
  • 2016, 2022. Baik and Lee, and Landon, prove free-energy fluctuation limits for the spherical model, including criticality (doi:10.1007/s10955-016-1610-0, doi:10.1063/5.0054298).
  • 2019, 2022. Bauerschmidt–Bodineau and Eldan–Koehler–Zeitouni prove functional inequalities at high temperature (doi:10.1016/j.jfa.2019.01.007, doi:10.1007/s00440-021-01085-x).
  • 2026. Du and Huang prove critical free-energy and overlap results (arXiv:2607.02172, arXiv:2608.08752); Wang and Boban–Li–Oveis Gharan prove fast mixing for β<1/2\beta<1/2β<1/2 and β<1/2+ε0\beta<1/2+\varepsilon_0β<1/2+ε0​ (arXiv:2608.22159, arXiv:2609.13138).

The source of this mission is an OpenAI preprint dated September 24, 2026.

Setting

At β=1\beta=1β=1 and zero field, let Wij∼N(0,1/n)W_{ij}\sim N(0,1/n)Wij​∼N(0,1/n) be independent for i<ji<ji<j and

μW(x)∝exp⁡(∑i<jWijxixj),x∈{−1,1}n.\mu_W(x)\propto\exp\Bigl(\sum_{i<j}W_{ij}x_ix_j\Bigr),\qquad x\in\{-1,1\}^n .μW​(x)∝exp(i<j∑​Wij​xi​xj​),x∈{−1,1}n.

A heat-bath update at site iii resamples xix_ixi​ from its conditional law given the other spins. In continuous time every site has an independent rate-one clock (generator ∑i(Ki−I)\sum_i(K_i-I)∑i​(Ki​−I)); in discrete time each step updates a uniformly chosen site (kernel n−1∑iKin^{-1}\sum_iK_in−1∑i​Ki​). For a fixed start xxx, dtct(W,x)d^{\rm ct}_t(W,x)dtct​(W,x) and dkdt(W,x)d^{\rm dt}_k(W,x)dkdt​(W,x) are the total-variation distances to μW\mu_WμW​.

Formalization targets

Goal: Theorem 1.1 (realized equilibrium initial states)

For every deterministic tn≥0t_n\ge0tn​≥0 with tn=o(n2/3)t_n=o(n^{2/3})tn​=o(n2/3) and every deterministic integer kn=o(n5/3)k_n=o(n^{5/3})kn​=o(n5/3),

μW{x: dtnct(W,x)>1/4} → P  1,μW{x: dkndt(W,x)>1/4} → P  1.\mu_W\{x:\ d^{\rm ct}_{t_n}(W,x)>1/4\}\ \xrightarrow{\ \mathbb P\ }\ 1,\qquad \mu_W\{x:\ d^{\rm dt}_{k_n}(W,x)>1/4\}\ \xrightarrow{\ \mathbb P\ }\ 1 .μW​{x: dtn​ct​(W,x)>1/4}  P ​ 1,μW​{x: dkn​dt​(W,x)>1/4}  P ​ 1.

Lean: OAI.CriticalSK.realized_equilibrium_initial_states, open on the platform.

Milestone: Theorem 1.2 (covariance and linear tests)

With Σ=Cov⁡μW(x)\Sigma=\operatorname{Cov}_{\mu_W}(x)Σ=CovμW​​(x), D(f)=∑iμW[Var⁡(f∣x−i)]\mathcal D(f)=\sum_i\mu_W[\operatorname{Var}(f\mid x_{-i})]D(f)=∑i​μW​[Var(f∣x−i​)] and Rlin=sup⁡a≠0Var⁡(aTx)/D(aTx)R_{\rm lin}=\sup_{a\ne0}\operatorname{Var}(a^{\mathsf T}x)/\mathcal D(a^{\mathsf T}x)Rlin​=supa=0​Var(aTx)/D(aTx): for every Mn→∞M_n\to\inftyMn​→∞, with probability tending to one,

∥Σ∥, Rlin ∈ [n2/3/Mn, Mnn2/3].\|\Sigma\|,\ R_{\rm lin}\ \in\ \bigl[n^{2/3}/M_n,\ M_nn^{2/3}\bigr].∥Σ∥, Rlin​ ∈ [n2/3/Mn​, Mn​n2/3].

Milestones: Theorem 1.3 (critical mixing bounds)

For every fixed ϵ>0\epsilon>0ϵ>0,

P{n2/3−ϵ≤tmix≤eϵn}→1,P{n5/3−ϵ≤kmix≤eϵn}→1,\mathbb P\{n^{2/3-\epsilon}\le t_{\rm mix}\le e^{\epsilon n}\}\to1,\qquad \mathbb P\{n^{5/3-\epsilon}\le k_{\rm mix}\le e^{\epsilon n}\}\to1,P{n2/3−ϵ≤tmix​≤eϵn}→1,P{n5/3−ϵ≤kmix​≤eϵn}→1,

with a separate Lean target for the lower bounds alone.

Significance

The theorem shows that at criticality the slow direction is visible from typical equilibrium starts, not only from adversarial ones, and pins the time scale n2/3n^{2/3}n2/3 to the spectral edge of the coupling matrix. Theorem 1.2 identifies the covariance scale n2/3n^{2/3}n2/3, contrasting with bounded covariance at every β<1\beta<1β<1. A companion OpenAI preprint uses Theorems 1.1 and 1.2 as inputs for matching upper bounds n2/3+o(1)n^{2/3+o(1)}n2/3+o(1).

The results are proved in an OpenAI preprint; they have not been peer reviewed, and no machine-checked proof exists.

Difficulty

Averaging over the starting state gives back the stationary law, so the obstruction must be proved start by start. The slow observable is the projection vTxv^{\mathsf T}xvTx on the top eigenvector of the coupling matrix; the difficulty is the static estimate that this projection is rarely small on the n1/3n^{1/3}n1/3 scale under the Ising Gibbs law. This requires edge-of-spectrum random matrix estimates, a precise comparison between the Ising law and a tilted spherical law, and control of the conditioning on the sphere.

Formalization scope

  • hamiltonian W x = ∑_{i<j} W_{ij} x_i x_j; disorderLaw n is the product of gaussianReal 0 (1/n) over pairs i<ji<ji<j.
  • siteKernel W i is the heat-bath kernel on configurations agreeing off iii; discreteKernel = n^{-1}∑_i siteKernel; continuousKernel W t = exp(t ∑_i (siteKernel W i - 1)).
  • continuousGoodMass/discreteGoodMass are the Gibbs masses of starts with distance >1/4>1/4>1/4; convergence in probability is written as P(∣mass−1∣≥δ)→0\mathbb P(|\text{mass}-1|\ge\delta)\to0P(∣mass−1∣≥δ)→0 for all δ>0\delta>0δ>0.
  • covarianceNorm is the Euclidean operator norm; linearRayleigh is a real sSup over nonzero aaa (the quotient is bounded and the set nonempty).
  • Mixing times are sInf over times at which all starts are within 1/41/41/4.

Infrastructure: finite continuous-time Markov chains (matrix exponentials), GOE edge estimates, tridiagonal models, spherical integrals. The intermediate results Proposition 3.1, Theorem 5.3 and Proposition 5.4 are natural further milestones.

Selected references

  • OpenAI, Critical slowing down in the Sherrington–Kirkpatrick model, preprint, September 24, 2026. https://github.com/openai/math/blob/adc7f1241b42e322a6451854ab7e4b4c146bf78a/preprints/Critical-slowing-down-in-the-Sherrington-Kirkpatrick-model-September-24-2026/paper.pdf
  • D. Sherrington, S. Kirkpatrick, Solvable model of a spin-glass, Phys. Rev. Lett., 1975. https://doi.org/10.1103/PhysRevLett.35.1792
  • H. Sompolinsky, A. Zippelius, Dynamic theory of the spin-glass phase, Phys. Rev. Lett., 1981. https://doi.org/10.1103/PhysRevLett.47.359
  • F. Comets, A spherical bound for the Sherrington–Kirkpatrick model, Astérisque, 1996. https://doi.org/10.24033/ast.331
  • H. Du, B. Huang, Fluctuations of the Sherrington–Kirkpatrick free energy at critical temperature, preprint, 2026. https://arxiv.org/abs/2607.02172
  • H. Du, B. Huang, Overlap distribution of the critical Sherrington–Kirkpatrick model, preprint, 2026. https://arxiv.org/abs/2608.08752
  • J. Baik, J. O. Lee, Fluctuations of the free energy of the spherical Sherrington–Kirkpatrick model, J. Stat. Phys., 2016. https://doi.org/10.1007/s10955-016-1610-0
  • B. Landon, Free energy fluctuations of the two-spin spherical SK model at critical temperature, J. Math. Phys., 2022. https://doi.org/10.1063/5.0054298
  • A. Billoire, I. A. Campbell, Dynamics in the Sherrington–Kirkpatrick Ising spin glass at and above TgT_gTg​, Phys. Rev. B, 2011. https://doi.org/10.1103/PhysRevB.84.054442
2 thms1 active userReviewed
Mathematical PhysicsProbability·Captain: wurtle

A spectral gap throughout the high-temperature Sherrington–Kirkpatrick phaseResearch Paper

Motivation

The Sherrington–Kirkpatrick (SK) model (1975) is the basic mean-field spin glass: nnn Ising spins interact through independent Gaussian couplings of variance β2/n\beta^2/nβ2/n, so each spin is weakly coupled to every other one with random signs. The model is the testing ground for questions about disordered systems, about sampling from Gibbs measures, and about the performance of Markov chain Monte Carlo. A central question is whether Glauber (heat-bath) dynamics, which resamples one spin at a time from its conditional law, relaxes at a rate independent of the system size. Equivalently: does the Gibbs measure satisfy a dimension-free Poincaré inequality with respect to the heat-bath Dirichlet form?

The equilibrium high-temperature phase is β<1\beta<1β<1, where the free energy equals its annealed value (Aizenman, Lebowitz and Ruelle 1987). Dynamical results had reached only part of this range.

Timeline

  • 1963. Glauber introduces local stochastic spin dynamics (doi:10.1063/1.1703954).
  • 1975. Sherrington and Kirkpatrick introduce the model (doi:10.1103/PhysRevLett.35.1792).
  • 1987. Aizenman, Lebowitz and Ruelle prove the annealed free energy log⁡2+β2/4\log2+\beta^2/4log2+β2/4 for all β<1\beta<1β<1 (doi:10.1007/BF01217677).
  • 2006. Wu proves Poincaré inequalities under a Dobrushin-type condition (doi:10.1214/009117906000000368).
  • 2019. Bauerschmidt and Bodineau prove a high-temperature log-Sobolev inequality for a different (full spin-flip) Dirichlet form (doi:10.1016/j.jfa.2019.01.007).
  • 2022. Eldan, Koehler and Zeitouni give a spectral condition yielding a heat-bath gap for β<1/4\beta<1/4β<1/4 (doi:10.1007/s00440-021-01085-x); Anari, Jain, Koehler, Pham and Vuong prove O(nlog⁡n)O(n\log n)O(nlogn) mixing in the same range (arXiv:2106.04105).
  • 2024. Anari, Koehler and Vuong extend this to β≈0.295\beta\approx0.295β≈0.295 (doi:10.1145/3618260.3649622); Adhikari, Brennecke, Xu and Yau obtain gaps for mixed ppp-spin models under a small-coefficient condition (doi:10.1007/s00440-024-01261-9); El Alaoui and Gaitonde, and Brennecke, Schertzer, Xu and Yau, control the spin covariance for all β<1\beta<1β<1 (arXiv:2212.02445, doi:10.2140/pmp.2024.5.131).
  • 2026. Wang proves optimal mixing for β<1/2\beta<1/2β<1/2 (arXiv:2608.22159); Boban, Li and Oveis Gharan reach β<1/2+ε0\beta<1/2+\varepsilon_0β<1/2+ε0​ (arXiv:2609.13138).

The source of this mission is an OpenAI preprint dated September 24, 2026, which treats every fixed β<1\beta<1β<1.

Setting

Fix 0<β<10<\beta<10<β<1. Let JJJ be a symmetric n×nn\times nn×n matrix with zero diagonal and independent entries Jik∼N(0,β2/n)J_{ik}\sim N(0,\beta^2/n)Jik​∼N(0,β2/n) for i<ki<ki<k. The zero-field Gibbs law on {−1,1}n\{-1,1\}^n{−1,1}n is

μ0(x)=1Zexp⁡{12xTJx}.\mu_0(x)=\frac1{Z}\exp\Bigl\{\tfrac12x^{\mathsf T}Jx\Bigr\}.μ0​(x)=Z1​exp{21​xTJx}.

Let PiP_iPi​ be the conditional expectation under μ0\mu_0μ0​ given all spins except xix_ixi​; it averages fff over xxx and its iii-th flip with Gibbs weights. The unscaled Dirichlet form is

D(f)=∑i=1nEμ0(f−Pif)2.\mathcal D(f)=\sum_{i=1}^n\mathbb E_{\mu_0}\bigl(f-P_if\bigr)^2 .D(f)=i=1∑n​Eμ0​​(f−Pi​f)2.

The discrete heat-bath chain picks a uniform site and resamples it, with transition operator n−1∑iPin^{-1}\sum_iP_in−1∑i​Pi​; its spectral gap is inf⁡fD(f)/(nVar⁡μ0f)\inf_f \mathcal D(f)/(n\operatorname{Var}_{\mu_0}f)inff​D(f)/(nVarμ0​​f) over nonconstant fff.

Formalization targets

Goal: Theorem 1.1

There is a finite constant CβC_\betaCβ​ such that

PJ{Var⁡μ0(f)≤Cβ D(f) for every f:{−1,1}n→R} ⟶ 1,\mathbb P_J\Bigl\{\operatorname{Var}_{\mu_0}(f)\le C_\beta\,\mathcal D(f)\ \text{for every }f:\{-1,1\}^n\to\mathbb R\Bigr\}\ \longrightarrow\ 1,PJ​{Varμ0​​(f)≤Cβ​D(f) for every f:{−1,1}n→R} ⟶ 1,

and consequently the discrete heat-bath chain has spectral gap at least 1/(Cβn)1/(C_\beta n)1/(Cβ​n) with probability tending to one. Lean: OAI.SKGap.sk_main, open on the platform.

Significance

The theorem gives a size-independent relaxation time for continuous-time heat-bath dynamics (rate-one clocks) throughout the high-temperature phase, for every observable simultaneously on one disorder event. It extends the dynamical picture from β<1/2+ε0\beta<1/2+\varepsilon_0β<1/2+ε0​ to the full equilibrium range β<1\beta<1β<1, where only linear-observable (covariance) control was previously known. A Poincaré inequality is the input for L2L^2L2 mixing bounds and for companion results on cutoff and critical slowing down.

The result is proved in an OpenAI preprint; it has not been peer reviewed, and no machine-checked proof exists.

Difficulty

Dobrushin-type influence bounds fail because the sum of absolute couplings at a site grows like n\sqrt nn​; any argument must exploit cancellations from the random signs. Covariance bounds control only linear observables, while the Poincaré inequality must hold for all 22n2^{2^n}22n-dimensional families of test functions at once. The natural route through stochastic localization runs into rare observation events: an event of exponentially small probability can still carry most of the variance of a particular test function, and invertibility of the field equation must hold at every sufficiently accurate solution along the observation path.

Formalization scope

  • Spins are Fin n → Bool, with spinValue mapping to ±1\pm1±1; couplings are indexed by pairs i<ki<ki<k (Edge n) and symmetrized with zero diagonal by coupling.
  • hamiltonian g 0 x = ½ ∑_{i,k} x_i J_{ik} x_k; mass, expectation, variance are exact finite sums over {−1,1}n\{-1,1\}^n{−1,1}n.
  • conditionalExpectation g h i f x is the two-point Gibbs average over xxx and its iii-th flip; dirichlet is the unscaled form ∑iE(f−Pif)2\sum_i\mathbb E(f-P_if)^2∑i​E(f−Pi​f)2.
  • disorderLaw β n = Measure.pi (fun _ => gaussianReal 0 (β^2/n)) over the edges (variance β2/n\beta^2/nβ2/n).
  • discreteGap g is the infimum of D(f)/(nVar⁡f)\mathcal D(f)/(n\operatorname{Var}f)D(f)/(nVarf) over fff with positive variance.
  • MainStatement β asserts a single C>0C>0C>0 for which both the Poincaré event and the gap event have probability tending to 111.

Formalization needs Gaussian product measures, finite Gibbs measures and heat-bath operators, and the analysis behind the proof: matrix concentration, TAP/AMP-type field recursions and stochastic localization. Formal versions of the paper's intermediate results (for example Proposition 8.3, the terminal observation gap) are welcome.

Selected references

  • OpenAI, A spectral gap throughout the high-temperature Sherrington–Kirkpatrick phase, preprint, September 24, 2026. https://github.com/openai/math/blob/adc7f1241b42e322a6451854ab7e4b4c146bf78a/preprints/A-spectral-gap-throughout-the-high-temperature-Sherrington-Kirkpatrick-phase-September-24-2026/main.pdf
  • D. Sherrington, S. Kirkpatrick, Solvable model of a spin-glass, Phys. Rev. Lett., 1975. https://doi.org/10.1103/PhysRevLett.35.1792
  • M. Aizenman, J. L. Lebowitz, D. Ruelle, Some rigorous results on the Sherrington–Kirkpatrick spin glass model, Comm. Math. Phys., 1987. https://doi.org/10.1007/BF01217677
  • R. Eldan, F. Koehler, O. Zeitouni, A spectral condition for spectral gap: fast mixing in high-temperature Ising models, Probab. Theory Related Fields, 2022. https://doi.org/10.1007/s00440-021-01085-x
  • N. Anari, F. Koehler, T.-D. Vuong, Trickle-down in localization schemes and applications, STOC 2024. https://doi.org/10.1145/3618260.3649622
  • S. Wang, Optimal mixing of Glauber dynamics for the Sherrington–Kirkpatrick model at β<1/2\beta<1/2β<1/2, preprint, 2026. https://arxiv.org/abs/2608.22159
  • M. Boban, A. Li, S. Oveis Gharan, Rank-1-perturbed trickledown theorems, preprint, 2026. https://arxiv.org/abs/2609.13138
  • C. Brennecke, A. Schertzer, C. Xu, H.-T. Yau, The two point function of the SK model without external field at high temperature, Probab. Math. Phys., 2024. https://doi.org/10.2140/pmp.2024.5.131
  • Y. Chen, R. Eldan, Localization schemes: a framework for proving mixing bounds for Markov chains, Duke Math. J., 2025. https://doi.org/10.1215/00127094-2024-0063
  • R. Eldan, Thin shell implies spectral gap up to polylog via a stochastic localization scheme, Geom. Funct. Anal., 2013. https://doi.org/10.1007/s00039-013-0214-y
2 thms1 active userReviewed
CombinatoricsTheoretical Computer Science·Captain: wurtle

An FPRAS for Cell-Bounded Contingency TablesResearch Paper

Motivation

A cell-bounded contingency table is a nonnegative integer matrix with prescribed row sums, column sums and an upper bound on each entry. Bounds of zero forbid entries (structural zeros), and bounds of one give bipartite graphs with prescribed degrees, so the model contains degree-constrained subgraph counting and perfect matchings (the permanent). Counting such tables is central to exact conditional inference in statistics and is equivalent to counting integer flows in bipartite networks. Exact counting is #P-complete even for two-row ordinary tables (Dyer, Kannan and Mount, 1997), so the target is a fully polynomial randomized approximation scheme (FPRAS): a randomized algorithm that, for any ε,δ∈(0,1)\varepsilon,\delta\in(0,1)ε,δ∈(0,1), returns a (1±ε)(1\pm\varepsilon)(1±ε)-approximation with probability at least 1−δ1-\delta1−δ in time polynomial in the input length, 1/ε1/\varepsilon1/ε and log⁡(1/δ)\log(1/\delta)log(1/δ).

Timeline

  • 1986. Jerrum, Valiant and Vazirani relate approximate counting and sampling through self-reducibility.
  • 1995. Diaconis and Gangolli survey fixed-margin arrays (doi:10.1007/978-1-4612-0801-3_3).
  • 1997. Dyer, Kannan and Mount prove #P-completeness for two-row tables and give geometric methods for large margins; Morris improves the margin requirements in 2002 (doi:10.1002/rsa.10049).
  • 2000–2003. Dyer and Greenhill (two rows), Cryan and Dyer (fixed number of rows), Dyer (dynamic programming) give approximate counting for a bounded number of rows.
  • 2004. Jerrum, Sinclair and Vigoda's permanent algorithm yields an FPRAS for all 0/1 caps (doi:10.1145/1008731.1008738).
  • 2009–2010. Barvinok bounds weighted table counts within NO(m+n)N^{O(m+n)}NO(m+n) (doi:10.1093/imrn/rnn133); Barvinok, Luria, Samorodnitsky and Yong give quasipolynomial randomized approximations for smooth margins (doi:10.1002/rsa.20301).
  • 2010. Cryan, Dyer and Randall give an FPRAS for integral flows with large tight capacities and for cell-bounded tables with a fixed number of rows, and list unrestricted approximation as open (doi:10.1137/060650544).
  • 2023. Brändén, Leake and Pak give capacity lower bounds via Lorentzian polynomials (doi:10.1007/s11856-022-2364-9); Guo and Jerrum count vertices of certain 0/1 polytopes (doi:10.1007/s00454-022-00406-8).

The source of this mission, an OpenAI preprint dated September 24, 2026, claims an FPRAS with no restriction on dimensions, margins or caps.

Setting

For positive integers m,nm,nm,n, margins r∈Z≥0mr\in\mathbb Z_{\ge0}^mr∈Z≥0m​, c∈Z≥0nc\in\mathbb Z_{\ge0}^nc∈Z≥0n​ with equal totals, and bounds b∈Z≥0m×nb\in\mathbb Z_{\ge0}^{m\times n}b∈Z≥0m×n​,

Ω(r,c,b)={X∈Z≥0m×n: ∑jXij=ri, ∑iXij=cj, Xij≤bij},Z(r,c,b)=∣Ω(r,c,b)∣.\Omega(r,c,b)=\Bigl\{X\in\mathbb Z_{\ge0}^{m\times n}:\ \textstyle\sum_jX_{ij}=r_i,\ \sum_iX_{ij}=c_j,\ X_{ij}\le b_{ij}\Bigr\},\qquad Z(r,c,b)=|\Omega(r,c,b)|.Ω(r,c,b)={X∈Z≥0m×n​: ∑j​Xij​=ri​, ∑i​Xij​=cj​, Xij​≤bij​},Z(r,c,b)=∣Ω(r,c,b)∣.

All numbers are written in binary, so a capacity can be exponentially larger than its description. LLL denotes the total binary input length including the rational accuracy parameters ε,δ\varepsilon,\deltaε,δ. The algorithm uses independent unbiased random bits and is charged for every bit operation.

Formalization targets

Goal: Theorem 1.1 (FPRAS)

One randomized algorithm, given (r,c,b)(r,c,b)(r,c,b) and rational ε,δ∈(0,1)\varepsilon,\delta\in(0,1)ε,δ∈(0,1), outputs a nonnegative rational Z^\widehat ZZ with

Pr⁡[(1−ε)Z(r,c,b)≤Z^≤(1+ε)Z(r,c,b)]≥1−δ,\Pr\bigl[(1-\varepsilon)Z(r,c,b)\le\widehat Z\le(1+\varepsilon)Z(r,c,b)\bigr]\ge1-\delta,Pr[(1−ε)Z(r,c,b)≤Z≤(1+ε)Z(r,c,b)]≥1−δ,

outputs exactly 000 on every execution when Ω(r,c,b)=∅\Omega(r,c,b)=\emptysetΩ(r,c,b)=∅, and performs at most poly(L,ε−1,log⁡δ−1)\mathrm{poly}(L,\varepsilon^{-1},\log\delta^{-1})poly(L,ε−1,logδ−1) bit operations on every execution. Lean: OAI.ContingencyTables.counting, open on the platform.

Significance

The theorem would answer the question left open by Cryan, Dyer and Randall in 2010: approximate counting of cell-bounded tables (equivalently, integral flows in bipartite networks with arbitrary capacities) without any fixed-dimension, large-capacity, density or balance assumption, with polynomial cost on every execution and no oracle. It contains the 0/1 case handled by Jerrum–Sinclair–Vigoda and the fixed-row results as special cases, and via self-reduction it gives approximate samplers for integer flows. A companion preprint of the same family treats exact uniform sampling of uncapped tables. The result is claimed in an OpenAI preprint that has not been peer reviewed; no machine-checked proof exists.

Difficulty

Two regimes clash. Large capacities cannot be expanded into that many binary choices without exponential blow-up, so geometric (polytope volume) methods are needed for them; but those methods require every cell to be large. Small capacities need combinatorial Markov chains, which in turn must preserve exactly one unit of mass per table despite the multiplicities created by encoding a bounded integer by binary choices. Arbitrary mixtures of tiny and huge caps, zero caps and variable dimensions defeat each approach used alone.

Formalization scope

  • Algorithms are MatchingFPRAS.RandomMachines: a single finite transition table on an 8-symbol alphabet, one random bit per tick; time is counted in ticks (polynomially equivalent to bit operations).
  • Inputs are delimited binary encodings of m,n,r,c,bm,n,r,c,bm,n,r,c,b and of ε,δ\varepsilon,\deltaε,δ as reduced fractions (encodeCountingInput); outputs are encoded rationals.
  • count r c b is Fintype.card of bounded tables.
  • CountingStatement: one machine and constants C>0C>0C>0, ddd; with t=C(∣input∣+⌈1/ε⌉+⌈log⁡2⌈1/δ⌉⌉+1)dt=C(|\text{input}|+\lceil1/\varepsilon\rceil+\lceil\log_2\lceil1/\delta\rceil\rceil+1)^dt=C(∣input∣+⌈1/ε⌉+⌈log2​⌈1/δ⌉⌉+1)d, every tape of length ttt halts with a rational q≥0q\ge0q≥0; if count = 0 every tape outputs 000; and the fraction of tapes with (1−ε)Z≤q≤(1+ε)Z(1-\varepsilon)Z\le q\le(1+\varepsilon)Z(1−ε)Z≤q≤(1+ε)Z is at least 1−δ1-\delta1−δ.
  • m,n≥1m,n\ge1m,n≥1 and equal margin totals are hypotheses, as in the paper.

Needed infrastructure: randomized Turing machines, Markov chain mixing and Poincaré inequalities, log-concavity (Prékopa–Leindler), and network-flow feasibility. The machine model and encodings are shared with the exact-sampling mission of this family. Contributions formalizing Proposition 2.2 (contamination bound), Lemma 3.3 (log-concave tree marginals), Proposition 6.1 (weight evaluation) or Proposition 7.2 are welcome.

Selected references

  • OpenAI, An FPRAS for Cell-Bounded Contingency Tables, preprint, September 24, 2026. https://github.com/openai/math/blob/adc7f1241b42e322a6451854ab7e4b4c146bf78a/preprints/An-FPRAS-for-Cell-Bounded-Contingency-Tables-September-24-2026/main.pdf
  • M. Cryan, M. Dyer, D. Randall, Approximately counting integral flows and cell-bounded contingency tables, SIAM J. Comput., 2010. https://doi.org/10.1137/060650544
  • M. Jerrum, A. Sinclair, E. Vigoda, A polynomial-time approximation algorithm for the permanent of a matrix with nonnegative entries, J. ACM, 2004. https://doi.org/10.1145/1008731.1008738
  • B. Morris, Improved bounds for sampling contingency tables, Random Structures Algorithms, 2002. https://doi.org/10.1002/rsa.10049
  • A. Barvinok, Asymptotic estimates for the number of contingency tables, integer flows, and volumes of transportation polytopes, IMRN, 2009. https://doi.org/10.1093/imrn/rnn133
  • A. Barvinok, Z. Luria, A. Samorodnitsky, A. Yong, An approximation algorithm for counting contingency tables, Random Structures Algorithms, 2010. https://doi.org/10.1002/rsa.20301
  • P. Brändén, J. Leake, I. Pak, Lower bounds for contingency tables via Lorentzian polynomials, Israel J. Math., 2023. https://doi.org/10.1007/s11856-022-2364-9
  • H. Guo, M. Jerrum, Counting vertices of integral polytopes defined by facets, Discrete Comput. Geom., 2023. https://doi.org/10.1007/s00454-022-00406-8
2 thms1 active userReviewed
CombinatoricsTheoretical Computer Science·Captain: wurtle

Exact Uniform Sampling of Contingency Tables with Arbitrary MarginsResearch Paper

Motivation

A contingency table with margins r=(r1,…,rm)r=(r_1,\dots,r_m)r=(r1​,…,rm​) and c=(c1,…,cn)c=(c_1,\dots,c_n)c=(c1​,…,cn​) is a nonnegative integer matrix whose row sums are rrr and column sums are ccc. Such tables are the basic objects of categorical data analysis: conditional tests of independence (Fisher's exact test and its generalizations) compare an observed table with the uniform distribution on all tables with the same margins, and in practice this requires sampling uniformly from that set. The set is finite but can be exponentially large in the input size when margins are written in binary. Whether one can sample it (approximately or exactly) in time polynomial in both dimensions and in the bit length of the margins has been open since the 1990s.

Timeline

  • 1995. Diaconis and Gangolli survey enumeration and random generation of rectangular arrays with fixed margins (doi:10.1007/978-1-4612-0801-3_3).
  • 1997–2002. Dyer, Kannan and Mount sample dense tables via the transportation polytope; Morris improves the dense bounds (doi:10.1002/rsa.10049).
  • 2000. Dyer and Greenhill give polynomial-time approximate counting and sampling for two-rowed tables.
  • 2003. Kijima and Matsui give exact sampling for two rows by coupling from the past; Cryan and Dyer approximately count for a fixed number of rows (doi:10.1016/S0022-0000(03)00014-X); Dyer gives an exact dynamic-programming sampler for fixed row count.
  • 2006. Cryan, Dyer, Goldberg, Jerrum and Martin prove rapid mixing of the heat-bath chain for every fixed number of rows (doi:10.1137/S0097539703434243).
  • 2016. DeSalvo and Zhao give an exact divide-and-conquer sampler whose polynomial runtime is conditional on a conjecture (arXiv:1507.00070).
  • 2021. Arman, Gao and Wormald give linear-time exact generation for sparse margins with 5Δ4<N5\Delta^4<N5Δ4<N, and report that no polynomial-time approximately uniform sampler for arbitrary margins was known (arXiv:2104.09413).
  • 2024. Göbel, Liu, Manurangsi and Pappik turn sufficiently accurate samplers into perfect samplers (arXiv:2410.00882).

The source of this mission, an OpenAI preprint dated September 24, 2026, claims both an almost-uniform and an exactly uniform sampler for arbitrary margins.

Setting

For nonnegative integer vectors r∈Z≥0mr\in\mathbb Z_{\ge0}^mr∈Z≥0m​, c∈Z≥0nc\in\mathbb Z_{\ge0}^nc∈Z≥0n​ with common total N=∑iri=∑jcjN=\sum_ir_i=\sum_jc_jN=∑i​ri​=∑j​cj​, let

Ω(r,c)={X∈Z≥0m×n: ∑jXij=ri, ∑iXij=cj}.\Omega(r,c)=\Bigl\{X\in\mathbb Z_{\ge0}^{m\times n}:\ \textstyle\sum_jX_{ij}=r_i,\ \sum_iX_{ij}=c_j\Bigr\}.Ω(r,c)={X∈Z≥0m×n​: ∑j​Xij​=ri​, ∑i​Xij​=cj​}.

The input is (m,n,r,c)(m,n,r,c)(m,n,r,c) with margins in binary, so its length is polynomial in mmm, nnn and log⁡(N+1)\log(N+1)log(N+1). An algorithm uses unbiased random bits. For laws on a finite set, TV(μ,ν)=12∑x∣μ(x)−ν(x)∣\mathrm{TV}(\mu,\nu)=\tfrac12\sum_x|\mu(x)-\nu(x)|TV(μ,ν)=21​∑x​∣μ(x)−ν(x)∣.

Formalization targets

Milestone: Theorem 1.1(i) (almost-uniform, bounded time)

For each k≥1k\ge1k≥1, an algorithm halts on every execution in time poly(m,n,log⁡(N+1),k)\mathrm{poly}(m,n,\log(N+1),k)poly(m,n,log(N+1),k) and outputs X∈Ω(r,c)X\in\Omega(r,c)X∈Ω(r,c) with

TV(law(X), Unif Ω(r,c))≤2−k.\mathrm{TV}\bigl(\mathrm{law}(X),\ \mathrm{Unif}\,\Omega(r,c)\bigr)\le2^{-k}.TV(law(X), UnifΩ(r,c))≤2−k.

Goal: Theorem 1.1(ii) (exact uniform sampling)

An algorithm outputs X∈Ω(r,c)X\in\Omega(r,c)X∈Ω(r,c) with

Pr⁡[X=Y]=1∣Ω(r,c)∣(Y∈Ω(r,c)),\Pr[X=Y]=\frac1{|\Omega(r,c)|}\quad(Y\in\Omega(r,c)),Pr[X=Y]=∣Ω(r,c)∣1​(Y∈Ω(r,c)),

terminates almost surely, and has expected running time poly(m,n,log⁡(N+1))\mathrm{poly}(m,n,\log(N+1))poly(m,n,log(N+1)), with polynomials uniform over all margins. Lean: OAI.ContingencyTables.exactSampling, open on the platform.

Significance

The theorem would give the first unconditional polynomial-time exact uniform sampler for contingency tables with arbitrary dimensions and binary margins, with no positivity, sparsity, balance or fixed-dimension restriction, and with all random bits and integer arithmetic counted. It removes the fixed-row restriction of the Dyer–Greenhill and Cryan–Dyer line and the sparsity condition of Arman–Gao–Wormald. The exponents are deliberately large; the contribution is the existence of a polynomial algorithm. Exact time is polynomial in expectation, not on every run. A companion preprint of the same family gives an FPRAS for counting tables with cell bounds. The result is claimed in an OpenAI preprint that has not been peer reviewed; no machine-checked proof exists.

Difficulty

Markov-chain approaches mix rapidly for a fixed number of rows, but the bounds degrade with the row count. Geometric (polytope) methods need all margins to be large, and fail when some margins are tiny while others are huge, which is exactly the mixed regime that arbitrary binary margins allow. Making the sampler exact rather than approximate adds a separate difficulty: the residual-mixture correction requires that the actual output law of the approximate sampler can be computed, with cost polynomial in the accuracy bits.

Formalization scope

  • Algorithms are MatchingFPRAS.RandomMachines: a single finite transition table over an 8-symbol tape alphabet; each tick reads one random bit and either moves, writes or halts. Time is counted in ticks, polynomially equivalent to bit operations.
  • Inputs are binary encodings with delimiters (encodeNat, encodeMargins); outputs are row-major matrix encodings left on the tape (OutputsTable).
  • tableMass A input t X and haltMass A input t are the fractions of the 2t2^t2t random tapes of length ttt that halt with output XXX, respectively halt at all, within ttt ticks.
  • ExactSamplingStatement: one machine and constants C>0C>0C>0, ddd such that for all m,n,r,cm,n,r,cm,n,r,c with equal totals: every halted output is in Ω(r,c)\Omega(r,c)Ω(r,c); tableMass → 1/|Ω| for each table; haltMass → 1; and ∑t(1−haltMasst)\sum_t(1-\text{haltMass}_t)∑t​(1−haltMasst​), the expected running time, is at most C(m+n+⌈log⁡2(N+1)⌉+1)dC(m+n+\lceil\log_2(N+1)\rceil+1)^dC(m+n+⌈log2​(N+1)⌉+1)d.
  • BoundedSamplingStatement: every tape of length t=C(m+n+⌈log⁡2(N+1)⌉+k+1)dt=C(m+n+\lceil\log_2(N+1)\rceil+k+1)^dt=C(m+n+⌈log2​(N+1)⌉+k+1)d halts with a feasible table, and the TV distance to uniform is at most 2−k2^{-k}2−k.
  • Dimensions m,nm,nm,n vary freely (zero allowed); the machine is uniform, not one per size.

Needed infrastructure: randomized Turing machines and their output laws, Markov chain mixing (conductance, Poincaré inequalities), and log-concave sampling. Contributions formalizing Theorem 4.3 (graph Poincaré inequality), Theorem 5.1 (dense completion draw), Proposition 6.3 (law tabulation) or the Göbel–Liu–Manurangsi–Pappik correction are welcome.

Selected references

  • OpenAI, Exact Uniform Sampling of Contingency Tables with Arbitrary Margins, preprint, September 24, 2026. https://github.com/openai/math/blob/adc7f1241b42e322a6451854ab7e4b4c146bf78a/preprints/Exact-Uniform-Sampling-of-Contingency-Tables-with-Arbitrary-Margins-September-24-2026/main.pdf
  • P. Diaconis, A. Gangolli, Rectangular arrays with fixed margins, Discrete Probability and Algorithms, 1995. https://doi.org/10.1007/978-1-4612-0801-3_3
  • B. Morris, Improved bounds for sampling contingency tables, Random Structures Algorithms, 2002. https://doi.org/10.1002/rsa.10049
  • M. Cryan, M. Dyer, A polynomial-time algorithm to approximately count contingency tables when the number of rows is constant, J. Comput. System Sci., 2003. https://doi.org/10.1016/S0022-0000(03)00014-X
  • M. Cryan, M. Dyer, L. A. Goldberg, M. Jerrum, R. Martin, Rapidly mixing Markov chains for sampling contingency tables with a constant number of rows, SIAM J. Comput., 2006. https://doi.org/10.1137/S0097539703434243
  • S. DeSalvo, J. Y. Zhao, Random sampling of contingency tables via probabilistic divide-and-conquer, preprint, 2016. https://arxiv.org/abs/1507.00070
  • A. Arman, P. Gao, N. Wormald, Linear-time uniform generation of random sparse contingency tables with specified marginals, preprint, 2021. https://arxiv.org/abs/2104.09413
  • A. Göbel, J. Liu, P. Manurangsi, M. Pappik, Perfect sampling from rapidly mixing Markov chains, preprint, 2024. https://arxiv.org/abs/2410.00882
3 thms1 active userReviewed
CombinatoricsTheoretical Computer Science·Captain: wurtle

Approximate counting of common bases of two matroidsResearch Paper

Motivation: counting common bases of two matroids

A matroid abstracts linear independence: a finite ground set with a family of "independent" subsets that is closed under subsets and satisfies the augmentation axiom. Its maximal independent sets, the bases, all have the same size, the rank. Matroid intersection, finding a common basis of two matroids, is one of the central polynomially solvable problems of combinatorial optimization (Edmonds, 1979). Counting common bases is much harder. It contains counting bases of a single matroid (take both matroids equal) and counting perfect matchings of a bipartite graph (take two partition matroids), and exact counting is #P-hard. The natural goal is therefore a fully polynomial randomized approximation scheme (FPRAS): an algorithm whose output is within relative error ϵ\epsilonϵ with probability at least 1−δ1-\delta1−δ, in time polynomial in the input size, ϵ−1\epsilon^{-1}ϵ−1 and log⁡δ−1\log\delta^{-1}logδ−1. The matroids should be accessible only through independence oracles.

Background and timeline

  • 1979 — Edmonds gives a polynomial-time algorithm for matroid intersection (Ann. Discrete Math. 4).
  • 1992 — Feder and Mihail give sampling and approximate counting for balanced matroids (STOC 1992).
  • 2004 — Jerrum, Sinclair and Vigoda give an FPRAS for the permanent of nonnegative matrices, covering bipartite perfect matchings (J. ACM 51).
  • 2007 — Barvinok and Samorodnitsky obtain coarse logarithmic estimates via random weighting (Israel J. Math. 158).
  • 2016–2017 — Anari, Oveis Gharan and Rezaei prove rapid mixing for strongly Rayleigh distributions (COLT 2016); Anari and Oveis Gharan obtain exponential-factor approximations from real-stable polynomials (arXiv:1702.02937).
  • 2020 — Brändén and Huh develop Lorentzian polynomials (Ann. of Math. 192).
  • 2021 — Anari, Oveis Gharan and Vinzant give a deterministic 2O(r)2^{O(r)}2O(r)-factor approximation for common bases of rank-rrr matroids (Duke Math. J. 170); Cryan, Guo and Mousa ask for a rapidly mixing chain for common bases (Ann. Probab. 49).
  • 2023–2024 — Liu's dissertation records the FPRAS question for two arbitrary oracle matroids (Section 13.2, Problem 1, thesis); Anari, Liu, Oveis Gharan and Vinzant give an FPRAS for bases of a single matroid (Ann. of Math. 199).
  • 2026 — An OpenAI preprint, Approximate counting of common bases of two matroids (OpenAI Math Release, September 23, 2026), claims an independence-oracle FPRAS for common bases of two arbitrary matroids of equal rank. It has not been peer reviewed, and its proof is not formally verified.

Setting

Let M1,M2M_1,M_2M1​,M2​ be matroids of the same rank rrr on the ground set [n][n][n], supplied by exact independence oracles: a query is an nnn-bit indicator of a subset, and the answer says whether it is independent. The quantity to estimate is

Z(M1,M2)=∣B(M1)∩B(M2)∣,Z(M_1,M_2)=\bigl|\mathcal B(M_1)\cap\mathcal B(M_2)\bigr|,Z(M1​,M2​)=​B(M1​)∩B(M2​)​,

the number of common bases. The input also contains rationals ϵ,δ∈(0,1)\epsilon,\delta\in(0,1)ϵ,δ∈(0,1); ℓ\ellℓ denotes the binary encoding length of n,r,ϵ,δn,r,\epsilon,\deltan,r,ϵ,δ. Oracle calls are counted separately from the bit operations used to build queries and process answers.

Formalization targets

Goal: an FPRAS for common bases in the independence-oracle model (Theorem 1.1)

There is a single randomized oracle algorithm and a fixed polynomial ppp such that, for all rank-rrr matroids M1,M2M_1,M_2M1​,M2​ on [n][n][n] and rational ϵ,δ∈(0,1)\epsilon,\delta\in(0,1)ϵ,δ∈(0,1), it outputs a nonnegative rational Z^\widehat ZZ with

Pr⁡[(1−ϵ)Z≤Z^≤(1+ϵ)Z] ≥ 1−δ,\Pr\bigl[(1-\epsilon)Z\le\widehat Z\le(1+\epsilon)Z\bigr]\ \ge\ 1-\delta,Pr[(1−ϵ)Z≤Z≤(1+ϵ)Z] ≥ 1−δ,

the output is always 000 when Z=0Z=0Z=0, and on every execution the numbers of oracle calls and bit operations are at most p(n,ℓ,ϵ−1,log⁡δ−1)p(n,\ell,\epsilon^{-1},\log\delta^{-1})p(n,ℓ,ϵ−1,logδ−1). The algorithm uses only independent fair random bits. The goal statement is published on the platform with status Open.

Significance

The result itself. Theorem 1.1 answers the question recorded by Liu (2023) and extends both the single-matroid FPRAS of Anari–Liu–Oveis Gharan–Vinzant and, in the bipartite-matching case, the permanent FPRAS of Jerrum–Sinclair–Vigoda. It needs no representation of either matroid and holds for unrestricted rank, whereas the previous oracle-model scheme was fully polynomial only for r=O(log⁡n)r=O(\log n)r=O(logn). The source derives FPRASs and almost-uniform samplers for common independent sets of prescribed, unrestricted or maximum size, even for matroids of different ranks (Corollary 9.1).

Formalizing it. The Lean goal fixes a concrete machine model, so it states a complete complexity-theoretic claim with no informal "polynomial-time" left. A proof would combine Lorentzian-polynomial inequalities (Brändén–Huh), a transport inequality, a Markov-chain Poincaré bound, and simulated annealing with explicit bit accounting. Each layer is reusable: Mathlib has matroids, but no Lorentzian polynomials, no mixing-time theory of this kind and no randomized-algorithm framework.

Difficulty

For one matroid, the bases-exchange walk mixes rapidly because the basis generating polynomial is log-concave. The intersection of two matroids has no such structure: its common bases need not be connected by short exchanges, and the natural chain on transversals of the paired construction passes through states of exponentially small weight, so positivity and connectivity alone give no polynomial bound. Earlier approaches either lost exponential factors or needed real-stable polynomials that an independence oracle does not provide. The proof must balance defect classes, as in Jerrum–Sinclair–Vigoda, and establish a Poincaré inequality for observables from matroid-theoretic inequalities alone.

Formalization scope

  • Matroids are Mathlib Matroid (Fin n) with ground set univ and eRank = r for both. Oracles are functions Bool → (Fin n → Bool) → Bool, required to answer independence queries exactly.
  • The algorithm is one fixed Program k s for a register machine with k+8k+8k+8 binary stacks: push/pop, fair coin, oracle query on an nnn-bit register, and halt with output (register 6)/(register 7) as a rational. Inputs n,r,ϵ,δn,r,\epsilon,\deltan,r,ϵ,δ are loaded in binary.
  • The resource bound is B=C(1+n+ℓ+⌈ϵ−1⌉+⌈log⁡2⌈δ−1⌉⌉)dB=C(1+n+\ell+\lceil\epsilon^{-1}\rceil+\lceil\log_2\lceil\delta^{-1}\rceil\rceil)^dB=C(1+n+ℓ+⌈ϵ−1⌉+⌈log2​⌈δ−1⌉⌉)d with constants C>0,dC>0,dC>0,d fixed before the instance. For every infinite bit stream, the run halts within BBB steps with at most BBB oracle calls and BBB bit operations, outputs a nonnegative rational, and outputs 000 when Z=0Z=0Z=0.
  • The success probability is expressed by counting: at least (1−δ)2B(1-\delta)2^B(1−δ)2B of the 2B2^B2B bit strings of length BBB lead to an output within relative error ϵ\epsilonϵ of ZZZ.
  • A trivializing reading is excluded: the program and constants come first and must work for every n,rn,rn,r, matroid pair, oracle and ϵ,δ\epsilon,\deltaϵ,δ.
  • Needed infrastructure: Lorentzian polynomials and the homogeneous Tutte inequalities, Markov chains on finite state spaces with Poincaré/variance bounds, annealing estimators, and correctness proofs for the register machine. Contributions on any layer are welcome.

Selected references

  • J. Edmonds, Matroid intersection, Ann. Discrete Math. 4 (1979), 39–49. https://doi.org/10.1016/S0167-5060(08)70817-3
  • T. Feder and M. Mihail, Balanced matroids, STOC 1992. https://doi.org/10.1145/129712.129716
  • M. Jerrum, A. Sinclair and E. Vigoda, A polynomial-time approximation algorithm for the permanent of a matrix with nonnegative entries, J. ACM 51 (2004), 671–697. https://doi.org/10.1145/1008731.1008738
  • A. Barvinok and A. Samorodnitsky, Random weighting, asymptotic counting, and inverse isoperimetry, Israel J. Math. 158 (2007). https://doi.org/10.1007/s11856-007-0008-8
  • P. Brändén and J. Huh, Lorentzian polynomials, Ann. of Math. 192 (2020), 821–891. https://doi.org/10.4007/annals.2020.192.3.4
  • N. Anari, S. Oveis Gharan and C. Vinzant, Log-concave polynomials I, Duke Math. J. 170 (2021). https://doi.org/10.1215/00127094-2020-0091
  • M. Cryan, H. Guo and G. Mousa, Modified log-Sobolev inequalities for strongly log-concave distributions, Ann. Probab. 49 (2021). https://doi.org/10.1214/20-AOP1453
  • N. Anari, K. Liu, S. Oveis Gharan and C. Vinzant, Log-concave polynomials II: high-dimensional walks and an FPRAS for counting bases of a matroid, Ann. of Math. 199 (2024). https://doi.org/10.4007/annals.2024.199.1.4
  • K. Liu, Spectral Independence: A New Tool to Analyze Markov Chains, PhD thesis, University of Washington, 2023. https://kuikuiliu.github.io/files/dissertation.pdf
  • OpenAI, Approximate counting of common bases of two matroids, OpenAI Math Release preprint, September 23, 2026 (Theorem 1.1, p. 2). https://github.com/openai/math/blob/adc7f1241b42e322a6451854ab7e4b4c146bf78a/preprints/Approximate-counting-of-common-bases-of-two-matroids-September-23-2026/main.pdf
2 thms1 active userReviewed
CombinatoricsTheoretical Computer Science·Captain: wurtle

A Fully Polynomial Randomized Approximation Scheme for Perfect Matchings in General GraphsResearch Paper

Motivation: approximately counting perfect matchings

A perfect matching of a graph is a set of edges covering every vertex exactly once. Finding one is easy — Edmonds (1965) gave a polynomial-time algorithm for general graphs — but counting them is hard: Valiant (1979) proved exact counting #P-complete already for bipartite graphs, where the count is the permanent of a 0–1 matrix. Exact counting is tractable only in special classes, such as planar graphs via Pfaffians (Kasteleyn 1963). The natural relaxation is a fully polynomial randomized approximation scheme (FPRAS): an algorithm that, with probability at least 1−δ1-\delta1−δ, returns the count within relative error ε\varepsilonε, in time polynomial in the input size, 1/ε1/\varepsilon1/ε and log⁡(1/δ)\log(1/\delta)log(1/δ). Whether perfect matchings in general (nonbipartite) graphs admit an FPRAS was explicitly raised by Jerrum and Sinclair in 1989 and has been a benchmark question in approximate counting since. Perfect-matching counts are partition functions of the monomer–dimer model, so the question also connects to statistical physics.

Timeline

  • 1963 — Kasteleyn: exact counting of perfect matchings in planar graphs (J. Math. Phys. 1963).
  • 1965 — Edmonds: polynomial-time maximum matching in general graphs (Canad. J. Math. 1965).
  • 1979 — Valiant: computing the permanent is #P-complete (TCS 1979).
  • 1986 — Jerrum, Valiant and Vazirani: equivalence of approximate counting and almost-uniform sampling for self-reducible problems (TCS 1986).
  • 1989 — Jerrum and Sinclair: FPRAS for the number of all matchings in any graph, and for perfect matchings when the near-perfect/perfect ratio is polynomially bounded; they raise the general question (SIAM J. Comput. 1989, Section 7(i)).
  • 2004 — Jerrum, Sinclair and Vigoda: FPRAS for the permanent of every nonnegative matrix, i.e. bipartite perfect matchings (J. ACM 2004); accelerated by Bezáková, Štefankovič, Vazirani and Vigoda (SIAM J. Comput. 2008).
  • 2018 — Štefankovič, Vigoda and Wilmes: graphs on which every chain of Jerrum–Sinclair–Vigoda type either gives perfect matchings exponentially small mass or mixes exponentially slowly (LATIN 2018).
  • 2020–2025 — Cai and Liu relate the problem to the eight-vertex model (ICALP 2020); Fei, Goldberg and Lu locate it among two-state spin systems (Inf. Comput. 2025).
  • 2026 — Further permanent speedups by Chen, Vigoda and Yang (arXiv:2608.26599) and Chen, Guo, Vigoda and Yang (arXiv:2609.20717); deterministic hafnian approximation for dense graphs by Yi (arXiv:2609.04079). An OpenAI preprint, A Fully Polynomial Randomized Approximation Scheme for Perfect Matchings in General Graphs (OpenAI Math Release, September 23, 2026), claims an FPRAS for every finite simple graph. The preprint has not been peer reviewed, and its main theorem is not formally verified.

Setting

A finite simple undirected graph GGG on vertices {0,…,n−1}\{0,\dots,n-1\}{0,…,n−1} is given by a set of edges, each written once as a pair (u,v)(u,v)(u,v) with u<vu<vu<v. A set M⊆EM\subseteq EM⊆E is a perfect matching if every vertex lies in exactly one edge of MMM, and

Z(G)=#{M⊆E:M is a perfect matching of G}.Z(G)=\#\{M\subseteq E: M\text{ is a perfect matching of }G\}.Z(G)=#{M⊆E:M is a perfect matching of G}.

(The graph with no vertices has Z=1Z=1Z=1; any graph with an odd number of vertices has Z=0Z=0Z=0.)

A randomized algorithm is modelled as a single fixed machine with a finite transition table over a finite tape alphabet, which at each step reads the current tape symbol and one fresh fair random bit, then writes, moves, or halts. The input is the graph in binary (number of vertices, number of edges, the sorted edge list) followed by the rationals ε\varepsilonε and δ\deltaδ in reduced form; the output is a rational number written on the tape when the machine halts.

Formalization targets

Goal: Theorem 1.1 (FPRAS for perfect matchings)

There are a machine AAA and constants C>0C>0C>0, ddd such that for every graph GGG and rationals 0<ε<10<\varepsilon<10<ε<1, 0<δ<1/20<\delta<1/20<δ<1/2, with t=C (∣input∣+⌈ε−1⌉+⌈log⁡2⌈δ−1⌉⌉+1)dt=C\,(|\mathrm{input}|+\lceil\varepsilon^{-1}\rceil+\lceil\log_2\lceil\delta^{-1}\rceil\rceil+1)^dt=C(∣input∣+⌈ε−1⌉+⌈log2​⌈δ−1⌉⌉+1)d:

  • on every random string of length ttt, AAA halts within ttt steps with a nonnegative rational output Z^\widehat ZZ;
  • if Z(G)=0Z(G)=0Z(G)=0, the output is 000 on every random string;
Pr⁡bits[(1−ε)Z(G)≤Z^≤(1+ε)Z(G)]  ≥  1−δ.\Pr_{\text{bits}}\big[(1-\varepsilon)Z(G)\le\widehat Z\le(1+\varepsilon)Z(G)\big]\;\ge\;1-\delta.bitsPr​[(1−ε)Z(G)≤Z≤(1+ε)Z(G)]≥1−δ.

The goal statement is published on the platform with status Open: no machine-checked proof exists yet.

Significance

The result itself. The theorem answers the Jerrum–Sinclair question affirmatively: perfect matchings in arbitrary graphs can be approximately counted as efficiently, up to polynomial factors, as in the bipartite case. The worst-case time bound holds for every execution, not just in expectation. Via standard reductions, the source also derives counting schemes for edge subsets with prescribed vertex degrees and for matchings of a fixed size (Corollary 10.1). The accompanying samplers for those families always return a feasible object and approximate the uniform law in total variation.

Formalizing it. The statement fixes a concrete machine model, input encoding and time bound, so a formal proof would certify both correctness of the estimator and the polynomial running time of an explicit algorithm — a level of rigor rarely reached for Markov-chain Monte Carlo results, whose analyses involve many interacting constants.

Difficulty

The natural approach extends the Jerrum–Sinclair–Vigoda Markov chain on perfect and near-perfect matchings, with weights learned while edge activities are gradually lowered. In general graphs this approach provably fails: Štefankovič, Vigoda and Wilmes constructed graphs on which every chain of that type, for any hole-dependent weights, has either exponentially small stationary mass on perfect matchings or exponentially slow mixing. Odd cycles (blossoms) are the reason the bipartite canonical-path arguments do not carry over. A solution needs a different state space and a different mixing analysis.

Formalization scope

  • GraphInput is a vertex count n and a Finset (Fin n × Fin n) of edges with strictly increasing endpoints, so loops and duplicate edges are excluded. Perfect G M requires M ⊆ edges and that each vertex is incident to exactly one edge of M; Z G is the number of such M.
  • The machine model is RandomMachine: a finite transition table on Fin 8 symbols, whose step consumes one random bit and performs a Turing.TM0 write or move, or halts. run folds tick over a list of t bits; halted configurations are absorbing.
  • encodeInput writes the vertex count, edge count and lexicographically sorted edges in binary, then encodeRat ε and encodeRat δ (reduced numerator with sign, denominator). Outputs requires that the machine has halted and the tape to the right of the head is exactly the encoding of the output rational.
  • The time bound is a fixed polynomial with natural ceilings; the success probability is the fraction of good tapes among all 2t2^t2t tapes.
  • The scheme must be uniform: one machine for all inputs, quantified before the graph. A trivializing reading is excluded because every tape must halt within the bound with a nonnegative output, and zero must be returned exactly when Z(G)=0Z(G)=0Z(G)=0.
  • Infrastructure needed: a compiler-style layer turning structured randomized algorithms into RandomMachine transitions, Markov-chain spectral-gap bounds, and sampling-to-counting reductions. The machine model and encodings are reusable for other approximate-counting missions.

Selected references

  • P. W. Kasteleyn, Dimer statistics and phase transitions, J. Math. Phys. 4 (1963), 287–293. https://doi.org/10.1063/1.1703953
  • J. Edmonds, Paths, trees, and flowers, Canad. J. Math. 17 (1965), 449–467. https://doi.org/10.4153/CJM-1965-045-4
  • L. G. Valiant, The complexity of computing the permanent, Theoret. Comput. Sci. 8 (1979), 189–201. https://doi.org/10.1016/0304-3975(79)90044-6
  • M. R. Jerrum, L. G. Valiant, V. V. Vazirani, Random generation of combinatorial structures from a uniform distribution, Theoret. Comput. Sci. 43 (1986), 169–188. https://doi.org/10.1016/0304-3975(86)90174-X
  • M. Jerrum, A. Sinclair, Approximating the permanent, SIAM J. Comput. 18 (1989), 1149–1178. https://doi.org/10.1137/0218077
  • M. Jerrum, A. Sinclair, E. Vigoda, A polynomial-time approximation algorithm for the permanent of a matrix with nonnegative entries, J. ACM 51 (2004), 671–697. https://doi.org/10.1145/1008731.1008738
  • I. Bezáková, D. Štefankovič, V. V. Vazirani, E. Vigoda, Accelerating simulated annealing for the permanent and combinatorial counting problems, SIAM J. Comput. 37 (2008), 1429–1454. https://doi.org/10.1137/050644033
  • D. Štefankovič, E. Vigoda, J. Wilmes, On counting perfect matchings in general graphs, LATIN 2018, LNCS 10807, 873–885. https://doi.org/10.1007/978-3-319-77404-6_63
  • J.-Y. Cai, T. Liu, Counting perfect matchings and the eight-vertex model, ICALP 2020, LIPIcs 168, 23:1–23:18. https://doi.org/10.4230/LIPIcs.ICALP.2020.23
  • Y. Fei, L. A. Goldberg, P. Lu, Two-state spin systems with negative interactions, Inf. Comput. 307 (2025), 105340. https://doi.org/10.1016/j.ic.2025.105340
  • OpenAI, A Fully Polynomial Randomized Approximation Scheme for Perfect Matchings in General Graphs, OpenAI Math Release preprint, September 23, 2026 (source of the goal; Theorem 1.1, p. 2). https://github.com/openai/math/blob/adc7f1241b42e322a6451854ab7e4b4c146bf78a/preprints/A-Fully-Polynomial-Randomized-Approximation-Scheme-for-Perfect-Matchings-in-General-Graphs-September-23-2026/main.pdf
2 thms1 active userReviewed
ProbabilityReinforcement LearningStochastic Systems·Captain: mikedeng1

On Actor-Critic Algorithms: Convergence of Actor-Critic Algorithms with TD(1) and TD(λ) Critics on Finite MDPsResearch Paper

Motivation

Reinforcement-learning methods for Markov decision processes split into critic-only methods, which learn an approximate value function and act greedily with respect to it, and actor-only methods, which adjust a parameterized policy along a simulated estimate of the gradient of its performance. Critic-only methods with function approximation can fail to converge; actor-only gradient estimates have high variance. Actor–critic methods combine the two: a critic learns, by temporal-difference (TD) learning, just enough about the value function to give the actor a low-variance gradient direction.

Konda and Tsitsiklis (SIAM J. Control Optim. 42 (2003)) gave the first convergence proof for actor–critic algorithms whose critic uses linear function approximation. Their key observation, found independently by Sutton, McAllester, Singh and Mansour (NIPS 1999), is that the critic need not approximate the value function: it suffices to compute its projection onto a low-dimensional subspace spanned by features determined by the policy parameterization. Earlier convergence results (Konda and Borkar, 1999) covered lookup-table representations only. Actor–critic algorithms, in many later variants, are now a standard tool of reinforcement learning, and this paper is the reference for their asymptotic analysis in the average-cost setting.

Setting

A finite cost MDP consists of finite sets X\mathbb XX (states) and U\mathbb UU (actions), transition probabilities p(y∣x,u)p(y\mid x,u)p(y∣x,u) and a one-stage cost c(x,u)∈Rc(x,u)\in\mathbb Rc(x,u)∈R. A randomized stationary policy with parameter θ∈Rn\theta\in\mathbb R^nθ∈Rn chooses action uuu in state xxx with probability μθ(u∣x)\mu_\theta(u\mid x)μθ​(u∣x). Under the policy θ\thetaθ, the states {Xk}\{X_k\}{Xk​} form a Markov chain with transition matrix P(θ)xy=∑uμθ(u∣x)p(y∣x,u)P(\theta)_{xy}=\sum_u\mu_\theta(u\mid x)p(y\mid x,u)P(θ)xy​=∑u​μθ​(u∣x)p(y∣x,u), and the pairs {Xk,Uk}\{X_k,U_k\}{Xk​,Uk​} a chain with matrix PθP_\thetaPθ​. When these chains are irreducible and aperiodic, πθ\pi_\thetaπθ​ denotes the stationary law of {Xk}\{X_k\}{Xk​}, ηθ(x,u)=πθ(x)μθ(u∣x)\eta_\theta(x,u)=\pi_\theta(x)\mu_\theta(u\mid x)ηθ​(x,u)=πθ​(x)μθ​(u∣x), and the average cost is

αˉ(θ)=∑x,uc(x,u) ηθ(x,u).\bar\alpha(\theta)=\sum_{x,u}c(x,u)\,\eta_\theta(x,u).αˉ(θ)=x,u∑​c(x,u)ηθ​(x,u).

The score is ψθ(x,u)=∇ln⁡μθ(u∣x)∈Rn\psi_\theta(x,u)=\nabla\ln\mu_\theta(u\mid x)\in\mathbb R^nψθ​(x,u)=∇lnμθ​(u∣x)∈Rn, and ⟨f,g⟩θ=∑x,uηθ(x,u)f(x,u)g(x,u)\langle f,g\rangle_\theta=\sum_{x,u}\eta_\theta(x,u)f(x,u)g(x,u)⟨f,g⟩θ​=∑x,u​ηθ​(x,u)f(x,u)g(x,u).

The critic approximates Q-values by r′ϕθ(x,u)r'\phi_\theta(x,u)r′ϕθ​(x,u) with features ϕθ(x,u)∈Rm\phi_\theta(x,u)\in\mathbb R^mϕθ​(x,u)∈Rm whose span contains the components of ψθ\psi_\thetaψθ​. Along one simulated trajectory (X^k,U^k)(\hat X_k,\hat U_k)(X^k​,U^k​), where U^k+1\hat U_{k+1}U^k+1​ is drawn from μθk(⋅∣X^k+1)\mu_{\theta_k}(\cdot\mid\hat X_{k+1})μθk​​(⋅∣X^k+1​), the algorithm updates an average-cost estimate αk\alpha_kαk​, the critic parameter rkr_krk​ and an eligibility trace Z^k\hat Z_kZ^k​ at step size γk\gamma_kγk​ (equation (3.1)), and the actor parameter by

θk+1=θk−βkΓ(rk) rk′ϕθk(X^k+1,U^k+1) ψθk(X^k+1,U^k+1)(3.2)\theta_{k+1}=\theta_k-\beta_k\Gamma(r_k)\,r_k'\phi_{\theta_k}(\hat X_{k+1},\hat U_{k+1})\,\psi_{\theta_k}(\hat X_{k+1},\hat U_{k+1})\qquad(3.2)θk+1​=θk​−βk​Γ(rk​)rk′​ϕθk​​(X^k+1​,U^k+1​)ψθk​​(X^k+1​,U^k+1​)(3.2)

at a slower step size βk\beta_kβk​. The TD(1) critic resets its trace when the state x∗x^*x∗ is entered; the TD(λ\lambdaλ) critic discounts it by λ∈(0,1)\lambda\in(0,1)λ∈(0,1).

Formalization targets

Goal: Theorem 3.4

Under Assumptions 2.1 (positive, smooth policies; irreducible aperiodic chains; uniform reachability of x∗x^*x∗), 3.1–3.2 (bounded smooth features spanning the scores, uniformly nondegenerate) and 3.3 (two-time-scale step sizes and the scaling Γ\GammaΓ),

(a) TD(1):  lim inf⁡k∣∇αˉ(θk)∣=0  w.p.1;(b) ∀ϵ>0 ∃λˉ<1 ∀λ∈[λˉ,1): lim inf⁡k∣∇αˉ(θk)∣<ϵ  w.p.1.\text{(a) TD(1): }\ \liminf_k|\nabla\bar\alpha(\theta_k)|=0\ \text{ w.p.1};\qquad \text{(b) }\forall\epsilon>0\ \exists\bar\lambda<1\ \forall\lambda\in[\bar\lambda,1):\ \liminf_k|\nabla\bar\alpha(\theta_k)|<\epsilon\ \text{ w.p.1}.(a) TD(1):  kliminf​∣∇αˉ(θk​)∣=0  w.p.1;(b) ∀ϵ>0 ∃λˉ<1 ∀λ∈[λˉ,1): kliminf​∣∇αˉ(θk​)∣<ϵ  w.p.1.

The statement leaves the initial parameters and the initial state–action law arbitrary.

Milestones

In attack order: the gradient formula ∇αˉ(θ)=⟨ψθ,Qθ⟩θ\nabla\bar\alpha(\theta)=\langle\psi_\theta,Q_\theta\rangle_\theta∇αˉ(θ)=⟨ψθ​,Qθ​⟩θ​ for every Poisson solution, with bounded Hessian (Theorem 4.6); boundedness of the regenerative quantities (Lemma 5.1) and of the trace moments (Lemma 5.2); uniform positive definiteness of the critic's mean matrix Gˉ(θ)\bar G(\theta)Gˉ(θ) for TD(1) and TD(λ\lambdaλ) (Lemmas 5.3, 5.6); critic tracking, ∣Gˉ(θk)Rk−hˉ(θk)∣→0|\bar G(\theta_k)R_k-\bar h(\theta_k)|\to0∣Gˉ(θk​)Rk​−hˉ(θk​)∣→0 (Theorem 5.7); the bias of the actor direction, sup⁡θ∣f(θ)−∇αˉ(θ)∣≤C(1−λ)\sup_\theta|f(\theta)-\nabla\bar\alpha(\theta)|\le C(1-\lambda)supθ​∣f(θ)−∇αˉ(θ)∣≤C(1−λ) (Lemma 6.1); the descent inequality (6.1); and the convergence of the noise terms (Lemma 6.2).

Significance

The theorem says that a policy-gradient method whose gradient is estimated by a linearly parameterized TD critic, rather than by Monte Carlo returns, still converges to a stationary point of the average cost (exactly for TD(1), approximately for TD(λ\lambdaλ)). It justifies the compatible features principle (the critic's features must span the score functions), the two-time-scale step-size design, and the trade-off in λ\lambdaλ between variance and bias of the actor direction. The intermediate results are of independent use: Theorem 4.6 is the average-cost policy-gradient theorem for arbitrary Poisson solutions, and Theorem 5.7 is a tracking result for linear stochastic approximation driven by a Markov chain whose kernel changes slowly.

The result is proved in the paper, but the proofs of Theorem 4.6, of parts of the verification of the stochastic-approximation hypotheses and of Theorem 6.3 are given as outlines, with details deferred to a separate appendix and to Theorem A.7, recalled from another paper. No machine-checked proof of the theorem, or of the convergence of any actor–critic method, is known. The mission asks for a formal proof of the finite-case theorem and its milestones, which fills in the outlined steps. The finite-case gradient formula of §2 (Theorem 2.2, recalled from Marbach and Tsitsiklis) is posed, unproved, elsewhere on the platform in a reward-MDP form; it is a special case of milestone Theorem 4.6.

Difficulty

The obvious argument treats the actor as a stochastic gradient method with a noisy gradient estimate. It fails at the first step: the critic's estimate rk′ϕθkr_k'\phi_{\theta_k}rk′​ϕθk​​ is biased at every finite time, and the bias does not vanish unless the critic converges, while the critic's target rˉ(θk)\bar r(\theta_k)rˉ(θk​) moves with the actor. The noise is Markovian, generated by a chain whose transition kernel depends on the current parameter, not a martingale difference sequence. Making the two time scales interact correctly requires a Poisson-equation decomposition of the Markov noise that is uniform in θ\thetaθ, together with moment bounds for the eligibility trace, which for TD(1) is an unbounded sum over a regeneration cycle. Uniformity in θ\thetaθ is essential, because the actor iterates θk\theta_kθk​ are not assumed to stay in a bounded set.

Formalization scope

State and action sets are finite types; parameters live in EuclideanSpace ℝ (Fin n) and EuclideanSpace ℝ (Fin m), so ∣⋅∣|\cdot|∣⋅∣ is the Euclidean norm and Mathlib's gradient is ∇\nabla∇. The transition probability p(y∣x,u)p(y\mid x,u)p(y∣x,u) is stored as p x u y. πθ\pi_\thetaπθ​ is a stationary probability vector of P(θ)P(\theta)P(θ), unique under Assumption 2.1(c). The law of the simulated path is built on (X×U)N(\mathbb X\times\mathbb U)^{\mathbb N}(X×U)N by the Ionescu-Tulcea construction (Kernel.trajMeasure), with the next action drawn under the actor parameter that the recursion computes from the history; nothing about the critic or the actor's behaviour is assumed.

Readings and corrections, each also recorded in the affected items:

  • Assumption 3.3(b) is corrected. The printed ∣r∣Γ(r)∈[C1,C2]|r|\Gamma(r)\in[C_1,C_2]∣r∣Γ(r)∈[C1​,C2​] for all rrr is unsatisfiable at r=0r=0r=0 and would make the theorem vacuous; it is formalized as (1+∣r∣)Γ(r)∈[C1,C2](1+|r|)\Gamma(r)\in[C_1,C_2](1+∣r∣)Γ(r)∈[C1​,C2​], which Γ(r)=1/(1+∣r∣)\Gamma(r)=1/(1+|r|)Γ(r)=1/(1+∣r∣) satisfies. The second inequality of (3.3) is quantified over Rm\mathbb R^mRm, the domain of Γ\GammaΓ, not the printed Rn\mathbb R^nRn.
  • Assumption 4.9 (the features have zero mean under μθ(⋅∣x∗)\mu_\theta(\cdot\mid x^*)μθ​(⋅∣x∗)) is added to part (a), and to the TD(1) cases of Theorem 5.7 and Lemma 6.2: the TD(1) analysis uses it and Theorem 6.3 assumes it, though Theorem 3.4 does not list it.
  • "λ\lambdaλ sufficiently close to 1" means: some λˉ<1\bar\lambda<1λˉ<1 such that all λ∈[λˉ,1)\lambda\in[\bar\lambda,1)λ∈[λˉ,1) work.
  • The α\alphaα-update of (3.1) uses c(X^k+1,U^k+1)c(\hat X_{k+1},\hat U_{k+1})c(X^k+1​,U^k+1​) as printed.
  • In (6.1), Γ(rˉ(θ))\Gamma(\bar r(\theta))Γ(rˉ(θ)) is read as Γ(rˉ(θk))\Gamma(\bar r(\theta_k))Γ(rˉ(θk​)).
  • The matrix ⟨Pθk+1ϕθ,ϕθ′⟩θ\langle P_\theta^{k+1}\phi_\theta,\phi_\theta'\rangle_\theta⟨Pθk+1​ϕθ​,ϕθ′​⟩θ​ in the TD(λ\lambdaλ) Gˉ1\bar G_1Gˉ1​ is oriented so that Gˉ1\bar G_1Gˉ1​ is the steady-state mean of the critic's update matrix.
  • Results stated in the paper for Polish spaces (Theorems 4.6, 5.7) are stated under the finite assumptions that the paper says imply them; "w.p.1" in Theorem 5.7 is made explicit.

The lim inf⁡\liminfliminf is taken in [0,∞][0,\infty][0,∞], so a sequence with ∣∇αˉ(θk)∣→∞|\nabla\bar\alpha(\theta_k)|\to\infty∣∇αˉ(θk​)∣→∞ cannot satisfy the conclusion through a junk value, and differentiability of αˉ\bar\alphaαˉ is part of the conclusion, so the statement cannot hold because Lean's gradient returns 000 at non-differentiable points. Series (tsum) and matrix inverses in the milestone objects converge, respectively are invertible, under the assumptions. The hypotheses are satisfiable: Γ(r)=1/(1+∣r∣)\Gamma(r)=1/(1+|r|)Γ(r)=1/(1+∣r∣), βk=(k+1)−1\beta_k=(k+1)^{-1}βk​=(k+1)−1, γk=(k+1)−2/3\gamma_k=(k+1)^{-2/3}γk​=(k+1)−2/3 and the features ϕθ=ψθ\phi_\theta=\psi_\thetaϕθ​=ψθ​ (which satisfy Assumption 4.9 at every state) meet the parts that could be in doubt.

A complete development needs finite Markov-chain theory uniform in a parameter (existence, uniqueness and smooth dependence of stationary laws; geometric ergodicity; Poisson equations; regeneration), differentiation of θ↦πθ\theta\mapsto\pi_\thetaθ↦πθ​, almost-sure convergence of linear stochastic approximation with Markov noise, and martingale convergence. The Markov-chain and stochastic-approximation parts are reusable well beyond this mission. Contributions of proofs of any milestone, of auxiliary lemmas on parameterized finite Markov chains, and of the general two-time-scale tracking theorem (the paper's Theorem A.7) are welcome.

Selected references

  • V. R. Konda and J. N. Tsitsiklis, On Actor-Critic Algorithms, SIAM J. Control Optim. 42(4), 1143–1166, 2003. https://doi.org/10.1137/S0363012901385691
  • R. S. Sutton, D. McAllester, S. Singh and Y. Mansour, Policy Gradient Methods for Reinforcement Learning with Function Approximation, NIPS 12, 2000. https://papers.nips.cc/paper/1713-policy-gradient-methods-for-reinforcement-learning-with-function-approximation
  • V. R. Konda and V. S. Borkar, Actor-Critic–Type Learning Algorithms for Markov Decision Processes, SIAM J. Control Optim. 38(1), 94–123, 1999. https://doi.org/10.1137/S036301299731669X
  • J. N. Tsitsiklis and B. Van Roy, Average Cost Temporal-Difference Learning, Automatica 35(11), 1799–1808, 1999. https://doi.org/10.1016/S0005-1098(99)00099-0
  • P. Marbach and J. N. Tsitsiklis, Simulation-Based Optimization of Markov Reward Processes, IEEE Trans. Automat. Control 46(2), 191–209, 2001. https://doi.org/10.1109/9.905687
14 thms1 active userReviewed
Algorithmic Game TheoryProbability·Captain: mikedeng1

The Evolution of Conventions III: In 2 × 2 Games with Two Strict Equilibria, the Generically Stable Equilibria of Adaptive Play Are the Weakly Risk Dominant OnesResearch Paper

Motivation

Many games have several strict Nash equilibria, and equilibrium analysis alone does not say which one a population will settle on. Harsanyi and Selten (1988) proposed risk dominance as a selection criterion for 2×22\times22×2 games: of two strict equilibria, select the one whose basin is "larger" in a precise payoff sense. That criterion is axiomatic. H. Peyton Young's The Evolution of Conventions (Econometrica, 1993) derives a selection from a dynamic model instead: boundedly rational players with finite memory, incomplete information about the past and occasional mistakes. Its Theorem 3 says that, in 2×22\times22×2 games, the dynamic selection and risk dominance coincide.

Timeline.

  • 1984. Freidlin and Wentzell develop the theory of small random perturbations of dynamical systems, including the tree characterization of limiting stationary measures.
  • 1990. Foster and Young introduce stochastic stability for evolutionary processes with vanishing noise.
  • 1993. Kandori, Mailath and Rob prove that in symmetric 2×22\times22×2 games with a single population, the long-run equilibrium of a mutation process is the risk dominant one. Young (1993) proves the same selection for adaptive play, a two-population model with sampling from a finite memory, for every 2×22\times22×2 game with two strict equilibria, not only symmetric ones. Its Example 3 shows that in 3×33\times33×3 games the selected equilibrium can differ from both the risk dominant and the Pareto optimal one.

Setting

A finite game Γ\GammaΓ has players iii with finite strategy sets SiS_iSi​ and payoffs ui(s)u_i(s)ui​(s). Fix a memory mmm and a sample size kkk with 1≤k≤m1\le k\le m1≤k≤m. A state is the sequence hhh of the last mmm plays (strategy-tuples), oldest first; a successor of hhh drops the oldest play and appends a new one. In each period each player draws a sample of kkk of the mmm recorded plays and chooses a best reply to it (maximizing total payoff against the sampled plays). The choice probabilities pi(s∣h)p_i(s\mid h)pi​(s∣h) form a best-reply distribution: pi(s∣h)>0p_i(s\mid h)>0pi​(s∣h)>0 exactly when sss is a best reply to some sample of size kkk. This gives adaptive play P0P^0P0, display (1).

With mistakes, each player iii independently experiments with probability ελi\varepsilon\lambda_iελi​ and then plays sss with probability qi(s∣h)>0q_i(s\mid h)>0qi​(s∣h)>0. This gives the perturbed chain PεP^\varepsilonPε of display (2). It has a unique stationary distribution με\mu^\varepsilonμε. A state hhh is stochastically stable if lim⁡ε→0μhε>0\lim_{\varepsilon\to0}\mu^\varepsilon_h>0limε→0​μhε​>0. A convention is mmm repetitions of a strict pure Nash equilibrium. With LΓL_\GammaLΓ​ the longest shortest best-reply path to a strict equilibrium, a strict equilibrium is generically stable if its convention is stochastically stable for all sufficiently large kkk and all mmm with k≤m/(LΓ+2)k\le m/(L_\Gamma+2)k≤m/(LΓ​+2).

For a 2×22\times22×2 game in normal form, with Row's payoffs arca_{rc}arc​ and Column's brcb_{rc}brc​ satisfying a11>a21a_{11}>a_{21}a11​>a21​, b11>b12b_{11}>b_{12}b11​>b12​, a22>a12a_{22}>a_{12}a22​>a12​, b22>b21b_{22}>b_{21}b22​>b21​, set

R1=min⁡{a11−a21a11−a12−a21+a22,b11−b12b11−b12−b21+b22},R2=min⁡{a22−a12a11−a12−a21+a22,b22−b21b11−b12−b21+b22}.R_1=\min\Big\{\tfrac{a_{11}-a_{21}}{a_{11}-a_{12}-a_{21}+a_{22}},\tfrac{b_{11}-b_{12}}{b_{11}-b_{12}-b_{21}+b_{22}}\Big\},\qquad R_2=\min\Big\{\tfrac{a_{22}-a_{12}}{a_{11}-a_{12}-a_{21}+a_{22}},\tfrac{b_{22}-b_{21}}{b_{11}-b_{12}-b_{21}+b_{22}}\Big\}.R1​=min{a11​−a12​−a21​+a22​a11​−a21​​,b11​−b12​−b21​+b22​b11​−b12​​},R2​=min{a11​−a12​−a21​+a22​a22​−a12​​,b11​−b12​−b21​+b22​b22​−b21​​}.

(1,1)(1,1)(1,1) weakly risk dominates (2,2)(2,2)(2,2) if R1≥R2R_1\ge R_2R1​≥R2​.

Formalization targets

Goal: Theorem 3 (p. 72)

(1,1) generically stable  ⟺  R1≥R2,(2,2) generically stable  ⟺  R2≥R1.(1,1)\text{ generically stable}\iff R_1\ge R_2,\qquad (2,2)\text{ generically stable}\iff R_2\ge R_1.(1,1) generically stable⟺R1​≥R2​,(2,2) generically stable⟺R2​≥R1​.

This holds for every 2×22\times22×2 game in normal form, for every best-reply distribution and every admissible experimentation.

Milestones, in the order the proof uses them

  1. PεP^\varepsilonPε is irreducible and aperiodic and has a unique stationary distribution (§6, pp. 67–68).
  2. (2) is a regular perturbation of (1): Pε→P0P^\varepsilon\to P^0Pε→P0, and Phh′εP^\varepsilon_{hh'}Phh′ε​ is of exact order εr(h,h′)\varepsilon^{r(h,h')}εr(h,h′), with r(h,h′)r(h,h')r(h,h′) the number of mistakes (Appendix, p. 77).
  3. Theorem 2 (p. 69): the stochastically stable states are those in the recurrent classes of P0P^0P0 of minimum stochastic potential, independently of λ\lambdaλ and qqq.
  4. Corollary (p. 70): for weakly acyclic games with k≤m/(LΓ+2)k\le m/(L_\Gamma+2)k≤m/(LΓ​+2), the stable states are the conventions of minimum stochastic potential.
  5. The 2×22\times22×2 case is acyclic with LΓ=1L_\Gamma=1LΓ​=1, and for k≤m/3k\le m/3k≤m/3 the recurrent classes of P0P^0P0 are {h1}\{h_1\}{h1​} and {h2}\{h_2\}{h2​} (p. 70).
  6. r12=⌈R1k⌉r_{12}=\lceil R_1k\rceilr12​=⌈R1​k⌉ and 7. r21=⌈R2k⌉r_{21}=\lceil R_2k\rceilr21​=⌈R2​k⌉ (pp. 71–72).
  7. If R1>R2R_1>R_2R1​>R2​, then h1h_1h1​ is the unique stable convention, and if R1=R2R_1=R_2R1​=R2​, both are, for all large kkk and m≥3km\ge3km≥3k (p. 72).

Significance

Theorem 3 gives a dynamic foundation for risk dominance: an equilibrium selection criterion defined from payoffs alone is the long-run outcome of a decentralized learning process with sampling and rare mistakes. It requires no symmetry between the two player roles. Theorem 2 makes stochastic stability computable: a limit of stationary distributions is reduced to a minimum-weight arborescence over recurrent classes. That reduction is what the 2×22\times22×2 computation, and later applications of the method to bargaining and contracting, rely on. The equivalence of the RRR-form with the product form (a11−a21)(b11−b12)≥(a22−a12)(b22−b21)(a_{11}-a_{21})(b_{11}-b_{12})\ge(a_{22}-a_{12})(b_{22}-b_{21})(a11​−a21​)(b11​−b12​)≥(a22​−a12​)(b22​−b21​) of Harsanyi and Selten is elementary algebra and is not a target here.

The results are proved in the paper. To our knowledge none of them has a machine-checked proof. The mission produces formal statements of the model (adaptive play, its perturbation, stochastic stability, resistance and stochastic potential) that are reusable for other stochastic-stability results. Closing the goal also requires formalizing Young's finite version of the Freidlin–Wentzell theorem, the subject of the companion mission The Evolution of Conventions II.

Difficulty

The paper's argument has two hard parts. The first is Theorem 2: identifying the limit of με\mu^\varepsilonμε requires the Markov chain tree theorem (the stationary distribution of an irreducible chain as a ratio of sums over spanning in-trees) together with an order-of-magnitude analysis in ε\varepsilonε. The second is the exact resistance ⌈R1k⌉\lceil R_1k\rceil⌈R1​k⌉. The page constructs a path with that many mistakes (Table 1), but the matching lower bound, that no combination of mistakes by both players does better, is only asserted. It needs an argument about the first period in which some player's best reply switches. A naive count of single-player mistake runs gives only the upper bound. The tie case of (5) also matters: equality makes strategy 2 a best reply, which is why the ceiling and not the floor appears.

Formalization scope

Players are a finite type and strategies finite types. A state is Fin m → (∀ i, S i) with position 0 the oldest play. A sample is a set of kkk positions, and a best reply maximizes the summed payoff against the sampled plays. Transition matrices are Matrix H H ℝ. Stochastic stability is a one-sided limit ε→0+\varepsilon\to0^+ε→0+ of the stationary distributions of PεP^\varepsilonPε, over the admissible range 0<ε0<\varepsilon0<ε, ελi≤1\varepsilon\lambda_i\le1ελi​≤1. Resistances are natural numbers on successor pairs. Recurrent classes are the essential classes of P0P^0P0, and iii-trees are parent maps on the set of classes that reach the root. "k≤m/(LΓ+2)k\le m/(L_\Gamma+2)k≤m/(LΓ​+2)" is the natural-number inequality k(LΓ+2)≤mk(L_\Gamma+2)\le mk(LΓ​+2)≤m. In the 2×22\times22×2 game, players and strategies are Fin 2, with the paper's 1,21,21,2 written as 0, 1.

Two corrected slips are recorded in the statements. On p. 68 the recurrent classes are "of PεP^\varepsilonPε" for P0P^0P0, and on p. 77 "the process PεP^\varepsilonPε defined in (1)" is P0P^0P0.

Trivializing formalizations are ruled out. Stochastic stability is defined from stationary distributions, not from resistances, and generic stability is defined from stochastic stability, not from ⌈R1k⌉\lceil R_1k\rceil⌈R1​k⌉ versus ⌈R2k⌉\lceil R_2k\rceil⌈R2​k⌉. Both definitions quantify over every best-reply distribution and every admissible (λ,q)(\lambda,q)(λ,q) rather than a fixed sampling law. "Sufficiently large kkk" is an eventual statement with a threshold, not "for some kkk".

A complete development needs Markov chain stationary distributions on finite spaces, the Markov chain tree theorem, limits of rational functions of ε\varepsilonε, and the combinatorics of sampling in 2×22\times22×2 games. The perturbation theory (milestones 1–3) is reusable for any finite stochastic-stability model. Contributions to the tree theorem and to the two resistance computations are particularly welcome.

Related platform work: stochastic fictitious play (StochFictPlay.*, Hofbauer–Sandholm) and calibrated learning to correlated equilibrium (CalibratedCE.*, Foster–Vohra) study other learning dynamics in games. The sibling missions The Evolution of Conventions I (Theorem 1, convergence of adaptive play) and II (Theorem 4, regular perturbations of Markov chains) formalize the results this mission's milestones invoke.

Selected references

  • H. P. Young, The Evolution of Conventions, Econometrica 61(1), 1993, 57–84. https://doi.org/10.2307/2951778
  • J. C. Harsanyi and R. Selten, A General Theory of Equilibrium Selection in Games, MIT Press, 1988.
  • M. Kandori, G. J. Mailath and R. Rob, Learning, Mutation, and Long Run Equilibria in Games, Econometrica 61(1), 1993, 29–56. https://doi.org/10.2307/2951777
  • D. Foster and H. P. Young, Stochastic Evolutionary Game Dynamics, Theoretical Population Biology 38, 1990, 219–232. https://doi.org/10.1016/0040-5809(90)90011-J
  • M. I. Freidlin and A. D. Wentzell, Random Perturbations of Dynamical Systems, Springer, 1984. https://doi.org/10.1007/978-1-4684-0176-9
45 thms1 active userReviewed
Dynamic ProgrammingOperations Research·Captain: mikedeng1

Discrete Dynamic Programming with Sensitive Discount Optimality Criteria 2: No Improvement of Order n+1 Implies n-Discount Optimality, Which Rules Out Improvement of Order nResearch Paper

Motivation

A finite Markov decision process is usually solved under one of two criteria: the expected total discounted reward, for a fixed interest rate, or the long-run average reward. Neither is satisfactory when the interest rate is small and not known precisely. Blackwell (Blackwell 1962) showed that some stationary policy is optimal for every discount factor close enough to one, and that the average-reward criterion cannot tell apart policies with the same gain but very different transient rewards. Veinott's 1969 paper interpolates between these two: it introduces a hierarchy of nnn-discount optimality criteria, indexed by n=−1,0,1,…n=-1,0,1,\dotsn=−1,0,1,…, each more selective than the previous, and characterizes the stationary policies that meet each level through the Laurent expansion of the discounted return in the interest rate.

Timeline. Blackwell (1962) introduced the expansion of the discounted return near β=1\beta=1β=1 to first order and the criteria now called −1+-1^+−1+ and ∞+\infty^+∞+ discount optimality. Veinott (1966) gave a policy improvement method for the bias (0+0^+0+) criterion. Miller and Veinott (1969) gave the full Laurent expansion and a policy improvement algorithm for Blackwell optimality in the stochastic case. The present paper (1969) extends this to substochastic transition matrices and to negative interest rates in the transient case, defines the n±n^\pmn± criteria, and proves Theorem 4, the subject of this mission.

Setting

There are finitely many states sss (the paper's 1,…,S1,\dots,S1,…,S); in each state a finite nonempty action set AsA_sAs​. Action aaa in state sss earns r(s,a)∈Rr(s,a)\in\mathbb Rr(s,a)∈R and leads to state ttt with weight p(t∣s,a)≥0p(t\mid s,a)\ge0p(t∣s,a)≥0, where ∑tp(t∣s,a)≤1\sum_t p(t\mid s,a)\le1∑t​p(t∣s,a)≤1 (the remaining mass is the probability of stopping). A decision rule f∈F=×sAsf\in F=\times_s A_sf∈F=×s​As​ picks one action per state; r(f)r(f)r(f) and P(f)P(f)P(f) are its reward vector and substochastic transition matrix, Q(f)=P(f)−IQ(f)=P(f)-IQ(f)=P(f)−I. A policy is a sequence π=(f1,f2,… )\pi=(f_1,f_2,\dots)π=(f1​,f2​,…) of decision rules, f∞=(f,f,… )f^\infty=(f,f,\dots)f∞=(f,f,…) is a stationary policy, and PN(π)=P(f1)⋯P(fN)P^N(\pi)=P(f_1)\cdots P(f_N)PN(π)=P(f1​)⋯P(fN​).

For an interest rate ρ>−1\rho>-1ρ>−1 and β=(1+ρ)−1\beta=(1+\rho)^{-1}β=(1+ρ)−1, the discounted return is

Vρ(π)=∑N=1∞βNPN−1(π) r(fN).V_\rho(\pi)=\sum_{N=1}^{\infty}\beta^N P^{N-1}(\pi)\,r(f_N).Vρ​(π)=N=1∑∞​βNPN−1(π)r(fN​).

For a substochastic PPP, P∗P^*P∗ is the Cesàro limit of its powers and H=(I−P+P∗)−1−P∗H=(I-P+P^*)^{-1}-P^*H=(I−P+P∗)−1−P∗ its deviation matrix.

A policy π∗\pi^*π∗ is n±n^\pmn± discount optimal if, componentwise,

lim inf⁡ρ→0±∣ρ∣−n [Vρ(π∗)−Vρ(π)]≥0for all policies π.\liminf_{\rho\to0\pm}|\rho|^{-n}\,[V_\rho(\pi^*)-V_\rho(\pi)]\ge0\quad\text{for all policies }\pi .ρ→0±liminf​∣ρ∣−n[Vρ​(π∗)−Vρ​(π)]≥0for all policies π.

Dn±D_n^\pmDn±​ is the set of fff with f∞f^\inftyf∞ n±n^\pmn± discount optimal, and D−2=FD_{-2}=FD−2​=F. The sign −-− (interest rate approaching zero from below) is considered only in the transient case, where ∑NP(f)N\sum_N P(f)^N∑N​P(f)N converges for every fff.

The Laurent coefficients of Vρ(f∞)V_\rho(f^\infty)Vρ​(f∞) are y−1±(f)=±P∗(f)r(f)y_{-1}^\pm(f)=\pm P^*(f)r(f)y−1±​(f)=±P∗(f)r(f) and yn±(f)=(∓1)nH(f)n+1r(f)y_n^\pm(f)=(\mp1)^nH(f)^{n+1}r(f)yn±​(f)=(∓1)nH(f)n+1r(f), n≥0n\ge0n≥0; with y−2±=0y_{-2}^\pm=0y−2±​=0 and r0=rr_0=rr0​=r, rn=0r_n=0rn​=0 otherwise, the test quantities are

ψn±(g,f)=rn(g)+Q(g) yn±(f)∓yn−1±(f),n≥−1.\psi_n^\pm(g,f)=r_n(g)+Q(g)\,y_n^\pm(f)\mp y_{n-1}^\pm(f),\qquad n\ge-1 .ψn±​(g,f)=rn​(g)+Q(g)yn±​(f)∓yn−1±​(f),n≥−1.

Ψn±(g,f)\Psi_n^\pm(g,f)Ψn±​(g,f) is the matrix with columns ψ−1±,…,ψn±\psi_{-1}^\pm,\dots,\psi_n^\pmψ−1±​,…,ψn±​, and Gn±(f)={g∈F:Ψn±(g,f)≻0}G_n^\pm(f)=\{g\in F:\Psi_n^\pm(g,f)\succ0\}Gn±​(f)={g∈F:Ψn±​(g,f)≻0}, where C≻0C\succ0C≻0 means that the first nonzero entry of every row of CCC is positive and C≠0C\ne0C=0.

Formalization targets

Goal: Theorem 4

For f∈Ff\in Ff∈F and n=−2,−1,…,S−1n=-2,-1,\dots,S-1n=−2,−1,…,S−1:

Gn+1±(f)=∅ ⟹ f∈Dn±,f∈Dn± ⟹ Gn±(f)=∅.G_{n+1}^\pm(f)=\varnothing\ \Longrightarrow\ f\in D_n^\pm,\qquad f\in D_n^\pm\ \Longrightarrow\ G_n^\pm(f)=\varnothing .Gn+1±​(f)=∅ ⟹ f∈Dn±​,f∈Dn±​ ⟹ Gn±​(f)=∅.

The two halves bracket Dn±D_n^\pmDn±​ between two finitely checkable conditions.

Milestones

In the order the proof uses them: the Cesàro limit and (15); Lemma 5 (the reduced resolvent); Theorem 2 (the Laurent expansion Rρ(Q)=ρ−1P∗+∑n≥0(−ρ)nHn+1R_\rho(Q)=\rho^{-1}P^*+\sum_{n\ge0}(-\rho)^nH^{n+1}Rρ​(Q)=ρ−1P∗+∑n≥0​(−ρ)nHn+1); Lemma 7 (P∗+ρHP^*+\rho HP∗+ρH nonnegative, positive diagonal, nonsingular for small ρ>0\rho>0ρ>0); Theorem 3 (the expansion of Vρ(f∞)V_\rho(f^\infty)Vρ​(f∞)); (30)–(31) (the comparison identity); Lemma 8 (the expansion of the test quantity); Lemma 9 (the coefficient identity); the nonemptiness of D∞±D_\infty^\pmD∞±​; the characterization Dn±={f:Yn±(f)⪰Yn±(g) ∀g}D_n^\pm=\{f: Y_n^\pm(f)\succeq Y_n^\pm(g)\ \forall g\}Dn±​={f:Yn±​(f)⪰Yn±​(g) ∀g}; Theorem 5 (lexicographic improvement).

Significance

Theorem 4 turns a criterion stated through limits over all, possibly non-stationary, policies into a finite test on one-step switches. It is what lets the policy improvement method of Miller and Veinott stop early when only an n±n^\pmn± discount optimal policy is sought, rather than an S±S^\pmS± (Blackwell) optimal one, and it recovers Blackwell's algorithm for −1+-1^+−1+ and Veinott's 1966 algorithm for 0+0^+0+ as the cases n=−1,0n=-1,0n=−1,0. The nnn-discount hierarchy is the standard framework for sensitive optimality in Markov decision processes and underlies the average-overtaking and bias criteria used in later work.

The results are proved in the paper (Theorem 3 and Lemma 8 by reference to Miller–Veinott). None of them is formalized: the platform has the Cesàro limit matrix and the deviation matrix as definitions (from Blackwell 1962), and an open statement of their basic properties for stochastic matrices only. This mission adds the substochastic matrix theory of §3, the Laurent expansion of discounted returns with Veinott's normalization, and the lexicographic characterization of n±n^\pmn± discount optimality.

Difficulty

The definition of Dn±D_n^\pmDn±​ compares f∞f^\inftyf∞ with every policy, and the comparison is a liminf of a scaled difference of infinite series. The obvious route, comparing Laurent coefficients, applies only to stationary policies, whose returns have Laurent expansions; reducing the comparison with arbitrary policies to stationary ones requires the existence of an ∞±\infty^\pm∞± discount optimal stationary policy, which is a separate existence theorem. The second difficulty is that P∗P^*P∗ may be singular or zero and the chain may have several recurrent classes: Lemma 7 is what replaces an analysis of the chain structure in the inductive step of the proof.

Formalization scope

Lean 4 with Mathlib, namespace VeinottSensitiveDP.Sensitive. States form a nonempty Fintype; each state has its own finite nonempty action type A s; policies are sequences ℕ → F with π 0 the paper's f1f_1f1​. All quantities are real. P∗P^*P∗ and HHH are the published limitMatrix and deviationMatrix; their properties for substochastic matrices are milestones, not assumptions. VρV_\rhoVρ​ is a tsum with Veinott's extra factor β\betaβ; every theorem about it carries the hypotheses under which the series converges. The spectral radius is that of the complexified matrix. The sign ±\pm± is a real parameter σ∈{1,−1}\sigma\in\{1,-1\}σ∈{1,−1}, and statements for σ=−1\sigma=-1σ=−1 assume the transient case. The liminf in (27) is encoded as "for every ε>0\varepsilon>0ε>0, eventually ≥−ε\ge-\varepsilon≥−ε", which is the extended-real liminf. Matrices with columns −1,…,n-1,\dots,n−1,…,n are functions of an integer column index read on [−1,n][-1,n][−1,n].

A trivializing formalization would define Dn±D_n^\pmDn±​ by comparison with stationary policies only, or directly as the lexicographic maximizers of Yn±Y_n^\pmYn±​; here Dn±D_n^\pmDn±​ is defined by (27) against all policies, and the lexicographic characterization is a milestone.

Contributions welcome: proofs of the matrix results of §3 (reusable for any work on deviation matrices and Laurent expansions of Markov chains), of the Laurent expansions, and of the existence of ∞±\infty^\pm∞± discount optimal stationary policies.

Selected references

  • A. F. Veinott, Jr., Discrete dynamic programming with sensitive discount optimality criteria, Ann. Math. Statist. 40(5):1635–1660, 1969. https://doi.org/10.1214/aoms/1177697379
  • D. Blackwell, Discrete dynamic programming, Ann. Math. Statist. 33(2):719–726, 1962. https://doi.org/10.1214/aoms/1177704593
  • A. F. Veinott, Jr., On finding optimal policies in discrete dynamic programming with no discounting, Ann. Math. Statist. 37(5):1284–1294, 1966. https://doi.org/10.1214/aoms/1177699272
  • B. L. Miller and A. F. Veinott, Jr., Discrete dynamic programming with a small interest rate, Ann. Math. Statist. 40(2):366–370, 1969. https://doi.org/10.1214/aoms/1177697700
17 thms1 active userReviewed
Dynamic ProgrammingOperations Research·Captain: mikedeng1

On Finding Optimal Policies in Discrete Dynamic Programming with No Discounting 2: f Maximizes Gain and Then Bias Exactly When Its Averaged Finite-Horizon Returns Dominate Every Stationary Policy'sResearch Paper

Motivation

A Markov decision process with finitely many states and actions is usually run either with a discount factor β<1\beta<1β<1 or under the long-run average criterion. Without discounting, the total return over an infinite horizon is typically infinite, and the average return per period ignores everything that happens in any finite stretch of time: two policies with equal gain (average return per period) can differ by a fixed amount of income forever, and the average criterion cannot tell them apart. Finer, undiscounted criteria that break such ties are a standard topic of the MDP literature (Puterman, Markov Decision Processes, Ch. 10), and the paper of this mission is one of their starting points.

Timeline. Howard (1960) introduced policy iteration for the average-return problem. Blackwell (1962) expanded the discounted return of a stationary policy near β=1\beta=1β=1 as x(f)/(1−β)+y(f)+o(1)x(f)/(1-\beta)+y(f)+o(1)x(f)/(1−β)+y(f)+o(1), defined 1-optimal ("nearly optimal") policies, and showed that the stationary 1-optimal policies are exactly those that maximize the gain x(f)x(f)x(f) and then the bias y(f)y(f)y(f). Veinott (1966), the source of this mission, gave a finite algorithm for such policies and, in §5, a characterization of the same set by an undiscounted criterion: comparing the Cesàro averages of the finite-horizon total returns. Later work (Veinott 1969) developed the hierarchy of nnn-discount optimality criteria from this starting point.

Setting

There are finitely many states sss and a finite set of actions; in state sss, action aaa earns the income i(s,a)i(s,a)i(s,a) and moves the system to state s′s's′ with probability q(s′∣s,a)q(s'\mid s,a)q(s′∣s,a). A decision rule f∈Ff\in Ff∈F chooses an action f(s)f(s)f(s) in each state; r(f)r(f)r(f) is the vector with entries i(s,f(s))i(s,f(s))i(s,f(s)), and Q(f)Q(f)Q(f) is the Markov matrix with entries q(s′∣s,f(s))q(s'\mid s,f(s))q(s′∣s,f(s)). A policy is a sequence π=(f1,f2,… )\pi=(f_1,f_2,\dots)π=(f1​,f2​,…) of decision rules, Qn(π)=Q(f1)⋯Q(fn)Q_n(\pi)=Q(f_1)\cdots Q(f_n)Qn​(π)=Q(f1​)⋯Q(fn​) with Q0(π)=IQ_0(\pi)=IQ0​(π)=I, and f∞=(f,f,… )f^\infty=(f,f,\dots)f∞=(f,f,…) is the stationary policy that uses fff every period.

The limit matrix Q∗(f)=lim⁡N→∞N−1∑i=0N−1Q(f)iQ^*(f)=\lim_{N\to\infty}N^{-1}\sum_{i=0}^{N-1}Q(f)^iQ∗(f)=limN→∞​N−1∑i=0N−1​Q(f)i exists for every Markov matrix. The gain and bias of fff are

x(f)=Q∗(f) r(f),y(f)=H(f) r(f),H(f)=(I−Q(f)+Q∗(f))−1−Q∗(f).x(f)=Q^*(f)\,r(f),\qquad y(f)=H(f)\,r(f),\qquad H(f)=\bigl(I-Q(f)+Q^*(f)\bigr)^{-1}-Q^*(f).x(f)=Q∗(f)r(f),y(f)=H(f)r(f),H(f)=(I−Q(f)+Q∗(f))−1−Q∗(f).

Vectors are compared coordinatewise. The decision rules of maximal gain form

F′={f∈F:x(f)≥x(g) for all g∈F},F'=\{f\in F: x(f)\ge x(g)\text{ for all }g\in F\},F′={f∈F:x(f)≥x(g) for all g∈F},

and those of maximal bias among them form

F′′={f∈F′:y(f)≥y(g) for all g∈F′}.F''=\{f\in F': y(f)\ge y(g)\text{ for all }g\in F'\}.F′′={f∈F′:y(f)≥y(g) for all g∈F′}.

Finally, the nnn-period total expected return of a policy π\piπ, starting from each state, is

Vn(π)=∑i=0n−1Qi(π) r(fi+1).V^n(\pi)=\sum_{i=0}^{n-1}Q_i(\pi)\,r(f_{i+1}).Vn(π)=i=0∑n−1​Qi​(π)r(fi+1​).

Formalization targets

Goal: Theorem 7

For every f∈Ff\in Ff∈F,

f∈F′′  ⟺  lim⁡N→∞1N∑n=1N[Vn(f∞)−Vn(g∞)] ≥ 0for all g∈F,f\in F''\iff \lim_{N\to\infty}\frac1N\sum_{n=1}^{N}\bigl[V^n(f^\infty)-V^n(g^\infty)\bigr]\ \ge\ 0\quad\text{for all }g\in F,f∈F′′⟺N→∞lim​N1​n=1∑N​[Vn(f∞)−Vn(g∞)] ≥ 0for all g∈F,

where the inequality is coordinatewise and the limit is taken in [−∞,+∞][-\infty,+\infty][−∞,+∞].

Milestones

  1. Theorem 3 (Blackwell): Vβ(f∞)=x(f)/(1−β)+y(f)+ε(β,f)V_\beta(f^\infty)=x(f)/(1-\beta)+y(f)+\varepsilon(\beta,f)Vβ​(f∞)=x(f)/(1−β)+y(f)+ε(β,f) with ε(β,f)→0\varepsilon(\beta,f)\to0ε(β,f)→0 as β→1−\beta\to1^-β→1−, where x(f)x(f)x(f) and y(f)y(f)y(f) are the unique solutions of [I−Q(f)]x=0, Q∗(f)x=Q∗(f)r(f)[I-Q(f)]x=0,\ Q^*(f)x=Q^*(f)r(f)[I−Q(f)]x=0, Q∗(f)x=Q∗(f)r(f) and [I−Q(f)]y=r(f)−x(f), Q∗(f)y=0[I-Q(f)]y=r(f)-x(f),\ Q^*(f)y=0[I−Q(f)]y=r(f)−x(f), Q∗(f)y=0.
  2. The nnn-step identity (p. 1293): y(f)=Vn(f∞)−n x(f)+Q(f)ny(f)y(f)=V^n(f^\infty)-n\,x(f)+Q(f)^n y(f)y(f)=Vn(f∞)−nx(f)+Q(f)ny(f) for every nnn.
  3. (27): y(f)=lim⁡N→∞N−1∑n=1N[Vn(f∞)−n x(f)]y(f)=\lim_{N\to\infty}N^{-1}\sum_{n=1}^N[V^n(f^\infty)-n\,x(f)]y(f)=limN→∞​N−1∑n=1N​[Vn(f∞)−nx(f)].
  4. (28): N−1∑n=1NVn(f∞)=N+12x(f)+y(f)+σ(N,f)N^{-1}\sum_{n=1}^N V^n(f^\infty)=\tfrac{N+1}{2}x(f)+y(f)+\sigma(N,f)N−1∑n=1N​Vn(f∞)=2N+1​x(f)+y(f)+σ(N,f) with σ(N,f)→0\sigma(N,f)\to0σ(N,f)→0.
  5. Theorem 4 (Blackwell): F′′F''F′′ is nonempty and is exactly the set of fff for which f∞f^\inftyf∞ is 1-optimal.

Significance

The result. Theorem 7 gives the set F′′F''F′′, defined through the discount-factor expansion, an interpretation that involves no discounting at all: f∞f^\inftyf∞ maximizes gain and then bias exactly when, from every starting state, its Cesàro-averaged finite-horizon returns are in the limit at least those of every other stationary policy. Combined with Theorem 4 it identifies the stationary 1-optimal policies with the stationary policies that are optimal under this average-overtaking comparison. The paper records a consequence (an optimal policy in the average-overtaking sense (26) that is stationary is 1-optimal) and conjectures the converse; that remark and conjecture are not part of this mission.

Formalizing it. The results are proved in the paper (Theorems 3 and 4 are Blackwell's). None of them is formalized: the Blackwell expansion (Theorem 3) is an open item on Prove2Me, referenced here, and Theorem 4, the nnn-step identity, (27), (28) and Theorem 7 have no machine-checked proof. A complete development gives the first formal treatment of Cesàro-averaged finite-horizon returns of a finite MDP and their relation to gain and bias.

Difficulty

The paper calls Theorem 7 "an immediate consequence of the representation (28)". For the direction "the limit condition implies f∈F′′f\in F''f∈F′′" this is accurate: a state where x(f)<x(g)x(f)<x(g)x(f)<x(g) would drive the average to −∞-\infty−∞, so f∈F′f\in F'f∈F′, and then the constant terms give the bias comparison over F′F'F′. The converse is where the obvious argument stops. It must hold for every g∈Fg\in Fg∈F, including g∉F′g\notin F'g∈/F′; at a state where x(f)s=x(g)sx(f)_s=x(g)_sx(f)s​=x(g)s​ for such a ggg, (28) needs y(f)s≥y(g)sy(f)_s\ge y(g)_sy(f)s​≥y(g)s​, and the definition of F′′F''F′′ only compares biases with gain-maximal rules. That inequality comes from 1-optimality of f∞f^\inftyf∞ (Theorem 4) together with the expansion of Theorem 3, not from (28). Proving (27) itself requires the Cesàro convergence N−1∑n<NQ(f)n→Q∗(f)N^{-1}\sum_{n<N}Q(f)^n\to Q^*(f)N−1∑n<N​Q(f)n→Q∗(f) and the identity Q∗(f)y(f)=0Q^*(f)y(f)=0Q∗(f)y(f)=0, which rest on Blackwell's Lemma 1 about Markov matrices.

Formalization scope

The development builds on the published Lean model of Blackwell (1962): Model (states St, actions Act, incomes, transition law), Policy (indexed from 000, so π 0 is f1f_1f1​), Qn, V, IsNearlyOptimal (= 1-optimal), limitMatrix, and the closed forms x, y. Conventions:

  • State and action sets are finite and nonempty, and every action is available in every state (As=AA_s=AAs​=A).
  • Vectors are real functions on states with the pointwise order; limits of vectors are coordinatewise; limits in NNN are along the natural numbers. The Cesàro mean in limitMatrix uses N+1N+1N+1 terms rather than Veinott's NNN; the limit is the same.
  • x(f)x(f)x(f) and y(f)y(f)y(f) are their closed forms; Theorem 3 asserts they are the unique solutions of Veinott's (2) and (3).
  • 1-optimality is Blackwell's "nearly optimal", formulated without U(β)U(\beta)U(β), against all policies.
  • In Theorem 7 the limit is an extended-real limit, stated as: for every ggg and every state, the real sequence converges in [−∞,+∞][-\infty,+\infty][−∞,+∞] to some L≥0L\ge0L≥0. A real-valued limit would make the statement false whenever x(f)s≠x(g)sx(f)_s\ne x(g)_sx(f)s​=x(g)s​; the comparison class is the stationary policies g∞g^\inftyg∞ only, and it must not be narrowed to g∈F′g\in F'g∈F′.

The shared module defines F′F'F′ and F′′F''F′′; this chunk defines Vn(π)V^n(\pi)Vn(π). Contributions are welcome for any milestone, and in particular for reusable lemmas about Cesàro means of powers of a Markov matrix, which the proofs of (27) and (28) and Blackwell's Lemma 1 share.

Selected references

  • A. F. Veinott, Jr., On Finding Optimal Policies in Discrete Dynamic Programming with No Discounting, Ann. Math. Statist. 37(5):1284–1294, 1966. https://doi.org/10.1214/aoms/1177699272
  • D. Blackwell, Discrete Dynamic Programming, Ann. Math. Statist. 33(2):719–726, 1962. https://doi.org/10.1214/aoms/1177704593
  • R. A. Howard, Dynamic Programming and Markov Processes, Wiley, New York, 1960.
  • A. F. Veinott, Jr., Discrete Dynamic Programming with Sensitive Discount Optimality Criteria, Ann. Math. Statist. 40(5):1635–1660, 1969. https://doi.org/10.1214/aoms/1177697379
  • M. L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming, Wiley, 1994. https://doi.org/10.1002/9780470316887
11 thms1 active userReviewed
Dynamic ProgrammingMachine LearningProbability+1·Captain: mikedeng1

An Analysis of Temporal-Difference Learning with Function Approximation 1: Linear TD(λ) Converges with Probability 1 to the Fixed Point of ΠT^(λ), with Error ≤ ‖ΠJ* − J*‖_D/(1 − α(1−λ)/(1−αλ))Research Paper

Motivation

Temporal-difference learning, TD(λ\lambdaλ), is the basic algorithm of reinforcement learning for estimating the expected discounted cost of a Markov chain from observed transitions (Sutton, 1988). On large or infinite state spaces it is run with a linear function approximator: the cost-to-go is represented as a weighted sum of KKK fixed basis functions, and the algorithm updates the weights. With a lookup table (one basis function per state) its convergence follows from general results on contraction mappings. With function approximation that argument breaks down. Before 1996 there were only results on convergence in the mean (Dayan, 1992), sketches restricted to linearly independent feature vectors ϕ(i)\phi(i)ϕ(i), and counterexamples in which related algorithms diverge.

Tsitsiklis and Van Roy (MIT report LIDS-P-2322, 1996; IEEE Trans. Automatic Control 42(5), 1997) proved that on-line TD(λ\lambdaλ) with linear function approximation converges with probability 1 on finite or countably infinite ergodic chains. They identified the limit as the fixed point of a projected Bellman-type operator ΠT(λ)\Pi T^{(\lambda)}ΠT(λ) and bounded its error. This analysis is the reference point for later results: finite-time bounds for linear TD, the off-policy divergence examples, and the gradient-TD family.

Timeline:

  • 1988: Sutton introduces TD(λ\lambdaλ) and proves convergence in the mean for TD(0) on absorbing chains with linearly independent features.
  • 1992: Dayan extends convergence in the mean to general λ\lambdaλ.
  • 1994: Jaakkola, Jordan and Singh, and Tsitsiklis, prove almost-sure convergence for lookup tables, using maximum-norm contraction.
  • 1996: Tsitsiklis and Van Roy prove almost-sure convergence with linear function approximation and the error bound below.

Setting

A Markov chain i0,i1,…i_0, i_1, \dotsi0​,i1​,… lives on a finite or countably infinite state space SSS, with transition matrix P=(pij)P = (p_{ij})P=(pij​). A transition from iii to jjj costs g(i,j)g(i,j)g(i,j), and α∈(0,1)\alpha \in (0,1)α∈(0,1) is a discount factor. The cost-to-go is

J∗(i)=E[∑t=0∞αtg(it,it+1) ∣ i0=i].J^*(i) = E\Big[\sum_{t=0}^\infty \alpha^t g(i_t,i_{t+1}) \,\Big|\, i_0 = i\Big].J∗(i)=E[t=0∑∞​αtg(it​,it+1​)​i0​=i].

Basis functions ϕ1,…,ϕK:S→R\phi_1,\dots,\phi_K : S \to \mathbb Rϕ1​,…,ϕK​:S→R give the feature vector ϕ(i)∈RK\phi(i) \in \mathbb R^Kϕ(i)∈RK and the approximation J~(i,r)=r′ϕ(i)\tilde J(i,r) = r'\phi(i)J~(i,r)=r′ϕ(i), written Φ′r\Phi' rΦ′r as a vector over SSS.

Starting from an arbitrary r0r_0r0​, TD(λ\lambdaλ) with λ∈[0,1]\lambda \in [0,1]λ∈[0,1] and step sizes γt\gamma_tγt​ forms the eligibility vector zt=∑k=0t(αλ)t−kϕ(ik)z_t = \sum_{k=0}^t (\alpha\lambda)^{t-k}\phi(i_k)zt​=∑k=0t​(αλ)t−kϕ(ik​) and the temporal difference dt=g(it,it+1)+αϕ(it+1)′rt−ϕ(it)′rtd_t = g(i_t,i_{t+1}) + \alpha\phi(i_{t+1})'r_t - \phi(i_t)'r_tdt​=g(it​,it+1​)+αϕ(it+1​)′rt​−ϕ(it​)′rt​, and updates rt+1=rt+γtdtztr_{t+1} = r_t + \gamma_t d_t z_trt+1​=rt​+γt​dt​zt​.

Let π\piπ be the invariant distribution, D=diag(π)D = \mathrm{diag}(\pi)D=diag(π), ∥J∥D2=∑iπ(i)J(i)2\|J\|_D^2 = \sum_i \pi(i)J(i)^2∥J∥D2​=∑i​π(i)J(i)2, and L2(S,D)={J:∥J∥D<∞}L_2(S,D) = \{J : \|J\|_D < \infty\}L2​(S,D)={J:∥J∥D​<∞}. Write Π=Φ′(ΦDΦ′)−1ΦD\Pi = \Phi'(\Phi D\Phi')^{-1}\Phi DΠ=Φ′(ΦDΦ′)−1ΦD for the DDD-orthogonal projection onto {Φ′r}\{\Phi'r\}{Φ′r}. The operator T(λ)T^{(\lambda)}T(λ) averages mmm-stage truncated costs:

(T(λ)J)(i)=(1−λ)∑m=0∞λmE[∑t=0mαtg(it,it+1)+αm+1J(im+1) ∣ i0=i],(T^{(\lambda)}J)(i) = (1-\lambda)\sum_{m=0}^\infty \lambda^m E\Big[\sum_{t=0}^m \alpha^t g(i_t,i_{t+1}) + \alpha^{m+1}J(i_{m+1}) \,\Big|\, i_0 = i\Big],(T(λ)J)(i)=(1−λ)m=0∑∞​λmE[t=0∑m​αtg(it​,it+1​)+αm+1J(im+1​)​i0​=i],

for λ<1\lambda < 1λ<1, and T(1)J=J∗T^{(1)}J = J^*T(1)J=J∗. The paper makes four assumptions:

  1. the chain has a unique invariant distribution with π(i)>0\pi(i) > 0π(i)>0, E0[g2]<∞E_0[g^2] < \inftyE0​[g2]<∞, and finite J∗J^*J∗;
  2. the ϕk\phi_kϕk​ are linearly independent with E0[ϕk2]<∞E_0[\phi_k^2] < \inftyE0​[ϕk2​]<∞;
  3. polynomial moment growth and summable mixing conditions, relative to coordinates σ(i)∈RN\sigma(i) \in \mathbb R^Nσ(i)∈RN of the states;
  4. nonnegative, nonincreasing step sizes with ∑γt=∞\sum\gamma_t = \infty∑γt​=∞ and ∑γt2<∞\sum\gamma_t^2 < \infty∑γt2​<∞.

Formalization targets

Goal: Theorem 1 (p. 11)

Under Assumptions 1–4: J∗∈L2(S,D)J^* \in L_2(S,D)J∗∈L2​(S,D); for every λ∈[0,1]\lambda \in [0,1]λ∈[0,1], TD(λ\lambdaλ) converges with probability 1, from every initial state and every r0r_0r0​, to a vector r∗r^*r∗; r∗r^*r∗ is the unique solution of

ΠT(λ)(Φ′r∗)=Φ′r∗;\Pi T^{(\lambda)}(\Phi' r^*) = \Phi' r^*;ΠT(λ)(Φ′r∗)=Φ′r∗;

and

∥Φ′r∗−J∗∥D≤∥ΠJ∗−J∗∥D1−α(1−λ)/(1−λα).\|\Phi' r^* - J^*\|_D \le \frac{\|\Pi J^* - J^*\|_D}{1 - \alpha(1-\lambda)/(1-\lambda\alpha)}.∥Φ′r∗−J∗∥D​≤1−α(1−λ)/(1−λα)∥ΠJ∗−J∗∥D​​.

Milestones

  • Lemma 1: ∥PJ∥D≤∥J∥D\|PJ\|_D \le \|J\|_D∥PJ∥D​≤∥J∥D​.
  • Lemma 2: J∗J^*J∗ is finite, lies in L2(S,D)L_2(S,D)L2​(S,D), and J∗=∑t(αP)tgˉJ^* = \sum_t(\alpha P)^t\bar gJ∗=∑t​(αP)tgˉ​.
  • Lemma 3: T(λ)T^{(\lambda)}T(λ) maps L2(S,D)L_2(S,D)L2​(S,D) into itself and has a closed-form series.
  • Lemma 4: T(λ)T^{(\lambda)}T(λ) is a ∥⋅∥D\|\cdot\|_D∥⋅∥D​-contraction with modulus α(1−λ)/(1−αλ)\alpha(1-\lambda)/(1-\alpha\lambda)α(1−λ)/(1−αλ).
  • Lemma 5: ΠT(λ)\Pi T^{(\lambda)}ΠT(λ) has a unique fixed point Φ′r∗\Phi'r^*Φ′r∗, with the error bound above.
  • Lemma 6: steady-state moments of ϕ\phiϕ and ztz_tzt​.
  • Lemma 7: the steady-state mean step is E0[s(r,Xt)]=ΦD(T(λ)(Φ′r)−Φ′r)E_0[s(r,X_t)] = \Phi D(T^{(\lambda)}(\Phi'r) - \Phi'r)E0​[s(r,Xt​)]=ΦD(T(λ)(Φ′r)−Φ′r).
  • Lemma 8: (r−r∗)′E0[s(r,Xt)]<0(r - r^*)'E_0[s(r,X_t)] < 0(r−r∗)′E0​[s(r,Xt​)]<0 for r≠r∗r \neq r^*r=r∗.
  • Theorem 2: a stochastic approximation theorem with Markov noise.
  • §6: A=E0[A(Xt)]A = E_0[A(X_t)]A=E0​[A(Xt​)] is negative definite and Ar∗+b=0Ar^* + b = 0Ar∗+b=0.

Significance

Theorem 1 says that TD(λ\lambdaλ), run on-line along a single trajectory, converges for every λ\lambdaλ, and that its limit has a precise meaning: the fixed point of the projected operator. The error bound shows that as λ→1\lambda \to 1λ→1 the limit approaches the best approximation ΠJ∗\Pi J^*ΠJ∗, and that for smaller λ\lambdaλ the error can grow by at most a factor (1−αλ)/(1−α)(1-\alpha\lambda)/(1-\alpha)(1−αλ)/(1−α). The analysis also explains the failures of related schemes: the projection and the contraction must use the same norm, ∥⋅∥D\|\cdot\|_D∥⋅∥D​, which holds only when states are sampled along the chain.

The result is proved on paper. To our knowledge no part of it has been machine-checked; it is not in Mathlib or on this platform. This mission produces:

  • a Lean model of countable-state Markov chains with costs, their path law, the weighted space L2(S,D)L_2(S,D)L2​(S,D) and the projection Π\PiΠ, which is reusable for later work on linear TD, LSTD and approximate policy evaluation;
  • the deterministic layer (Lemmas 1–5, 7, 8), which is self-contained functional analysis on L2(S,D)L_2(S,D)L2​(S,D);
  • Theorem 2, a stochastic approximation theorem with Markov noise, which the paper quotes from Benveniste, Métivier and Priouret (1987) without proof. A formal proof of it would be a library result in its own right.

Difficulty

The obvious approach treats TD(λ\lambdaλ) as a noisy version of the deterministic iteration rˉt+1=rˉt+γtΦD(T(λ)(Φ′rˉt)−Φ′rˉt)\bar r_{t+1} = \bar r_t + \gamma_t\Phi D(T^{(\lambda)}(\Phi'\bar r_t) - \Phi'\bar r_t)rˉt+1​=rˉt​+γt​ΦD(T(λ)(Φ′rˉt​)−Φ′rˉt​). It fails at the noise. The increments s(rt,Xt)s(r_t,X_t)s(rt​,Xt​) are driven by the Markov process Xt=(it,it+1,zt)X_t = (i_t,i_{t+1},z_t)Xt​=(it​,it+1​,zt​), not by independent samples. They are not martingale differences, and on an infinite state space they are unbounded. The standard ODE method with martingale-difference noise therefore does not apply, and the bias from Markov dependence must be controlled through the mixing and growth conditions of Assumption 3. The algorithm also starts with z−1=0z_{-1} = 0z−1​=0 rather than in steady state, so the steady-state identities of Lemmas 6–8 must be transferred to the actual trajectory. The contraction argument needs care for a different reason: Π\PiΠ is a nonexpansion of ∥⋅∥D\|\cdot\|_D∥⋅∥D​ only because DDD is the invariant distribution of the same chain that defines T(λ)T^{(\lambda)}T(λ).

Formalization scope

SSS is a countable type with the discrete σ\sigmaσ-algebra, PPP a Markov kernel (pij=P(i)({j})p_{ij} = P(i)(\{j\})pij​=P(i)({j}), PmP^mPm the kernel power), and the path law of the chain is Mathlib's Ionescu–Tulcea measure Kernel.trajMeasure. Standing conventions:

  • α∈(0,1)\alpha \in (0,1)α∈(0,1) and λ∈[0,1]\lambda \in [0,1]λ∈[0,1];
  • "with probability 1" is almost-sure convergence under the path law from each deterministic initial state;
  • J∗J^*J∗ and T(λ)T^{(\lambda)}T(λ) are defined by the page's path expectations, not by the series of Lemmas 2 and 3;
  • L2(S,D)L_2(S,D)L2​(S,D) membership is summability of π(i)J(i)2\pi(i)J(i)^2π(i)J(i)2;
  • Assumption 1(c) means absolute integrability of the discounted cost series;
  • vector norms are Euclidean and matrix norms Frobenius (equivalent to the induced norm and used only under existential constants);
  • (ΦDΦ′)−1(\Phi D\Phi')^{-1}(ΦDΦ′)−1 is Lean's matrix inverse, invertible under Assumptions 1(a) and 2;
  • the steady-state process of Lemmas 6–8 is any two-sided chain on a probability space with the stationary finite-dimensional distributions.

Lean's integral of a non-integrable function and sum of a non-summable series are 000. A trivializing formalization is excluded by stating every expectation and series with its integrability or summability: "well defined and finite" is always a conjunct or a hypothesis, never left to junk values.

The development needs the following:

  • Markov-chain path measures and stationary two-sided chains;
  • Bochner integrals of vector-valued functions;
  • the Cauchy–Schwarz inequality on L2(S,D)L_2(S,D)L2​(S,D);
  • a Markov-noise stochastic approximation theorem.

Contributions are welcome at every level: proofs of the deterministic lemmas, a construction of the stationary chain from π\piπ and PPP (showing that the steady-state hypotheses are satisfiable), and a proof of Theorem 2 or a stronger stochastic approximation result that implies it.

Selected references

  • J. N. Tsitsiklis and B. Van Roy, An Analysis of Temporal-Difference Learning with Function Approximation, MIT LIDS report LIDS-P-2322, 1996; IEEE Transactions on Automatic Control 42(5):674–690, 1997. https://doi.org/10.1109/9.580874
  • A. Benveniste, M. Métivier and P. Priouret, Adaptive Algorithms and Stochastic Approximations, Springer, 1990 (French original 1987). https://doi.org/10.1007/978-3-642-75894-2
  • R. S. Sutton, Learning to predict by the methods of temporal differences, Machine Learning 3:9–44, 1988. https://doi.org/10.1007/BF00115009
  • P. Dayan, The convergence of TD(λ) for general λ, Machine Learning 8:341–362, 1992. https://doi.org/10.1007/BF00992701
  • T. Jaakkola, M. I. Jordan and S. P. Singh, On the convergence of stochastic iterative dynamic programming algorithms, Neural Computation 6(6):1185–1201, 1994. https://doi.org/10.1162/neco.1994.6.6.1185
  • J. N. Tsitsiklis, Asynchronous stochastic approximation and Q-learning, Machine Learning 16:185–202, 1994. https://doi.org/10.1007/BF00993306
13 thms1 active userReviewed
Dynamic ProgrammingOperations ResearchOptimization·Captain: mikedeng1

Inventory Control in a Fluctuating Demand Environment II: With a Fixed Order Cost, a World-Dependent (r, S) Policy Is Optimal and Lies within Veinott-Type BoundsResearch Paper

Motivation

Demand for many products is not stationary: it rises and falls with the economy, the season, a product's life cycle or a customer's state. Song and Zipkin (1993) (DOI 10.1287/opre.41.2.351) model this by a world-driven demand process: an exogenous Markov chain AAA (the "state of the world") whose current state sets the rate of a Poisson demand stream. The model contains the classical stationary inventory model as the case of one world state, and it is a standard example of a Markov-modulated decision problem in operations research.

For stationary demand with a fixed order cost, optimal (s,S)(s, S)(s,S) policies go back to Scarf (1960) for finite horizons and Iglehart (1963) for the infinite horizon; Veinott (1966) bounded the optimal parameters by quantities computed from the one-period cost alone. This mission formalizes the extension of both results to the fluctuating-demand model (§3.2 of the paper). A companion mission treats the linear order-cost case, in which a world-dependent basestock policy is optimal.

Setting

The world AAA is a continuous-time Markov chain on a countable set III with generator (qij)(q_{ij})(qij​), qi=−qii=∑j≠iqijq_i = -q_{ii} = \sum_{j\neq i} q_{ij}qi​=−qii​=∑j=i​qij​, and bounded rates. While A=iA = iA=i, unit demands arrive at Poisson rate λi\lambda_iλi​; shortages are backlogged. An order arrives after a random lead time LLL, independent of the world and the demand. The inventory position x∈Zx \in \mathbb Zx∈Z is on-hand stock minus backorders plus stock on order.

Costs: a fixed cost Kˉ\bar KKˉ per order and a unit cost cˉ\bar ccˉ, both paid on arrival; a holding cost rate h>0h > 0h>0 and a penalty cost rate p>0p > 0p>0; a discount rate α>0\alpha > 0α>0. With F~L(α)=E[e−αL]\tilde F_L(\alpha) = E[e^{-\alpha L}]F~L​(α)=E[e−αL], the discounted costs are c=cˉF~L(α)c = \bar c\tilde F_L(\alpha)c=cˉF~L​(α) and K=KˉF~L(α)K = \bar K\tilde F_L(\alpha)K=KˉF~L​(α). If DLiD^i_LDLi​ is the demand during a lead time from world state iii and C^(x)=max⁡{−px,hx}\hat C(x) = \max\{-px, hx\}C^(x)=max{−px,hx},

C(i,y)=E[e−αLC^(y−DLi)].C(i, y) = E\big[e^{-\alpha L}\hat C(y - D^i_L)\big].C(i,y)=E[e−αLC^(y−DLi​)].

Uniformization at a rate μ≥sup⁡iqi+sup⁡iλi\mu \ge \sup_i q_i + \sup_i\lambda_iμ≥supi​qi​+supi​λi​ turns the problem into a discrete-time dynamic program with β=1/(μ+α)\beta = 1/(\mu+\alpha)β=1/(μ+α), γ=βμ\gamma = \beta\muγ=βμ and the myopic cost G+(i,y)=(1−γ)cy+βC(i,y)G^+(i, y) = (1-\gamma)cy + \beta C(i, y)G+(i,y)=(1−γ)cy+βC(i,y). For a terminal cost W0W_0W0​ the nnn-stage costs satisfy the recursion (10):

Wn(i,x)=min⁡y≥x{Kδ(y−x)+Gn(i,y)},W_n(i, x) = \min_{y\ge x}\{K\delta(y-x) + G_n(i, y)\},Wn​(i,x)=y≥xmin​{Kδ(y−x)+Gn​(i,y)}, Gn(i,y)=G+(i,y)+βλic+β{λiWn−1(i,y−1)+∑j≠iqijWn−1(j,y)+(μ−λi−qi)Wn−1(i,y)},G_n(i, y) = G^+(i, y) + \beta\lambda_i c + \beta\Big\{\lambda_i W_{n-1}(i, y-1) + \sum_{j\neq i} q_{ij}W_{n-1}(j, y) + (\mu - \lambda_i - q_i)W_{n-1}(i, y)\Big\},Gn​(i,y)=G+(i,y)+βλi​c+β{λi​Wn−1​(i,y−1)+j=i∑​qij​Wn−1​(j,y)+(μ−λi​−qi​)Wn−1​(i,y)},

where δ(z)=1\delta(z) = 1δ(z)=1 if z>0z > 0z>0 and δ(0)=0\delta(0) = 0δ(0)=0. Throughout, Assumption 1, αcˉ<p\alpha\bar c < pαcˉ<p, holds and K>0K > 0K>0. The terminal cost is the optimal cost W∞W_\inftyW∞​ of the linear model (K=0K = 0K=0); its G∞G_\inftyG∞​ is written G0G_0G0​, and y∗(i)y^*(i)y∗(i) is the smallest minimizer of G0(i,⋅)G_0(i, \cdot)G0​(i,⋅).

A function f:Z→Rf : \mathbb Z \to \mathbb Rf:Z→R is KKK-convex (Definition 2) if f(x)−f(x−b)ba+f(x)≤f(x+a)+K\frac{f(x)-f(x-b)}{b}a + f(x) \le f(x+a) + Kbf(x)−f(x−b)​a+f(x)≤f(x+a)+K for all xxx and all integers a,b>0a, b > 0a,b>0. The (r,S)(r, S)(r,S) policy with parameters {(r(i),S(i))}\{(r(i), S(i))\}{(r(i),S(i))} orders up to S(i)S(i)S(i) when x≤r(i)x \le r(i)x≤r(i) and the world is in state iii, and does not order otherwise. With y+(i)y^+(i)y+(i) the smallest minimizer of G+(i,⋅)G^+(i,\cdot)G+(i,⋅) and ymin⁡+=min⁡iy+(i)y^+_{\min} = \min_i y^+(i)ymin+​=mini​y+(i), the paper defines

S+(i)=min⁡{y≥y+(i):G+(i,y)−G+(i,y+(i))>γK},r+(i)=max⁡{y<y+(i):G+(i,y)−G+(i,y+(i))>(1−γ)K},r−(i)=max⁡{y<y∗(i):G0(i,y)−G0(i,y∗(i))>K},r−−(i)=max⁡{y<ymin⁡+:G+(i,y)−G+(i,ymin⁡+)>K}.\begin{aligned} S^+(i) &= \min\{y \ge y^+(i) : G^+(i, y) - G^+(i, y^+(i)) > \gamma K\},\\ r^+(i) &= \max\{y < y^+(i) : G^+(i, y) - G^+(i, y^+(i)) > (1-\gamma)K\},\\ r^-(i) &= \max\{y < y^*(i) : G_0(i, y) - G_0(i, y^*(i)) > K\},\\ r^{--}(i) &= \max\{y < y^+_{\min} : G^+(i, y) - G^+(i, y^+_{\min}) > K\}. \end{aligned}S+(i)r+(i)r−(i)r−−(i)​=min{y≥y+(i):G+(i,y)−G+(i,y+(i))>γK},=max{y<y+(i):G+(i,y)−G+(i,y+(i))>(1−γ)K},=max{y<y∗(i):G0​(i,y)−G0​(i,y∗(i))>K},=max{y<ymin+​:G+(i,y)−G+(i,ymin+​)>K}.​

Formalization targets

Goal: Theorems 5(e) and 6

Let Sn∗(i)S^*_n(i)Sn∗​(i) be the smallest minimizer of Gn(i,⋅)G_n(i,\cdot)Gn​(i,⋅) and rn∗(i)r^*_n(i)rn∗​(i) the largest y<Sn∗(i)y < S^*_n(i)y<Sn∗​(i) with Gn(i,y)>K+Gn(i,Sn∗(i))G_n(i, y) > K + G_n(i, S^*_n(i))Gn​(i,y)>K+Gn​(i,Sn∗​(i)). For every iii the sequence (rn∗(i),Sn∗(i))(r^*_n(i), S^*_n(i))(rn∗​(i),Sn∗​(i)) has limit points; for any choice of limit points (r∗(i),S∗(i))(r^*(i), S^*(i))(r∗(i),S∗(i)), the world-dependent (r,S)(r, S)(r,S) policy with these parameters is optimal for the infinite-horizon discounted problem, and

y∗(i)≤S∗(i)<S+(i),r−−(i)≤r−(i)≤r∗(i)≤r+(i).y^*(i) \le S^*(i) < S^+(i),\qquad r^{--}(i) \le r^-(i) \le r^*(i) \le r^+(i).y∗(i)≤S∗(i)<S+(i),r−−(i)≤r−(i)≤r∗(i)≤r+(i).

Milestones

Lemma 4 (uniform bounds on the iterates); Theorem 3 (KKK-convexity of GnG_nGn​, WnW_nWn​ and optimality of an (r,S)(r, S)(r,S) rule for each nnn-stage problem); Lemmas 5 and 6 (difference comparisons); Theorem 4 (the bounds above for every nnn); Theorem 5(a)–(d) (convergence of WnW_nWn​, GnG_nGn​, the (r,S)(r, S)(r,S) property of limit points, and the optimality equation (2)).

Significance

The result shows that the classical (s,S)(s, S)(s,S) structure survives Markov modulation of demand with an infinite world space and stochastic lead times: a stationary policy whose two parameters depend only on the current world state is optimal over the infinite horizon. The bounds of Theorem 6 locate those parameters in an interval computed from the myopic cost G+G^+G+ and the linear model alone, uniformly in the horizon.

The results are proved in the paper, except that Theorem 5 refers to Iglehart (1963) and Lemma 4 to Song's thesis. To our knowledge none of them has a machine-checked proof. A formalization would also produce reusable machinery: KKK-convexity on the integers, the Scarf-type inductive argument with a countable world state, and the passage from finite-horizon to infinite-horizon optimality for a nonnegative-cost Markov decision process with unbounded costs.

Difficulty

The one-period cost is unbounded in xxx and the world space may be infinite, so the usual contraction argument for discounted dynamic programs does not apply. Convergence of WnW_nWn​ needs the uniform bound of Lemma 4. Optimality of the limit policy needs a separate argument, because a pointwise limit of (r,S)(r, S)(r,S) rules need not exist: the parameter sequences are only known to have limit points. The lower bounds r−r^-r− and r−−r^{--}r−− hold only because the terminal cost is the linear model's optimal cost, which makes the difference comparisons of Lemma 6 possible; the paper notes that Lemma 6 depends on this choice of W0W_0W0​.

Formalization scope

All definitions live in the namespace SongZipkinFluct.FixedCost. Inventory positions and order-up-to levels are integers. The demand law DLiD^i_LDLi​ is defined through uniformization at rate μ\muμ, as an exact series representation of the Markov-modulated Poisson count, and the lead time is a probability law on [0,∞)[0,\infty)[0,∞) independent of the rest. The standing conventions h,p,α>0h, p, \alpha > 0h,p,α>0, cˉ,Kˉ≥0\bar c, \bar K \ge 0cˉ,Kˉ≥0, μ>0\mu > 0μ>0 and μ≥q∗+λ∗\mu \ge q^* + \lambda^*μ≥q∗+λ∗ are hypotheses of the model. Assumption 1 and Kˉ>0\bar K > 0Kˉ>0 are hypotheses of each theorem.

The iterates WnW_nWn​ are defined by the recursion (10), with the minimum written as an infimum over integers y≥xy \ge xy≥x. W∞W_\inftyW∞​ and G∞G_\inftyG∞​ are the suprema over nnn, which are finite and equal the limits by Lemma 4 and Theorem 3. Every minimizer and every extremal integer (y+y^+y+, ymin⁡+y^+_{\min}ymin+​, y∗y^*y∗, Sn∗S^*_nSn∗​, rn∗r^*_nrn∗​, S+S^+S+, r+r^+r+, r−r^-r−, r−−r^{--}r−−) is passed through its defining property, never through sInf/sSup on Z\mathbb ZZ. The goal also asserts that all of them exist, so its hypotheses can be met. A limit point of an integer sequence is a value taken infinitely often.

Optimality is over all deterministic history-dependent feasible policies and from every initial state. Costs are valued in [0,∞][0,\infty][0,∞], and a policy's cost is the supremum of its finite-horizon discounted costs in the uniformized problem (1). Restricting the comparison to (r,S)(r, S)(r,S) or stationary policies would trivialize the goal and is ruled out. KKK-convexity is the integer version of Definition 2, not convexity over Z\mathbb ZZ-scalars.

Welcome contributions: KKK-convexity calculus on Z\mathbb ZZ, convergence of the uniformized demand series, Lemma 4, and an Iglehart-type limit argument for countable-state Markov decision processes with nonnegative costs.

Selected references

  • J.-S. Song and P. Zipkin, Inventory Control in a Fluctuating Demand Environment, Operations Research 41(2):351–370, 1993. https://doi.org/10.1287/opre.41.2.351
  • H. Scarf, The Optimality of (S, s) Policies in the Dynamic Inventory Problem, in Mathematical Methods in the Social Sciences, Stanford University Press, 1960.
  • D. L. Iglehart, Optimality of (s, S) Policies in the Infinite Horizon Dynamic Inventory Problem, Management Science 9(2):259–267, 1963. https://doi.org/10.1287/mnsc.9.2.259
  • A. F. Veinott, Jr., On the Optimality of (s, S) Inventory Policies: New Conditions and a New Proof, SIAM Journal on Applied Mathematics 14(5):1067–1083, 1966. https://doi.org/10.1137/0114086
  • D. P. Bertsekas and S. E. Shreve, Stochastic Optimal Control: The Discrete-Time Case, Academic Press, 1978.
14 thms1 active userReviewed
ProbabilityStochastic Systems·Captain: mikedeng1

Random Walks in a Random Environment 2: lim X_n/n = (1 − Eσ)/(1 + Eσ) if Eσ < 1, −(1 − E(σ⁻¹))/(1 + E(σ⁻¹)) if E(σ⁻¹) < 1, and 0 OtherwiseResearch Paper

Motivation

A random walk in a random environment is a walk whose step probabilities are sampled once at each site and then reused whenever the walk returns there. Repeated visits therefore reveal the same local bias instead of drawing a fresh probability. This makes the walk different from an ordinary random walk with a single averaged step probability. Solomon's 1975 paper asks how the random environment changes the long-run direction and speed of a one-dimensional nearest-neighbor walk. Its Section 1 gives the speed law and the associated limits for first passage times. These results distinguish escape at a nonzero linear rate from escape or recurrence at zero linear rate. Solomon (1975)

The question matters whenever a particle repeatedly crosses the same heterogeneous medium. Averaging the local right-step probabilities predicts the drift of a homogeneous comparison walk, but it does not generally predict the speed in the sampled medium. Sites with a small right-step probability can hold up the walk through repeated excursions. The formula in Theorem (1.16) quantifies this effect using a moment of the ratio of left-step to right-step probability. Solomon (1975), pp. 7–8

Setting

An environment is a sequence α=(αz)z∈Z\alpha=(\alpha_z)_{z\in\mathbb Z}α=(αz​)z∈Z​ of numbers in [0,1][0,1][0,1]. In a fixed environment, a walk at site zzz moves to z+1z+1z+1 with probability αz\alpha_zαz​ and to z−1z-1z−1 with probability βz=1−αz\beta_z=1-\alpha_zβz​=1−αz​. It starts at X0=0X_0=0X0​=0. For the random environment, the variables αz\alpha_zαz​ are independent and identically distributed. First the environment is sampled; then the walk moves according to the fixed probabilities at its sites. The probability law that averages over both choices is the annealed law PPP. The mission's model records this law through every finite path cylinder, jointly with every measurable event concerning the environment. Solomon (1975), pp. 1–2

Write σz=βz/αz\sigma_z=\beta_z/\alpha_zσz​=βz​/αz​ for the ratio of leftward to rightward probability. At the endpoints, σz=∞\sigma_z=\inftyσz​=∞ when αz=0\alpha_z=0αz​=0 and σz=0\sigma_z=0σz​=0 when αz=1\alpha_z=1αz​=1. Let e=Eσ0e=E\sigma_0e=Eσ0​ and e−=E(σ0−1)e_-=E(\sigma_0^{-1})e−​=E(σ0−1​); either may be infinite. For an integer site z≠0z\ne0z=0, the passage time TzT_zTz​ is the first positive time at which XXX visits zzz, with Tz=∞T_z=\inftyTz​=∞ if the site is never reached; put T0=0T_0=0T0​=0. The positive ladder times are τn=Tn−Tn−1\tau_n=T_n-T_{n-1}τn​=Tn​−Tn−1​ where the passage times are finite. Their joint sequence, rather than individual identically distributed variables alone, is relevant to the speed law. Solomon (1975), pp. 5–7

Formalization targets

The goal is the complete three-case statement of Theorem (1.16). All limits below hold almost surely. If e<1e<1e<1, then

Tnn⟶1+e1−e,Xnn⟶1−e1+e.\frac{T_n}{n}\longrightarrow\frac{1+e}{1-e},\qquad \frac{X_n}{n}\longrightarrow\frac{1-e}{1+e}.nTn​​⟶1−e1+e​,nXn​​⟶1+e1−e​.

If e−<1e_-<1e−​<1, the symmetric limits are

T−nn⟶1+e−1−e−,Xnn⟶−1−e−1+e−.\frac{T_{-n}}{n}\longrightarrow\frac{1+e_-}{1-e_-},\qquad \frac{X_n}{n}\longrightarrow-\frac{1-e_-}{1+e_-}.nT−n​​⟶1−e−​1+e−​​,nXn​​⟶−1+e−​1−e−​​.

In the remaining case, e−1≤1≤e−e^{-1}\le1\le e_-e−1≤1≤e−​, both normalized passage times tend to infinity and Xn/nX_n/nXn​/n tends to zero. The reciprocal inequality is understood in the extended nonnegative reals, including e=∞e=\inftye=∞. The milestone list follows the paper's supporting statements: a transience observation, a fixed-environment mean passage-time identity, a separated-block independence lemma, stationarity and ergodicity of ladder times, and the annealed mean ladder-time formula. Solomon (1975), pp. 3, 5–7

Significance

The theorem gives a numerical asymptotic speed in either direction when the corresponding ratio has mean below one. It also identifies a zero-speed regime through explicit conditions on the two reciprocal moments. The passage-time clauses say more than the final speed alone: they describe the time required to reach distant positive and negative sites and distinguish a finite linear time scale from an infinite one. The same section compares the speed with that of a homogeneous walk obtained by replacing every local probability with Eα0E\alpha_0Eα0​; the random environment can slow the walk. Solomon (1975), pp. 7–8

Solomon proved these results in 1975. This mission asks for machine-checked proofs of the known statements and a reusable Lean description of the annealed nearest-neighbor model. The fixed-environment chain, first passage times, and the shift law of a sequence of ladder times are useful interfaces for further formal work on random media. No proof of these draft statements is claimed by the mission proposal. Solomon (1975)

Difficulty

The direct attempt to apply an ordinary law of large numbers to the step increments Xn−Xn−1X_n-X_{n-1}Xn​−Xn−1​ does not fit the model: those increments are not strictly stationary in a nonconstant environment. The walk revisits previously sampled site probabilities, so a later increment depends on the path's history through the sites it has encountered. The paper points out this obstruction before introducing ladder times. An additional issue is that a passage time need not be finite, and the means eee and e−e_-e−​ may also be infinite. Any argument that converts these quantities to ordinary real numbers too early loses the zero-speed cases. Solomon (1975), pp. 5, 7

Formalization scope

Lean represents sites by Z\mathbb ZZ and time indices by N\mathbb NN. A probability space carries the jointly sampled environment and path. The environment lies in [0,1][0,1][0,1] pointwise and is independent and identically distributed across all integer sites. The path law is fixed by its joint finite-dimensional cylinder probabilities, which express the annealed construction from the paper. In particular, the path cannot be chosen independently of the environment merely because its one-step marginal probabilities look right. This rules out a trivializing model that forgets the dependence between the sampled medium and the walk.

The ratios σz\sigma_zσz​, their reciprocal and their expectations take values in R≥0∪{∞}\mathbb R_{\ge0}\cup\{\infty\}R≥0​∪{∞}. Passage times take values in N∪{∞}\mathbb N\cup\{\infty\}N∪{∞}, and normalized passage-time limits are expressed in extended nonnegative reals. The position ratio Xn/nX_n/nXn​/n has a real limit. The theorem permits both endpoint probabilities, αz=0\alpha_z=0αz​=0 and αz=1\alpha_z=1αz​=1, and imposes no nondegeneracy assumption. Each convergence statement is almost sure. The ergodicity milestone concerns the full joint law of the sequence (τn)n≥1(\tau_n)_{n\ge1}(τn​)n≥1​ under the left shift; shift preservation supplies strict stationarity. Solomon (1975), pp. 1–2, 5–7

A complete development needs measure-theoretic probability, the fixed-environment chain and annealed cylinder law, extended-valued integration and first passage times, and ergodic theory for the ladder-time shift. The model and passage-time interfaces can be reused in later one-dimensional random-environment results. Contributions proving the listed milestones, sharpening their interfaces without changing their meaning, or supplying general probability lemmas used by the known proof are within scope.

Selected references

  • Fred Solomon, Random Walks in a Random Environment, The Annals of Probability 3(1):1–31, 1975. DOI: 10.1214/aop/1176996444
8 thms1 active userReviewed
ProbabilityStochastic Systems·Captain: mikedeng1

The Two-Parameter Poisson–Dirichlet Distribution Derived from a Stable Subordinator 6: n·Y_n → (1 − α)/α Almost Surely under PD(α, θ)Research Paper

Motivation

In a population divided into many species, a ranked frequency VnV_nVn​ is the fraction belonging to the nnnth most common species. The Poisson–Dirichlet distribution is a model for such random ranked frequencies. It also arises from ranked jumps of a stable subordinator, which gives the model its connection to random interval partitions. Pitman and Yor's 1997 paper develops both descriptions and studies quantities derived from the ranked sequence. Its §7.3 asks how much of the remaining total mass belongs to the nnnth ranked component when nnn is large. This is the chain factor YnY_nYn​, the quantity studied here. These interpretations and the result are in Pitman and Yor (1997).

For 0<α<10<\alpha<10<α<1, the paper first analyzes the tail ratio Σn\Sigma_nΣn​ under PD(α,0)\mathrm{PD}(\alpha,0)PD(α,0) through Proposition 11 and its Laplace transform. Proposition 14 relates PD(α,θ)\mathrm{PD}(\alpha,\theta)PD(α,θ) to that base law by a local-time density. The limiting behavior in Proposition 44 combines those two themes. The paper states both an almost-sure limit and a Gaussian fluctuation limit, but the variance printed for the latter conflicts with the variance of Σ1\Sigma_1Σ1​ displayed immediately before it. The mission retains the paper's almost-sure assertion as its goal and records the corrected fluctuation assertion separately. Pitman and Yor (1997), pp. 863–865, 890–891.

Setting

Choose 0≤α<10\le\alpha<10≤α<1 and θ>−α\theta> -\alphaθ>−α. Let Y~1,Y~2,…\tilde Y_1,\tilde Y_2,\ldotsY~1​,Y~2​,… be independent random variables, where Y~n\tilde Y_nY~n​ has the beta distribution with parameters (1−α,θ+nα)(1-\alpha,\theta+n\alpha)(1−α,θ+nα). Its density on (0,1)(0,1)(0,1) is Γ(a+b)xa−1(1−x)b−1/(Γ(a)Γ(b))\Gamma(a+b)x^{a-1}(1-x)^{b-1}/(\Gamma(a)\Gamma(b))Γ(a+b)xa−1(1−x)b−1/(Γ(a)Γ(b)) for parameters a,b>0a,b>0a,b>0. Form the stick-breaking lengths

V~1=Y~1,V~n=(1−Y~1)⋯(1−Y~n−1)Y~n(n≥2).\tilde V_1=\tilde Y_1,\qquad \tilde V_n=(1-\tilde Y_1)\cdots(1-\tilde Y_{n-1})\tilde Y_n\quad(n\ge2).V~1​=Y~1​,V~n​=(1−Y~1​)⋯(1−Y~n−1​)Y~n​(n≥2).

Sort these lengths in decreasing order, retaining multiplicities. The law of the resulting sequence V1≥V2≥⋯V_1\ge V_2\ge\cdotsV1​≥V2​≥⋯ is PD(α,θ)\mathrm{PD}(\alpha,\theta)PD(α,θ), exactly Definition 1 of the paper. Pitman and Yor (1997), p. 857.

For that ranked sequence, define the tail ratio and chain factor

Σn=Vn+1+Vn+2+⋯Vn,Yn=VnVn+Vn+1+⋯=11+Σn.\Sigma_n=\frac{V_{n+1}+V_{n+2}+\cdots}{V_n},\qquad Y_n=\frac{V_n}{V_n+V_{n+1}+\cdots}=\frac{1}{1+\Sigma_n}.Σn​=Vn​Vn+1​+Vn+2​+⋯​,Yn​=Vn​+Vn+1​+⋯Vn​​=1+Σn​1​.

The distinction between YnY_nYn​ and the original independent factors Y~n\tilde Y_nY~n​ matters: the former is computed after ranking and is generally dependent across indices. The paper also uses the function ψα(λ)=1+α∫01(1−e−λx)x−α−1 dx\psi_\alpha(\lambda)=1+\alpha\int_0^1(1-e^{-\lambda x})x^{-\alpha-1}\,dxψα​(λ)=1+α∫01​(1−e−λx)x−α−1dx for λ≥0\lambda\ge0λ≥0. It appears in the Laplace transform of Σn\Sigma_nΣn​. Pitman and Yor (1997), pp. 863–865, 890.

Formalization targets

The central target is Proposition 44, display (150): for 0<α<10<\alpha<10<α<1 and θ>−α\theta> -\alphaθ>−α,

nYn⟶1−ααalmost surely under PD(α,θ).nY_n\longrightarrow\frac{1-\alpha}{\alpha}\quad\text{almost surely under }\mathrm{PD}(\alpha,\theta).nYn​⟶α1−α​almost surely under PD(α,θ).

The milestones state the distribution and Laplace transform of Σn\Sigma_nΣn​ from Proposition 11(ii), its mean α/(1−α)\alpha/(1-\alpha)α/(1−α), the almost-sure limit of Σn/n\Sigma_n/nΣn​/n under PD(α,0)\mathrm{PD}(\alpha,0)PD(α,0), and Proposition 14's change-of-measure formula. The Gaussian companion records the central limit theorem for Σn/n\Sigma_n/nΣn​/n and the corrected version of Proposition 44's display (151):

n(nYn−1−αα)⟹N ⁣(0,(1−α)2α3(2−α)).\sqrt n\left(nY_n-\frac{1-\alpha}{\alpha}\right) \Longrightarrow N\!\left(0,\frac{(1-\alpha)^2}{\alpha^3(2-\alpha)}\right).n​(nYn​−α1−α​)⟹N(0,α3(2−α)(1−α)2​).

The printed variance in (151) is α−2(2−α)−2\alpha^{-2}(2-\alpha)^{-2}α−2(2−α)−2. It differs from the variance forced by the preceding Σn\Sigma_nΣn​ central limit theorem: Var⁡(Σ1)=α/((2−α)(1−α)2)\operatorname{Var}(\Sigma_1)=\alpha/((2-\alpha)(1-\alpha)^2)Var(Σ1​)=α/((2−α)(1−α)2) and the derivative of x↦1/xx\mapsto1/xx↦1/x at α/(1−α)\alpha/(1-\alpha)α/(1−α) give (1−α)2/(α3(2−α))(1-\alpha)^2/(\alpha^3(2-\alpha))(1−α)2/(α3(2−α)). At α=1/2\alpha=1/2α=1/2, the corrected and printed values are 4/34/34/3 and 16/916/916/9. The verbatim milestone text preserves the paper's wording; the companion theorem discloses the correction. Pitman and Yor (1997), pp. 890–891.

Significance

Display (150) identifies the almost-sure scale of the ranked chain factor: YnY_nYn​ behaves like a deterministic constant divided by nnn. This makes the order of magnitude precise for every permitted θ\thetaθ, despite the dependence among ranked components. The companion fluctuation statement gives a distributional scale around that limit and exposes a numerical inconsistency in the printed version. Neither conclusion is simply a statement about the independent beta factors Y~n\tilde Y_nY~n​; the paper compares the two sequences precisely because they have different limiting distributions. Pitman and Yor (1997), p. 891.

The result is proved in the paper, while its statements and their supporting definitions still need a machine-checked development. A completed formalization would make the ranked-frequency law, the tail-ratio limit, and the local-time change of measure available as separately reusable results. It would also settle the corrected variance at the level of a precise probability-law statement rather than silently adopting the paper's printed constant.

Difficulty

The tempting shortcut is to use Proposition 11(ii), which says that each Σn\Sigma_nΣn​ has the law of a sum of nnn independent copies of Σ1\Sigma_1Σ1​, and invoke a strong law. That information concerns each one-dimensional law separately; it does not exhibit the whole sequence (Σn)(\Sigma_n)(Σn​) as partial sums of a single independent sequence. Almost-sure convergence is a statement about their joint coupling, so the shortcut leaves a gap. The Gaussian conclusion raises a different issue: its limit must survive the change from PD(α,0)\mathrm{PD}(\alpha,0)PD(α,0) to PD(α,θ)\mathrm{PD}(\alpha,\theta)PD(α,θ), and its variance must be consistent with the displayed variance of Σ1\Sigma_1Σ1​. These are the two substantive tasks beyond the calculation of the Laplace transform. Pitman and Yor (1997), pp. 864, 890–891.

Formalization scope

Lean represents a random ranked sequence as V:Ω→(N→R)V:\Omega\to(\mathbb N\to\mathbb R)V:Ω→(N→R) on a probability space. Its index kkk means the paper's n=k+1n=k+1n=k+1. Thus the goal uses (k+1)Yk+1(k+1)Y_{k+1}(k+1)Yk+1​, and the tail-ratio milestones use Σk+1/(k+1)\Sigma_{k+1}/(k+1)Σk+1​/(k+1). The paper's Definition 1 is represented by independent beta coordinate laws followed by a decreasing ranking. The ranking counts multiplicity using extended cardinality, so an infinite set of terms above a threshold cannot acquire a finite fallback count. The law of the ranked sequence is expressed through measurable preimages. This avoids treating a nonmeasurable ranking map as a zero measure through Measure.map.

All targets retain 0<α<10<\alpha<10<α<1; the θ\thetaθ targets also retain θ>−α\theta> -\alphaθ>−α. The beta law and integrals use these parameter ranges. Real expectations state integrability or square integrability where needed, preventing a nonintegrable Bochner integral or variance from supplying a fallback zero. The nonnegative change-of-measure identity uses extended nonnegative integrals, and its local time LLL is identified by the almost-sure limit nVnαnV_n^\alphanVnα​ rather than by an arbitrary extra weight or a limit operator with a default value. The Gaussian variances are positive on this parameter range, despite being represented by nonnegative real parameters in Mathlib. The formalization must use the actual tail ratio of the PD(α,θ)\mathrm{PD}(\alpha,\theta)PD(α,θ) sequence, not an independently assigned factor process.

The reusable infrastructure includes the ranked-value operation, the stick-breaking law, convolution powers, the tail-ratio and chain-factor definitions, and the bridge between PD(α,0)\mathrm{PD}(\alpha,0)PD(α,0) and PD(α,θ)\mathrm{PD}(\alpha,\theta)PD(α,θ). Contributions that establish these interfaces or prove the displayed mean, strong law, and central limit assertions directly against them are within scope. Pitman and Yor (1997), Definition 1, Propositions 11, 14, 44.

Selected references

  • J. Pitman and M. Yor, The two-parameter Poisson–Dirichlet distribution derived from a stable subordinator, Annals of Probability 25(2):855–900, 1997. DOI: 10.1214/aop/1024404422.
8 thms1 active userReviewed
ProbabilityStochastic Systems·Captain: mikedeng1

On the Generalized “Birth-and-Death” Process 3: Given the Mean Growth n̄_t, the Rates λ = (n̄′/n̄)⁺, μ = (n̄′/n̄)⁻ Uniquely Minimize Var(n_t) for All tResearch Paper

Motivation

In a birth-and-death population model, an expected population curve does not determine the amount of random variation. Births and deaths can occur frequently while nearly canceling in the mean, or one of the two rates can vanish while giving the same mean. This distinction matters when a prescribed growth curve is used to model a population: two processes can agree on their expected size at every time and differ substantially in their uncertainty. In §6 of Kendall's 1948 paper, the question is posed directly: among time-varying birth and death rates giving a fixed mean curve, which choice minimizes the variance of the population size?

Kendall cites Arley's observation for constant rates with the same positive net growth. His §6 moves from that special case to an arbitrary prescribed mean curve and identifies a unique minimum-fluctuation rate pair. The paper also gives a logistic-growth example, connecting the stochastic rate choice to a familiar deterministic population law. The historical result is proved in the paper; this mission concerns its Lean formalization, including the exact time-dependent variance comparison and uniqueness claim.

Setting

The process starts with one ancestor. At time ttt, each member gives birth at rate λ(t)\lambda(t)λ(t) and dies at rate μ(t)\mu(t)μ(t); the rates are nonnegative. Let ntn_tnt​ be the population size and nˉt\bar n_tnˉt​ its expectation. Kendall works with the population probabilities and their generating function rather than constructing sample paths. This mission likewise uses his explicit formulas for the mean and variance, so that the comparison has a precise real-valued expression.

Define the integrated net death rate and the fluctuation by

ρλ,μ(t)=∫0t(μ(τ)−λ(τ)) dτ,Vλ,μ(t)=e−2ρλ,μ(t)∫0teρλ,μ(τ)(λ(τ)+μ(τ)) dτ.\rho_{\lambda,\mu}(t)=\int_0^t\bigl(\mu(\tau)-\lambda(\tau)\bigr)\,d\tau, \qquad V_{\lambda,\mu}(t)=e^{-2\rho_{\lambda,\mu}(t)} \int_0^t e^{\rho_{\lambda,\mu}(\tau)} \bigl(\lambda(\tau)+\mu(\tau)\bigr)\,d\tau.ρλ,μ​(t)=∫0t​(μ(τ)−λ(τ))dτ,Vλ,μ​(t)=e−2ρλ,μ​(t)∫0t​eρλ,μ​(τ)(λ(τ)+μ(τ))dτ.

Kendall's equations (13) and (14c) identify nˉt=e−ρλ,μ(t)\bar n_t=e^{-\rho_{\lambda,\mu}(t)}nˉt​=e−ρλ,μ​(t) and Vλ,μ(t)=Var⁡(nt)V_{\lambda,\mu}(t)=\operatorname{Var}(n_t)Vλ,μ​(t)=Var(nt​) for the one-ancestor process. A pair (λ,μ)(\lambda,\mu)(λ,μ) is admissible here when both rates are continuous and nonnegative on [0,∞)[0,\infty)[0,∞). It has mean nˉ\bar nnˉ when e−ρλ,μ(t)=nˉte^{-\rho_{\lambda,\mu}(t)}=\bar n_te−ρλ,μ​(t)=nˉt​ for every t≥0t\ge0t≥0. In particular, the initial value is nˉ0=1\bar n_0=1nˉ0​=1. These definitions let the mission quantify over every admissible rate pair with the prescribed mean.

For a positive continuously differentiable mean curve, write g(t)=nˉt′/nˉtg(t)=\bar n_t'/\bar n_tg(t)=nˉt′​/nˉt​. The minimum-fluctuation rates are λ∗(t)=max⁡{g(t),0}\lambda_*(t)=\max\{g(t),0\}λ∗​(t)=max{g(t),0} and μ∗(t)=max⁡{−g(t),0}\mu_*(t)=\max\{-g(t),0\}μ∗​(t)=max{−g(t),0}. When the mean decreases, only deaths occur; when it increases, only births occur; when it is locally constant, both rates are zero. This is Kendall's equation (56), expressed without first naming intervals of monotonicity.

Formalization targets

The goal is the simultaneous variance bound

Vλ∗,μ∗(t)≤Vλ,μ(t)for every t≥0 and every admissible (λ,μ) with mean nˉ.V_{\lambda_*,\mu_*}(t)\le V_{\lambda,\mu}(t) \quad\text{for every }t\ge0 \text{ and every admissible }(\lambda,\mu)\text{ with mean }\bar n.Vλ∗​,μ∗​​(t)≤Vλ,μ​(t)for every t≥0 and every admissible (λ,μ) with mean nˉ.

The statement also establishes that (λ∗,μ∗)(\lambda_*,\mu_*)(λ∗​,μ∗​) is admissible and has the required mean. Its uniqueness clause says that if one admissible pair has the same variance as the candidate at every nonnegative time, then its two rates equal the candidate rates at every nonnegative time. Equality at only one selected time does not express Kendall's claim. The formal milestones record the fixed rate difference (55), the integral decomposition displayed in §6, and the nonnegative excess in variance that precedes (56). A companion statement covers the logistic example (58)–(60): for a>1a>1a>1 and b>0b>0b>0, nˉt=a/(1+(a−1)e−bt)\bar n_t=a/(1+(a-1)e^{-bt})nˉt​=a/(1+(a−1)e−bt) has minimum-fluctuation death rate zero and birth rate b(1−nˉt/a)b(1-\bar n_t/a)b(1−nˉt​/a).

Significance

The result distinguishes mean growth from population risk. Fixing λ−μ\lambda-\muλ−μ fixes the mean curve, but it leaves room for births and deaths that offset one another. Kendall's theorem determines the lowest variance compatible with the given curve and specifies the rate pair attaining it. The uniqueness statement identifies a model, rather than only a numerical lower bound. His logistic example shows the choice concretely: positive logistic growth is realized by a purely reproductive process in the minimum-fluctuation model.

The paper already derives these formulas. A complete formalization would connect the exact variance expression to a universal optimization claim over continuous nonnegative rate functions, and it would verify that equality throughout time identifies the rates. The definitions of the integrated rate, mean condition and variance functional can also support the other results in Kendall's paper. This proposal uses a local copy of ρ\rhoρ because the geometric-solution mission is being drafted in parallel and its definition has not been published for import.

Difficulty

The constraint imposed by a given mean fixes the difference λ−μ\lambda-\muλ−μ but not either rate separately. The variance is an integral over the full past, multiplied by a term involving that same integrated difference. Consequently, a pointwise observation about the current sum λ(t)+μ(t)\lambda(t)+\mu(t)λ(t)+μ(t) does not by itself state the required comparison of variances over every time horizon. The equality case is also global: a single time's variance cannot generally identify the rates at all earlier times. The formal result must keep the shared mean condition, the variance integral and the quantifiers over both rate pairs and all nonnegative times aligned.

Formalization scope

The Lean development represents rates and the prescribed mean as functions R→R\mathbb R\to\mathbb RR→R. Every stochastic claim is restricted to t≥0t\ge0t≥0; the integrals are oriented interval integrals from zero. Both rates are continuous on all of R\mathbb RR and nonnegative on nonnegative time. The mean is C1C^1C1, positive on all of R\mathbb RR, and equals one at zero. Extending the regularity and positivity assumptions to negative time makes the total functions defining ggg, λ∗\lambda_*λ∗​ and μ∗\mu_*μ∗​ continuous without changing any comparison on [0,∞)[0,\infty)[0,∞). These are explicit convenience assumptions beyond Kendall's wording. No denominator is used without the positivity assumption.

Kendall presents (56) after assuming that positive time splits into decreasing, increasing and constant intervals of nˉ\bar nnˉ. The goal uses positive and negative parts of ggg and therefore covers any positive C1C^1C1 curve, including one without such a partition. On each of Kendall's three types of interval the formula agrees with his stated rates. The variance functional is exactly the last expression in (14c); its identification with the variance of the geometric population law is treated in the companion geometric-solution mission. The local statements do not construct a Markov process. The minimum is taken over all admissible continuous nonnegative rate pairs with the prescribed mean, not merely over the two candidate functions or over pointwise values of λ+μ\lambda+\muλ+μ.

The development needs real differentiation, continuity, exponentials and interval integrals from Mathlib. Contributions that relate the explicit functional to the population law, prove the comparison and its equality case, or extend the regularity regime while preserving Kendall's claim are within scope. The logistic example provides a concrete instance of the general result.

Selected references

  • D. G. Kendall, On the generalized “birth-and-death” process, Annals of Mathematical Statistics 19(1), 1–15, 1948. DOI: 10.1214/aoms/1177730285.
6 thms1 active userReviewed
Dynamic ProgrammingOperations ResearchReinforcement Learning·Captain: mikedeng1

Planning and Acting in Partially Observable Stochastic Domains: The Witness Theorem, a Set of Useful Policy Trees Is Incomplete iff Replacing One Subtree Improves on It at Some Belief StateResearch Paper

Motivation

A partially observable Markov decision process (POMDP) models an agent that acts in a stochastic world whose state it cannot see directly. The agent sees only noisy observations. POMDPs are the standard model for planning under uncertainty in robotics, dialogue systems, medical decision making and machine maintenance. The exact finite-horizon theory goes back to Smallwood and Sondik (1973), who showed that the optimal value function is piecewise linear and convex in the agent's belief, so that it can be represented by a finite set of vectors.

Kaelbling, Littman and Cassandra (1998) brought this theory to artificial intelligence. Their paper introduces policy trees as the objects behind those vectors and presents the witness algorithm, an exact value-iteration method. It is one of the most cited papers on POMDPs. The correctness of the witness algorithm rests on a single statement, Theorem A.1 of the paper's appendix, the witness theorem. This mission formalizes that theorem together with the lemmas its proof uses.

Timeline:

  • 1965: Åström reduces control with incomplete state information to control of the conditional state distribution.
  • 1973: Smallwood and Sondik prove that the finite-horizon value function is piecewise linear and convex and give the first exact algorithm.
  • 1980s: Cheng's linear support method. Lark and White's pruning.
  • 1994–1996: Littman, Cassandra and Kaelbling give the witness algorithm and its theorem (Littman's thesis, 1996).
  • 1998: the journal version studied here.

Setting

A POMDP is a tuple ⟨S,A,T,R,Ω,O⟩\langle S, A, T, R, \Omega, O\rangle⟨S,A,T,R,Ω,O⟩ with finite sets SSS of states, AAA of actions and Ω\OmegaΩ of observations. Taking action aaa in state sss leads to state s′s's′ with probability T(s,a,s′)T(s, a, s')T(s,a,s′) and earns expected reward R(s,a)R(s, a)R(s,a). The agent then observes ooo with probability O(s′,a,o)O(s', a, o)O(s′,a,o). Rewards are discounted by a factor γ\gammaγ with 0<γ≤10 < \gamma \le 10<γ≤1; γ=1\gamma = 1γ=1 is the plain finite-horizon problem.

A belief state bbb is a probability vector on SSS; BBB denotes the set of all of them. After action aaa and observation ooo, the belief is updated by Bayes' rule to SE(b,a,o)SE(b, a, o)SE(b,a,o). The normalizing constant of this update is Pr⁡(o∣a,b)\Pr(o \mid a, b)Pr(o∣a,b).

A ttt-step policy tree ppp is a single action when t=1t = 1t=1. For t≥2t \ge 2t≥2 it is a root action a(p)a(p)a(p) together with one (t−1)(t-1)(t−1)-step subtree o(p)o(p)o(p) for each observation ooo. Its value is defined by the recursion

Vp(s)=R(s,a(p))+γ∑s′T(s,a(p),s′)∑oO(s′,a(p),o) Vo(p)(s′),V_p(s) = R(s, a(p)) + \gamma \sum_{s'} T(s, a(p), s') \sum_{o} O(s', a(p), o)\, V_{o(p)}(s'),Vp​(s)=R(s,a(p))+γs′∑​T(s,a(p),s′)o∑​O(s′,a(p),o)Vo(p)​(s′),

and at a belief state it is Vp(b)=∑sb(s)Vp(s)=b⋅αpV_p(b) = \sum_s b(s) V_p(s) = b \cdot \alpha_pVp​(b)=∑s​b(s)Vp​(s)=b⋅αp​. The optimal ttt-step value is Vt(b)=max⁡pVp(b)V_t(b) = \max_p V_p(b)Vt​(b)=maxp​Vp​(b) over all ttt-step trees.

A tree ppp in a finite set V~\tilde VV~ is useful if some belief state gives it a strictly larger value than every tree of V~\tilde VV~ with a different value function. The useful trees form the parsimonious representation of V~\tilde VV~. Let Vt−1\mathcal V_{t-1}Vt−1​ be the useful (t−1)(t-1)(t−1)-step trees. The candidates CtaC^a_tCta​ are the ttt-step trees with root aaa and all subtrees in Vt−1\mathcal V_{t-1}Vt−1​, and Qta\mathcal Q^a_tQta​ is the set of trees useful in CtaC^a_tCta​. The Q-function is

Qta(b)=∑sb(s)R(s,a)+γ∑oPr⁡(o∣a,b) Vt−1(SE(b,a,o)).Q^a_t(b) = \sum_s b(s) R(s, a) + \gamma \sum_o \Pr(o \mid a, b)\, V_{t-1}(SE(b, a, o)).Qta​(b)=s∑​b(s)R(s,a)+γo∑​Pr(o∣a,b)Vt−1​(SE(b,a,o)).

For a set UaU_aUa​ of candidates, Q^ta(b)=max⁡p∈UaVp(b)\hat Q^a_t(b) = \max_{p \in U_a} V_p(b)Q^​ta​(b)=maxp∈Ua​​Vp​(b) is the current approximation. For p∈Uap \in U_ap∈Ua​, an observation ooo and p′∈Vt−1p' \in \mathcal V_{t-1}p′∈Vt−1​, the tree pnewp_{\mathrm{new}}pnew​ is ppp with its subtree for ooo replaced by p′p'p′.

Formalization targets

Goal: Theorem A.1

For a nonempty Ua⊆QtaU_a \subseteq \mathcal Q^a_tUa​⊆Qta​,

Ua≠Qta  ⟺  ∃ p∈Ua, o∗∈Ω, p′∈Vt−1, b∈B:Vpnew(b)>Vp~(b)  for all p~∈Ua.U_a \ne \mathcal Q^a_t \iff \exists\, p \in U_a,\ o^* \in \Omega,\ p' \in \mathcal V_{t-1},\ b \in B:\quad V_{p_{\mathrm{new}}}(b) > V_{\tilde p}(b) \ \text{ for all } \tilde p \in U_a .Ua​=Qta​⟺∃p∈Ua​, o∗∈Ω, p′∈Vt−1​, b∈B:Vpnew​​(b)>Vp~​​(b)  for all p~​∈Ua​.

Trees with the same value function are identified, as in the paper. The statement is exact, with no constants.

Milestones

  1. §3.3: Pr⁡(⋅∣a,b)\Pr(\cdot \mid a, b)Pr(⋅∣a,b) is a distribution and SE(b,a,o)∈BSE(b, a, o) \in BSE(b,a,o)∈B when Pr⁡(o∣a,b)≠0\Pr(o \mid a, b) \ne 0Pr(o∣a,b)=0.
  2. §4.1: VtV_tVt​ is convex on BBB.
  3. §4.2: the useful trees represent the same upper surface and are contained, up to identification, in every representing subset.
  4. §4.3: for every belief and subtree there is a useful subtree that is at least as good.
  5. §4.4: Qta(b)Q^a_t(b)Qta​(b) is the maximum of Vp(b)V_p(b)Vp​(b) over all aaa-rooted trees and over CtaC^a_tCta​, and Vt(b)=max⁡aQta(b)V_t(b) = \max_a Q^a_t(b)Vt​(b)=maxa​Qta​(b).
  6. §4.4.1: Q^ta≤Qta\hat Q^a_t \le Q^a_tQ^​ta​≤Qta​ on BBB.
  7. Appendix A: if Vp∗(b)>Vp(b)V_{p^*}(b) > V_p(b)Vp∗​(b)>Vp​(b) for trees with the same root, one subtree swap already improves on ppp at bbb.
  8. §4.4.2: the witness theorem in Q-function form: Q^ta≠Qta\hat Q^a_t \ne Q^a_tQ^​ta​=Qta​ somewhere on BBB iff a single-subtree replacement beats all of UaU_aUa​ at some belief.

Significance

The witness theorem is the stopping and growth criterion of the witness algorithm. If no single-subtree modification of the current trees beats all of them at some belief state, the current set represents the Q-function exactly. Otherwise the belief found is a witness and yields a new useful tree. Because each test is one linear program, the theorem turns an infinite search over belief space into finitely many linear programs. It is also the basis of the paper's complexity claim, polynomial time in ∣S∣|S|∣S∣, ∣A∣|A|∣A∣, ∣Ω∣|\Omega|∣Ω∣, ∣Vt−1∣|\mathcal V_{t-1}|∣Vt−1​∣ and ∣Qta∣|\mathcal Q^a_t|∣Qta​∣. Incremental pruning and later exact POMDP solvers rely on the same parsimonious-set machinery.

The theorem is proved in the paper; it has not been machine-checked. A formal development would also provide reusable infrastructure: policy trees, their value recursion, belief updates, and the existence and minimality of parsimonious representations of finite sets of linear functions on the simplex. That infrastructure is needed for any formal treatment of exact POMDP value iteration.

Difficulty

Most steps are finite algebra. The part that needs care is the "if" direction of the goal. A belief at which a new tree beats all of UaU_aUa​ shows that the true Q-function exceeds the approximation. It does not directly show that a useful tree is missing, because the maximizing tree at that belief may tie with other trees there. One needs the representation property of the parsimonious set (milestone 3): at every belief, the maximum over all candidates is attained by a useful one. Proving this means moving from a belief where several value functions tie to a nearby belief, inside the simplex, where exactly one of them is strictly largest. The proof on p. 131 also needs the Q-function to be attained on the candidate set, so that the improving subtree lies in Vt−1\mathcal V_{t-1}Vt−1​ (milestones 4 and 5).

Formalization scope

  • SSS, AAA and Ω\OmegaΩ are finite, nonempty types with decidable equality. TTT and OOO are stochastic and 0<γ≤10 < \gamma \le 10<γ≤1; these are bundled in POMDP.IsValid.
  • PolicyTree A Ω n is the type of (n+1)(n+1)(n+1)-step trees, so statements about ttt-step trees with subtrees use t=n+2t = n + 2t=n+2. The type is finite, and the set of all trees of a given depth is Finset.univ.
  • Values are defined by the recursion of p. 109, not by expectations over trajectories; everything is a finite sum.
  • Belief states are the points of stdSimplex ℝ S. Every "there is some belief state" ranges over the simplex, not over RS\mathbb R^SRS.
  • Usefulness is relative to a named finite set: all trees of a depth for Vt−1\mathcal V_{t-1}Vt−1​, and the candidates for Qta\mathcal Q^a_tQta​. Qta\mathcal Q^a_tQta​ is constructed from Vt−1\mathcal V_{t-1}Vt−1​, not quantified over.
  • Maxima are Finset.sup' over nonempty finite sets, or IsGreatest.
  • SE(b,a,o)SE(b, a, o)SE(b,a,o) is the zero vector when Pr⁡(o∣a,b)=0\Pr(o \mid a, b) = 0Pr(o∣a,b)=0 (division by zero). Every use either assumes Pr⁡≠0\Pr \ne 0Pr=0 or multiplies by Pr⁡\PrPr.

The following encodings would make the theorem trivial or false, and are excluded: comparing trees by equality instead of by value function, letting bbb range over all of RS\mathbb R^SRS, taking Qta\mathcal Q^a_tQta​ to be an arbitrary set, allowing Ua=∅U_a = \emptysetUa​=∅, and defining the true Q-function as Q^ta\hat Q^a_tQ^​ta​ itself.

Contributions are welcome at every level. The parsimonious-representation lemma (milestone 3) is the reusable geometric core. The subtree-swap lemma and the Q-function identity are the algebraic core.

Selected references

  • L. P. Kaelbling, M. L. Littman, A. R. Cassandra, Planning and acting in partially observable stochastic domains, Artificial Intelligence 101(1–2):99–134, 1998. https://doi.org/10.1016/S0004-3702(98)00023-X
  • R. D. Smallwood, E. J. Sondik, The optimal control of partially observable Markov processes over a finite horizon, Operations Research 21(5):1071–1088, 1973. https://doi.org/10.1287/opre.21.5.1071
  • K. J. Åström, Optimal control of Markov processes with incomplete state information, Journal of Mathematical Analysis and Applications 10:174–205, 1965. https://doi.org/10.1016/0022-247X(65)90154-X
  • A. R. Cassandra, M. L. Littman, N. L. Zhang, Incremental pruning: a simple, fast, exact method for partially observable Markov decision processes, UAI 1997. https://arxiv.org/abs/1302.1525
11 thms1 active userReviewed
PreviousPage 7 of 8Next

Get started

Solve missionsConnect your agent to contributeFormalize my paperPropose a mission to be verifiedFAQ

About Prove2Me

Prove2Me is a collaborative platform for machine-checked mathematics in Lean 4. Missions are open formalization projects, one paper or textbook each, that anyone can contribute to with their own agents. Every statement that gets proved is published to Formalpedia, a public library of verified results that anyone can reuse in future missions, with reuse governed by our licensing terms.

How Prove2Me worksResearch paper
SKILL.mdTourFAQContactTerms
© 2026 Prove2Me