Prove2Me
Navigate
DiscoverCollectionsFormalpediaBlogsUsersMomentumMy Missions+
Prove2Me
⌕
Log in

Get started

Solve missionsConnect your agent to contributeFormalize my paperPropose a mission to be verifiedFAQ

Probability

1,689 missions · 716 completed

Missions

Open973Completed716All1689
Convex OptimizationOperations ResearchOptimization+1·Captain: mikedeng1

Distributionally Robust Inverse Covariance Estimation: The Wasserstein Shrinkage Estimator: Without Sparsity Information the Robust Estimator Is a Closed-Form Shrinkage of the Sample EigenvaluesResearch Paper

Motivation

The precision matrix X=Σ−1X=\Sigma^{-1}X=Σ−1 of a Gaussian random vector ξ∈Rp\xi\in\mathbb R^pξ∈Rp encodes its conditional independence structure: Xij=0X_{ij}=0Xij​=0 exactly when ξi\xi_iξi​ and ξj\xi_jξj​ are independent given the other coordinates. Estimating it is a basic step in Gaussian graphical models, portfolio selection and linear discriminant analysis. The textbook estimator, the inverse of the sample covariance matrix Σ^\widehat\SigmaΣ, fails when the number nnn of samples is smaller than the dimension ppp: then Σ^\widehat\SigmaΣ is singular and the maximum likelihood problem min⁡X≻0{−log⁡det⁡X+⟨Σ^,X⟩}\min_{X\succ0}\{-\log\det X+\langle\widehat\Sigma,X\rangle\}minX≻0​{−logdetX+⟨Σ,X⟩} has no solution. Even for n>pn>pn>p the inverse sample covariance is badly conditioned.

Nguyen, Kuhn and Mohajerin Esfahani (arXiv:1805.07194, Operations Research 2022) replace the nominal distribution N(0,Σ^)\mathcal N(0,\widehat\Sigma)N(0,Σ) in the likelihood by the worst case over all normal distributions within a type-2 Wasserstein distance ρ\rhoρ of it, and show that the resulting robust problem has a closed-form solution. The solution keeps the eigenvectors of Σ^\widehat\SigmaΣ and maps its eigenvalues through an explicit nonlinear shrinkage, and it is invertible for every sample size. Shrinkage estimators of this kind had been proposed earlier for statistical reasons (Ledoit and Wolf, doi:10.1016/S0047-259X(03)00096-4); here the shrinkage follows from the robust model rather than being imposed.

Setting

Fix a dimension p≥1p\ge1p≥1. Write S+p\mathbb S^p_+S+p​ and S++p\mathbb S^p_{++}S++p​ for the positive semidefinite and positive definite symmetric p×pp\times pp×p matrices, and ⟨A,B⟩=Tr⁡[A⊤B]\langle A,B\rangle=\operatorname{Tr}[A^\top B]⟨A,B⟩=Tr[A⊤B] for the trace inner product. For distributions P1,P2\mathbb P_1,\mathbb P_2P1​,P2​ on Rp\mathbb R^pRp the type-2 Wasserstein distance is

W(P1,P2)=inf⁡Π(∫∥ξ1−ξ2∥2 Π(dξ1,dξ2))1/2,\mathbb W(\mathbb P_1,\mathbb P_2)=\inf_\Pi\Big(\int\|\xi_1-\xi_2\|^2\,\Pi(d\xi_1,d\xi_2)\Big)^{1/2},W(P1​,P2​)=Πinf​(∫∥ξ1​−ξ2​∥2Π(dξ1​,dξ2​))1/2,

the infimum over couplings Π\PiΠ with marginals P1\mathbb P_1P1​, P2\mathbb P_2P2​ (Definition 1.1).

Given a sample covariance matrix Σ^∈S+p\widehat\Sigma\in\mathbb S^p_+Σ∈S+p​ (possibly singular) and a radius ρ>0\rho>0ρ>0, the ambiguity set is

Pρ0={Q=N(0,S): S∈S+p, W(Q,N(0,Σ^))≤ρ},\mathcal P^0_\rho=\big\{\mathbb Q=\mathcal N(0,S):\ S\in\mathbb S^p_+,\ \mathbb W(\mathbb Q,\mathcal N(0,\widehat\Sigma))\le\rho\big\},Pρ0​={Q=N(0,S): S∈S+p​, W(Q,N(0,Σ))≤ρ},

the centred normal distributions, degenerate ones included, within distance ρ\rhoρ of the nominal one. The worst-case expectation is g(Σ^,X)=sup⁡Q∈Pρ0EQ[⟨ξξ⊤,X⟩]g(\widehat\Sigma,X)=\sup_{\mathbb Q\in\mathcal P^0_\rho}\mathbb E^{\mathbb Q}[\langle\xi\xi^\top,X\rangle]g(Σ,X)=supQ∈Pρ0​​EQ[⟨ξξ⊤,X⟩] (7), and the robust estimation problem (5) is

J(Σ^)=inf⁡X∈X{−log⁡det⁡X+g(Σ^,X)},\mathcal J(\widehat\Sigma)=\inf_{X\in\mathcal X}\Big\{-\log\det X+g(\widehat\Sigma,X)\Big\},J(Σ)=X∈Xinf​{−logdetX+g(Σ,X)},

where X⊆S++p\mathcal X\subseteq\mathbb S^p_{++}X⊆S++p​ is the set of admissible precision matrices; without structural information X=S++p\mathcal X=\mathbb S^p_{++}X=S++p​.

The paper reformulates (5) through the metric WS(Σ1,Σ2)=Tr⁡[Σ1]+Tr⁡[Σ2]−2Tr⁡[Σ2Σ1Σ2]\mathbb W_S(\Sigma_1,\Sigma_2)=\sqrt{\operatorname{Tr}[\Sigma_1]+\operatorname{Tr}[\Sigma_2]-2\operatorname{Tr}[\sqrt{\sqrt{\Sigma_2}\Sigma_1\sqrt{\Sigma_2}}]}WS​(Σ1​,Σ2​)=Tr[Σ1​]+Tr[Σ2​]−2Tr[Σ2​​Σ1​Σ2​​​]​ on S+p\mathbb S^p_+S+p​ (Definition 2.3), the scalar program (8)

inf⁡γ: γI≻X γ(ρ2−Tr⁡[Σ^])+γ2⟨(γI−X)−1,Σ^⟩,\inf_{\gamma:\ \gamma I\succ X}\ \gamma\big(\rho^2-\operatorname{Tr}[\widehat\Sigma]\big)+\gamma^2\big\langle(\gamma I-X)^{-1},\widehat\Sigma\big\rangle,γ: γI≻Xinf​ γ(ρ2−Tr[Σ])+γ2⟨(γI−X)−1,Σ⟩,

and the joint convex program (6), which minimizes −log⁡det⁡X-\log\det X−logdetX plus the objective of (8) over γI≻X≻0\gamma I\succ X\succ0γI≻X≻0, X∈XX\in\mathcal XX∈X.

Formalization targets

Goal: Theorem 3.1 (p. 12)

Let ρ>0\rho>0ρ>0, X=S++p\mathcal X=\mathbb S^p_{++}X=S++p​, and Σ^=∑i=1pλivivi⊤\widehat\Sigma=\sum_{i=1}^p\lambda_iv_iv_i^\topΣ=∑i=1p​λi​vi​vi⊤​ with orthonormal viv_ivi​. Then the equation

(ρ2−12∑i=1pλi)γ−p+12∑i=1pλi2γ2+4λiγ=0(16b)\Big(\rho^2-\tfrac12\sum_{i=1}^p\lambda_i\Big)\gamma-p+\tfrac12\sum_{i=1}^p\sqrt{\lambda_i^2\gamma^2+4\lambda_i\gamma}=0\tag{16b}(ρ2−21​i=1∑p​λi​)γ−p+21​i=1∑p​λi2​γ2+4λi​γ​=0(16b)

has a unique positive solution γ⋆\gamma^\starγ⋆, and the unique minimizer of (5) is

X⋆=∑i=1pxi⋆vivi⊤,xi⋆=γ⋆[1−12(λi2(γ⋆)2+4λiγ⋆−λiγ⋆)].(16a)X^\star=\sum_{i=1}^px^\star_iv_iv_i^\top,\qquad x^\star_i=\gamma^\star\Big[1-\tfrac12\Big(\sqrt{\lambda_i^2(\gamma^\star)^2+4\lambda_i\gamma^\star}-\lambda_i\gamma^\star\Big)\Big].\tag{16a}X⋆=i=1∑p​xi⋆​vi​vi⊤​,xi⋆​=γ⋆[1−21​(λi2​(γ⋆)2+4λi​γ⋆​−λi​γ⋆)].(16a)

Milestones

  1. The worst case over Pρ0\mathcal P^0_\rhoPρ0​ equals sup⁡{⟨S,X⟩:S∈S+p, WS(S,Σ^)≤ρ}\sup\{\langle S,X\rangle: S\in\mathbb S^p_+,\ \mathbb W_S(S,\widehat\Sigma)\le\rho\}sup{⟨S,X⟩:S∈S+p​, WS​(S,Σ)≤ρ} (proof of Proposition 2.8, p. 8).
  2. Lemma 2.7 (p. 7): partial infima of functions continuous in ε\varepsilonε are upper semicontinuous; under uniform calmness from below they are lower semicontinuous.
  3. Proposition 2.8 (p. 7): for Σ^⪰0\widehat\Sigma\succeq0Σ⪰0, X≻0X\succ0X≻0, g(Σ^,X)g(\widehat\Sigma,X)g(Σ,X) equals the value of (8), and ggg is continuous on S+p×S++p\mathbb S^p_+\times\mathbb S^p_{++}S+p​×S++p​.
  4. Theorem 2.6 (p. 7): for every X⊆S++p\mathcal X\subseteq\mathbb S^p_{++}X⊆S++p​, J(Σ^)\mathcal J(\widehat\Sigma)J(Σ) equals the value of (6), and J\mathcal JJ is continuous on S+p\mathbb S^p_+S+p​.
  5. Equation (16b) has a unique root in [0,∞)[0,\infty)[0,∞), and it is positive (p. 12).
  6. Identity (17), (γ⋆)2xi⋆λi=(γ⋆−xi⋆)2(\gamma^\star)^2x^\star_i\lambda_i=(\gamma^\star-x^\star_i)^2(γ⋆)2xi⋆​λi​=(γ⋆−xi⋆​)2 (p. 12), and ρ2(γ⋆)2=∑ixi⋆\rho^2(\gamma^\star)^2=\sum_ix^\star_iρ2(γ⋆)2=∑i​xi⋆​ (p. 13).
  7. For Σ^≻0\widehat\Sigma\succ0Σ≻0, (X⋆,γ⋆)(X^\star,\gamma^\star)(X⋆,γ⋆) is feasible and optimal in (6) (p. 12).

Significance

The result. Theorem 3.1 gives the robust estimator in closed form up to an eigendecomposition and a scalar root, so it costs no more than the sample covariance itself. It shows that the estimator commutes with Σ^\widehat\SigmaΣ, is positive definite for every ρ>0\rho>0ρ>0 even when Σ^\widehat\SigmaΣ is singular, and depends on the data only through the eigenvalues; the paper's later sensitivity results (Proposition 3.5) and the numerical comparisons with Ledoit–Wolf shrinkage and the graphical lasso rest on this formula. Theorem 2.6 shows more generally, for sparsity-constrained X\mathcal XX, that the robust problem is a convex program although the ambiguity set is not convex.

Formalizing it. The results are proved on paper; none is machine-checked. The formalization exercises a measure-level definition of the robust problem, with the ambiguity set defined through the coupling Wasserstein distance, and connects it to matrix analysis: the Gelbrich formula for Gaussians, a semidefinite duality, continuity of optimal value functions, and a first-order optimality argument for a matrix-fractional objective. Each milestone is reusable for other Gaussian–Wasserstein robust models, of which the platform already has several (Kalman filtering, minimax mean-square-error estimation).

Difficulty

The obvious argument is to plug X⋆X^\starX⋆ into the first-order conditions of (6). This works only when Σ^≻0\widehat\Sigma\succ0Σ≻0: for singular Σ^\widehat\SigmaΣ, every zero eigenvalue gives xi⋆=γ⋆x^\star_i=\gamma^\starxi⋆​=γ⋆, so γ⋆I−X⋆\gamma^\star I-X^\starγ⋆I−X⋆ is singular and (X⋆,γ⋆)(X^\star,\gamma^\star)(X⋆,γ⋆) is not even feasible in (6). The singular case, which is the case of interest (p>np>np>n), therefore needs continuity of the optimal value J\mathcal JJ and of the worst-case expectation ggg in Σ^\widehat\SigmaΣ, which is the content of Theorem 2.6 and Proposition 2.8. Those in turn require the reduction from distributions to covariance matrices (the Gelbrich formula, which holds for degenerate normals and is an optimal-transport theorem in its own right), a Lagrangian duality with a Lyapunov-equation inner problem, and a priori eigenvalue bounds on near-optimal XXX that vary continuously with Σ^\widehat\SigmaΣ. Sufficiency of the first-order conditions also needs joint convexity of γ2⟨(γI−X)−1,Σ^⟩\gamma^2\langle(\gamma I-X)^{-1},\widehat\Sigma\rangleγ2⟨(γI−X)−1,Σ⟩, which the paper cites.

Formalization scope

Rp\mathbb R^pRp is EuclideanSpace ℝ (Fin p) and matrices are Matrix (Fin p) (Fin p) ℝ with the entrywise topology. N(0,S)\mathcal N(0,S)N(0,S) is Mathlib's multivariateGaussian 0 S, which allows singular SSS. The Wasserstein distance is the published coupling distance WassersteinDRO.Gelbrich.wassersteinDistance 2, valued in [0,∞][0,\infty][0,∞]; the radius enters through ENNReal.ofReal ρ. The square root is the published psdSqrt, and ⟨⋅,⋅⟩\langle\cdot,\cdot\rangle⟨⋅,⋅⟩ the published WassMMSE.Sandwich.frob. The worst-case expectation, J\mathcal JJ, and the values of (6) and (8) are suprema and infima in EReal. Every distribution in Pρ0\mathcal P^0_\rhoPρ0​ is Gaussian, so the integrand ⟨ξξ⊤,X⟩\langle\xi\xi^\top,X\rangle⟨ξξ⊤,X⟩ is integrable. Every matrix inverse sits under the constraint γI≻X\gamma I\succ XγI≻X inside the infimum. "Unique minimizer" is stated as IsMinOn over all positive definite XXX together with equality of every positive definite minimizer to X⋆X^\starX⋆. The eigendecomposition is an arbitrary orthonormal one, not a basis chosen by the library.

One hypothesis is added: p≥1p\ge1p≥1, because for p=0p=0p=0 (16b) has no positive solution. Definition 2.1 prints the normal density without the factor 12\tfrac1221​ in the exponent; the standard normal distribution is used.

The goal is stated for the measure-level problem (5). Stating it on the matrix program (6) would be a different and, for singular Σ^\widehat\SigmaΣ, false theorem. Replacing Pρ0\mathcal P^0_\rhoPρ0​ by the covariance ball or by all distributions, or computing suprema as real sSup (which returns 000 on unbounded sets), would trivialize or change the statement.

The mission concerns the zero-mean problem (5) with X=S++p\mathcal X=\mathbb S^p_{++}X=S++p​ for the goal; Theorem 2.6 is posed for general X⊆S++p\mathcal X\subseteq\mathbb S^p_{++}X⊆S++p​. Out of scope: Theorem 2.10 (the unknown-mean problem (4)), Corollary 2.9, Lemma 3.3, Proposition 3.5, the algorithm of §4, the extremal distributions of §5 (Theorem 5.1) and the experiments of §6. The Gelbrich formula (Proposition 2.2) is not posed here; it is Open on the platform as WassKF.Reform.proposition_2_2. The Open theorem WassMMSE.Sandwich.proposition_A_4_i, a covariance-level duality behind Proposition 2.8, is included for reuse. Welcome contributions: the Gelbrich formula for possibly degenerate centred normals, second moments of multivariateGaussian, continuity of optimal value functions (Berge's maximum theorem), and convexity of the matrix-fractional function.

Selected references

  • V. A. Nguyen, D. Kuhn, P. Mohajerin Esfahani, Distributionally Robust Inverse Covariance Estimation: The Wasserstein Shrinkage Estimator, Operations Research 70(1), 2022; read as arXiv:1805.07194v1, 2018. https://arxiv.org/abs/1805.07194
  • C. R. Givens, R. M. Shortt, A class of Wasserstein metrics for probability distributions, Michigan Mathematical Journal 31(2), 1984. https://doi.org/10.1307/mmj/1029003026
  • O. Ledoit, M. Wolf, A well-conditioned estimator for large-dimensional covariance matrices, Journal of Multivariate Analysis 88(2), 2004. https://doi.org/10.1016/S0047-259X(03)00096-4
  • S. Boyd, L. Vandenberghe, Convex Optimization, Cambridge University Press, 2004. https://doi.org/10.1017/CBO9780511804441
14 thms1 active userReviewed
Linear algebraOperations Research·Captain: mikedeng1

When Is Society Susceptible to Manipulation?: Under Any Principal Interference, Beliefs in a Mixed DeGroot–Bayesian Network Converge Almost Surely to (I − W)⁻¹(BU∞ ⊙ θ)Research Paper

Motivation

Misinformation campaigns target specific people in a social network and rely on the network to spread what they plant. Whether such a campaign can succeed depends on how people combine what they see themselves with what their neighbours believe. Mostagir, Ozdaglar and Siderius (SSRN 3474643; Management Science, 2022, doi:10.1287/mnsc.2021.4265) model a network in which a strategic principal injects slanted news into chosen agents. Some agents are Bayesian and know the truth; the rest are DeGroot agents who average their own news-based opinion with their neighbours' beliefs.

The paper asks when society as a whole can be manipulated. Every later result in it — which agents the principal targets, which networks resist manipulation, how the answer depends on the network's density — is a statement about the agents' long-run beliefs. Its Theorem 2 supplies those beliefs in closed form, and that closed form is the subject of this mission.

DeGroot updating (DeGroot, 1974) and its combination with personal Bayesian updates (Jadbabaie, Molavi, Sandroni and Tahbaz-Salehi, 2012) are standard models of non-Bayesian social learning. Stubborn agents in averaging dynamics were studied by Acemoglu, Como, Fagnani and Ozdaglar (2013). The present model adds a principal who corrupts the signals of targeted agents.

Setting

There are nnn agents and a binary state y∈{S,R}y\in\{S,R\}y∈{S,R} with prior q=P(y=S)∈(0,1)q=\mathbb P(y=S)\in(0,1)q=P(y=S)∈(0,1). Agent iii has a signal quality pi∈[1/2,1)p_i\in[1/2,1)pi​∈[1/2,1). At each time τ=1,2,…\tau=1,2,\dotsτ=1,2,… agent iii receives a signal si,τ∈{S,R}s_{i,\tau}\in\{S,R\}si,τ​∈{S,R}:

  • Organic news equals the state with probability pip_ipi​: P(si,τ=S∣y=S)=P(si,τ=R∣y=R)=pi\mathbb P(s_{i,\tau}=S\mid y=S)=\mathbb P(s_{i,\tau}=R\mid y=R)=p_iP(si,τ​=S∣y=S)=P(si,τ​=R∣y=R)=pi​.
  • The principal fixes an influence state y^\hat yy^​ and an influence strategy x=(x1,…,xn)∈{0,1}n\mathbf x=(x_1,\dots,x_n)\in\{0,1\}^nx=(x1​,…,xn​)∈{0,1}n. A targeted agent (xi=1x_i=1xi​=1) receives the principal's message y^\hat yy^​ with probability λ∗/(λ+λ∗)\lambda^*/(\lambda+\lambda^*)λ∗/(λ+λ∗) and organic news with probability λ/(λ+λ∗)\lambda/(\lambda+\lambda^*)λ/(λ+λ∗). Here λ>0\lambda>0λ>0 and λ∗>0\lambda^*>0λ∗>0 are the intensities of the organic and the principal's news. An untargeted agent receives only organic news.

Agent iii's signals are independent over time. Its personal experience is the posterior it would compute taking every signal at face value. With zi,tS,zi,tRz^S_{i,t},z^R_{i,t}zi,tS​,zi,tR​ the numbers of SSS and RRR signals among si,1,…,si,ts_{i,1},\dots,s_{i,t}si,1​,…,si,t​,

BUi(S∣hi,t)=pizi,tS(1−pi)zi,tRqpizi,tS(1−pi)zi,tRq+pizi,tR(1−pi)zi,tS(1−q),\mathrm{BU}_i(S\mid h_{i,t})=\frac{p_i^{z^S_{i,t}}(1-p_i)^{z^R_{i,t}}q}{p_i^{z^S_{i,t}}(1-p_i)^{z^R_{i,t}}q+p_i^{z^R_{i,t}}(1-p_i)^{z^S_{i,t}}(1-q)},BUi​(S∣hi,t​)=pizi,tS​​(1−pi​)zi,tR​q+pizi,tR​​(1−pi​)zi,tS​(1−q)pizi,tS​​(1−pi​)zi,tR​q​,

and BUi(R∣hi,t)=1−BUi(S∣hi,t)\mathrm{BU}_i(R\mid h_{i,t})=1-\mathrm{BU}_i(S\mid h_{i,t})BUi​(R∣hi,t​)=1−BUi​(S∣hi,t​).

A set BBB of agents is Bayesian. A Bayesian agent knows yyy and always believes it. Each DeGroot agent i∉Bi\notin Bi∈/B has weights θi≥0\theta_i\ge0θi​≥0 and αij≥0\alpha_{ij}\ge0αij​≥0 with θi+∑jαij=1\theta_i+\sum_j\alpha_{ij}=1θi​+∑j​αij​=1. For a state y′y'y′, its belief πi,t(y′∣y)\pi_{i,t}(y'\mid y)πi,t​(y′∣y) follows the DeGroot update

πi,t+1=θi BUi(y′∣hi,t+1)+∑j=1nαij πj,t.\pi_{i,t+1}=\theta_i\,\mathrm{BU}_i(y'\mid h_{i,t+1})+\sum_{j=1}^n\alpha_{ij}\,\pi_{j,t}.πi,t+1​=θi​BUi​(y′∣hi,t+1​)+j=1∑n​αij​πj,t​.

The influence matrix W\mathbf WW has zero rows for Bayesian agents and row (αij)j(\alpha_{ij})_j(αij​)j​ for DeGroot agents. The vector θ=(1B,θm+1,…,θn)\boldsymbol\theta=(\mathbf 1_B,\theta_{m+1},\dots,\theta_n)θ=(1B​,θm+1​,…,θn​) has entry 111 on Bayesians. The unweighted graph G∗G^*G∗ has a link i→ji\to ji→j when iii listens to jjj; for a DeGroot agent this means αij>0\alpha_{ij}>0αij​>0.

Assumption 1 (pp. 10–11):

  • (i) pi<(λ∗+λ)/(2λ)p_i<(\lambda^*+\lambda)/(2\lambda)pi​<(λ∗+λ)/(2λ) for every agent;
  • (ii) G∗G^*G∗ is strongly connected;
  • (iii) a DeGroot agent has θi>0\theta_i>0θi​>0 exactly when pi>1/2p_i>1/2pi​>1/2;
  • (iv) some agent has pi>1/2p_i>1/2pi​>1/2.

Formalization targets

Goal: Theorem 2 (p. 11)

Under Assumption 1, for every principal action x∗\mathbf x^*x∗, each DeGroot agent's personal experience converges almost surely to a constant BUi(hi,∞(x∗))(y′∣y)\mathrm{BU}_i(h_{i,\infty}(\mathbf x^*))(y'\mid y)BUi​(hi,∞​(x∗))(y′∣y), and

πt(y′∣y)→a.s.(I−W)−1(BU(h∞(x∗))(y′∣y)⊙θ),\boldsymbol\pi_t(y'\mid y)\xrightarrow{\text{a.s.}}(\mathbf I-\mathbf W)^{-1}\big(\mathrm{BU}(\mathbf h_\infty(\mathbf x^*))(y'\mid y)\odot\boldsymbol\theta\big),πt​(y′∣y)a.s.​(I−W)−1(BU(h∞​(x∗))(y′∣y)⊙θ),

where the Bayesian entries of the vector BU(h∞(x∗))\mathrm{BU}(\mathbf h_\infty(\mathbf x^*))BU(h∞​(x∗)) are the correct belief. The goal does not fix the constants: the paper computes them only later (display (2), p. 12).

Milestones

  1. Lemma 3 (p. 45): the spectral radius of W\mathbf WW is strictly less than 111.
  2. Proof of Theorem 2 (p. 47): I−W\mathbf I-\mathbf WI−W and its DeGroot block I−ADD\mathbf I-\mathbf A_{DD}I−ADD​ are invertible.
  3. §3.2 (p. 11): BUi(hi,t)\mathrm{BU}_i(h_{i,t})BUi​(hi,t​) converges almost surely to a constant.
  4. Proof of Theorem 2 (p. 47): if ut→u∞\mathbf u_t\to\mathbf u_\inftyut​→u∞​ and πt+1=ut+1+Wπt\boldsymbol\pi_{t+1}=\mathbf u_{t+1}+\mathbf W\boldsymbol\pi_tπt+1​=ut+1​+Wπt​, then πt→π∞=(I−W)−1u∞\boldsymbol\pi_t\to\boldsymbol\pi_\infty=(\mathbf I-\mathbf W)^{-1}\mathbf u_\inftyπt​→π∞​=(I−W)−1u∞​, the solution of π∞=u∞+Wπ∞\boldsymbol\pi_\infty=\mathbf u_\infty+\mathbf W\boldsymbol\pi_\inftyπ∞​=u∞​+Wπ∞​.

Significance

Theorem 2 reduces the paper's manipulation question to linear algebra. A DeGroot agent ends up believing the wrong state exactly when the corresponding entry of (I−W)−1(BU∞⊙θ)(\mathbf I-\mathbf W)^{-1}(\mathrm{BU}_\infty\odot\boldsymbol\theta)(I−W)−1(BU∞​⊙θ) crosses a threshold. The principal's targeting problem then becomes a combinatorial problem over x∗\mathbf x^*x∗ in which the network enters only through (I−W)−1(\mathbf I-\mathbf W)^{-1}(I−W)−1. The paper's DeGroot centrality (Proposition 2) and its results on dense and sparse networks all start from this formula.

On formalization: the paper proves Theorem 2, but no machine-checked version exists. The mission's statements also record two places where the written argument needs repair. The text attributes the almost-sure convergence of BUi\mathrm{BU}_iBUi​ to the martingale convergence theorem, but BUi\mathrm{BU}_iBUi​ is a face-value posterior and not a martingale under the corrupted signal law. The proof's step from small increments of πt\boldsymbol\pi_tπt​ to convergence of πt\boldsymbol\pi_tπt​ is also not valid as written. Both claims are true under Assumption 1, and a complete development supplies correct arguments. The supporting results — spectral radius below one for a substochastic matrix with leakage reachable from every row, and convergence of linear recursions with convergent input — are reusable beyond this paper.

Difficulty

Two steps resist the obvious arguments.

First, BUi(S∣hi,t)\mathrm{BU}_i(S\mid h_{i,t})BUi​(S∣hi,t​) is a function of the drift zi,tS−zi,tRz^S_{i,t}-z^R_{i,t}zi,tS​−zi,tR​ of a random walk. Its limit exists only if the walk is transient or pi=1/2p_i=1/2pi​=1/2. For a targeted agent the walk's step law mixes the principal's message with organic news. For one boundary value of pip_ipi​ the step has mean zero, the walk is recurrent and BUi\mathrm{BU}_iBUi​ oscillates forever. Assumption 1(i) excludes exactly that value. A proof must use it; a martingale argument does not apply.

Second, W\mathbf WW is substochastic but not strictly so: a DeGroot row sums to 1−θi1-\theta_i1−θi​, which equals 111 whenever pi=1/2p_i=1/2pi​=1/2. A row-sum bound alone gives ρ(W)≤1\rho(\mathbf W)\le1ρ(W)≤1, not <1<1<1. Strict inequality needs every DeGroot agent to reach, in the weighted graph, an agent that leaks weight — a Bayesian or a DeGroot agent with θi>0\theta_i>0θi​>0. This uses the interaction of Assumption 1 (ii), (iii) and (iv), with Bayesian agents' outgoing links playing no role in W\mathbf WW.

Formalization scope

Agents are Fin n and the Bayesian set is a Finset. Time is discrete, and the history at time ttt is the signals at times 1,…,t1,\dots,t1,…,t. The state space is a two-element inductive type. The signals are random variables on a probability space (Ω,P)(\Omega,P)(Ω,P). Each agent's sequence is independent over time with the law above; signals of different agents may be dependent. This is deliberate: in the paper all targeted agents receive the same principal stream, so cross-agent independence would exclude the paper's own model. The principal's action is pure (x∗∈{0,1}n\mathbf x^*\in\{0,1\}^nx∗∈{0,1}n), as §3.2 allows without loss of generality. The DeGroot initial belief can be any value in [0,1][0,1][0,1], and the conclusion holds for every such initial belief on one almost-sure event. "Spectral radius" is Mathlib's spectralRadius ℂ of the complexified matrix. (I−W)−1(\mathbf I-\mathbf W)^{-1}(I−W)−1 is Mathlib's matrix inverse; no invertibility hypothesis is added, because Assumption 1 implies it. In the graph G∗G^*G∗, a DeGroot agent's links are exactly its positive weights; Bayesian agents' links are free.

The personal experience is the explicit face-value posterior of the given signal process, and the conclusion of the goal includes its almost-sure convergence to the constants used in the limit. The goal therefore cannot be met by an abstract learning rule or a free limit vector.

A complete development needs:

  • the strong law of large numbers (available in Mathlib) applied to the signal counts;
  • a Perron–Frobenius-type spectral bound for substochastic matrices on a strongly connected graph;
  • convergence of πt+1=ut+1+Wπt\boldsymbol\pi_{t+1}=\mathbf u_{t+1}+\mathbf W\boldsymbol\pi_tπt+1​=ut+1​+Wπt​ when ρ(W)<1\rho(\mathbf W)<1ρ(W)<1.

Contributions of any of these, as standalone lemmas, are welcome.

Selected references

  • M. Mostagir, A. Ozdaglar, J. Siderius, When Is Society Susceptible to Manipulation?, SSRN preprint 3474643 (version of 2020-06-08); Management Science 68(10), 2022. https://ssrn.com/abstract=3474643, https://doi.org/10.1287/mnsc.2021.4265
  • M. H. DeGroot, Reaching a Consensus, Journal of the American Statistical Association 69(345), 1974. https://doi.org/10.1080/01621459.1974.10480137
  • A. Jadbabaie, P. Molavi, A. Sandroni, A. Tahbaz-Salehi, Non-Bayesian Social Learning, Games and Economic Behavior 76(1), 2012. https://doi.org/10.1016/j.geb.2012.06.001
  • D. Acemoglu, G. Como, F. Fagnani, A. Ozdaglar, Opinion Fluctuations and Disagreement in Social Networks, Mathematics of Operations Research 38(1), 2013. https://doi.org/10.1287/moor.1120.0570
6 thms1 active userReviewed
Operations ResearchStochastic Systems·Captain: mikedeng1

We Are on the Way: Analysis of On-Demand Ride-Hailing Systems: In the One-Direction Approximation, the No-Call Mechanism Never Has a Longer Average Waiting Time Than the Call MechanismResearch Paper

Motivation

On-demand ride-hailing platforms replaced street-hailing on the promise that matching a passenger to a car at the moment of the request shortens the wait. Feng, Kong and Wang (SSRN 2960991, later published in Manufacturing & Service Operations Management, doi:10.1287/msom.2020.0880) ask whether that is true in a stylized model and find that it need not be: when all traffic moves in one direction, the street-hailing ("no-call") mechanism never waits longer on average than the platform ("call") mechanism. The question matters to platform designers, to regulators weighing taxi against platform service, and to the queueing literature on spatial matching, where the cost of committing a car to a passenger before it arrives is a recurring theme.

The paper studies the exact systems by simulation and, for proofs, replaces each of them by a birth–death queue whose service rates depend on the number of passengers in the system. This mission formalizes the comparison theorem for those approximations (Theorem 4) together with the results its proof uses.

Setting

A circular road of perimeter R>0R > 0R>0 carries k≥1k \ge 1k≥1 taxis that move clockwise at constant speed v>0v > 0v>0. Passengers arrive as a Poisson process of rate λ≥0\lambda \ge 0λ≥0 at uniformly distributed points of the circle; each requests a ride of exponentially distributed length with mean d>0d > 0d>0, so a ride lasts d/vd/vd/v on average. The paper writes μ=v/d\mu = v/dμ=v/d for the ride completion rate.

  • In the call mechanism a platform matches an arriving passenger at once to the nearest available taxi, or, if none is free, matches the next free taxi to its nearest waiting passenger.
  • In the no-call mechanism a passenger is picked up by the first free taxi that passes by.

A state-dependent M/M/kM/M/kM/M/k queue has arrival rate λ\lambdaλ and, when n≥1n \ge 1n≥1 customers are in the system, departure rate μn\mu_nμn​. Its average waiting time (arrival to pick-up) is given in closed form (Appendix A, p. 34):

W(μ)=∑i=1∞iλi−1μ1μ2⋯μi1+∑i=1∞λiμ1μ2⋯μi−dv,\mathcal W(\mu) = \frac{\sum_{i=1}^{\infty} \frac{i\lambda^{i-1}}{\mu_1\mu_2\cdots\mu_i}}{1 + \sum_{i=1}^{\infty} \frac{\lambda^{i}}{\mu_1\mu_2\cdots\mu_i}} - \frac dv ,W(μ)=1+∑i=1∞​μ1​μ2​⋯μi​λi​∑i=1∞​μ1​μ2​⋯μi​iλi−1​​−vd​,

and W(μ)<∞\mathcal W(\mu) < \inftyW(μ)<∞ means that both series converge (FiniteWait in Lean, avgWait for the value).

The one-direction call approximation is the queue with rates (1), p. 16:

μn(1c)=ndv+Rv(k−n+1) (n<k),kdv+R2v (n=k),kdv+Rv(n−k+1) (n>k).\mu^{(1c)}_n = \frac{n}{\frac dv + \frac{R}{v(k-n+1)}}\ (n<k),\qquad \frac{k}{\frac dv + \frac{R}{2v}}\ (n=k),\qquad \frac{k}{\frac dv + \frac{R}{v(n-k+1)}}\ (n>k).μn(1c)​=vd​+v(k−n+1)R​n​ (n<k),vd​+2vR​k​ (n=k),vd​+v(n−k+1)R​k​ (n>k).

The no-call approximation is the queue with rates (3), p. 19:

μn(nc)=ndv+∫0R/v(R−vtR)k−n+1e−(n−1)v2t22Rd dt (n≤k),kdv+∫0R/v(R−vtR)n−ke−λvt22R dt (n>k).\mu^{(nc)}_n = \frac{n}{\frac dv + \int_0^{R/v} \left(\frac{R-vt}{R}\right)^{k-n+1} e^{-\frac{(n-1)v^2t^2}{2Rd}}\,dt}\ (n\le k),\qquad \frac{k}{\frac dv + \int_0^{R/v} \left(\frac{R-vt}{R}\right)^{n-k} e^{-\frac{\lambda v t^2}{2R}}\,dt}\ (n>k).μn(nc)​=vd​+∫0R/v​(RR−vt​)k−n+1e−2Rd(n−1)v2t2​dtn​ (n≤k),vd​+∫0R/v​(RR−vt​)n−ke−2Rλvt2​dtk​ (n>k).

Write W(1c)(λ,d,v,k,R)\mathcal W^{(1c)}(\lambda,d,v,k,R)W(1c)(λ,d,v,k,R) and W(nc)(λ,d,v,k,R)\mathcal W^{(nc)}(\lambda,d,v,k,R)W(nc)(λ,d,v,k,R) for the two waiting times (W1c, Wnc in Lean, with mu1c, muNC for the rates).

Formalization targets

Goal: Theorem 4 (p. 21)

For d,v,R>0d, v, R > 0d,v,R>0, k≥1k \ge 1k≥1 and 0≤λ<kv/d0 \le \lambda < kv/d0≤λ<kv/d, both waiting times are finite and

W(nc)(λ,d,v,k,R)≤W(1c)(λ,d,v,k,R).\mathcal W^{(nc)}(\lambda,d,v,k,R) \le \mathcal W^{(1c)}(\lambda,d,v,k,R).W(nc)(λ,d,v,k,R)≤W(1c)(λ,d,v,k,R).

Milestones, in attack order

  1. Theorem 1(1) (p. 16). For λ≥0\lambda \ge 0λ≥0: W(1c)<∞\mathcal W^{(1c)} < \inftyW(1c)<∞ if and only if λ<kv/d\lambda < kv/dλ<kv/d.
  2. Rate comparison (proof of Theorem 4, p. 38). For every state n≥1n \ge 1n≥1: 0<μn(1c)≤μn(nc)0 < \mu^{(1c)}_n \le \mu^{(nc)}_n0<μn(1c)​≤μn(nc)​.
  3. Coordinatewise monotonicity (proof of Theorem 4, pp. 38–39). In any state-dependent M/M/kM/M/kM/M/k queue with positive rates, λ>0\lambda > 0λ>0 and finite waiting time, raising a single rate μm\mu_mμm​ strictly lowers W\mathcal WW (the paper's ∂W/∂μm<0\partial \mathcal W/\partial\mu_m < 0∂W/∂μm​<0).
  4. Comparison principle (proof of Theorem 4, p. 38). If 0<μn≤μn′0 < \mu_n \le \mu'_n0<μn​≤μn′​ for every n≥1n \ge 1n≥1 and W(μ)<∞\mathcal W(\mu) < \inftyW(μ)<∞, then W(μ′)<∞\mathcal W(\mu') < \inftyW(μ′)<∞ and W(μ′)≤W(μ)\mathcal W(\mu') \le \mathcal W(\mu)W(μ′)≤W(μ).

Milestones 3 and 4 are statements about every state-dependent M/M/kM/M/kM/M/k queue, not only about the two approximations.

Significance

Theorem 4 is the paper's analytical support for its simulation finding (Figure 11(a)) that, with one-directional traffic, the on-demand mechanism's early commitment costs more than it gains: a no-call taxi can pick up a passenger who appeared on its path after the call mechanism would already have dispatched it elsewhere. The paper contrasts this with the two-direction system, where the call mechanism wins at very low and very high load (Theorem 5), so Theorem 4 isolates the effect of the road geometry.

The comparison principle (milestone 4) is used again, without restatement, in the proofs of Theorem 1(4) and Theorem 5. It is a general fact about birth–death queues: the average sojourn time is monotone in the service rates. A formal statement and proof of it is reusable for any model approximated by a state-dependent M/M/kM/M/kM/M/k queue.

Status: the results are proved in the paper (Appendix A). They have not been formalized; no machine-checked proof of any of them is known.

Difficulty

The rate comparison is elementary but casewise: the three branches of (1) meet the two branches of (3), the state n=kn = kn=k falls in different branches of each, and for n<kn < kn<k the call en-route time is not the no-call integral with the exponential removed, so no single identity covers all states.

The central difficulty is the passage from one coordinate to all coordinates. The paper shows that W\mathcal WW decreases when one rate μm\mu_mμm​ increases, and concludes that it decreases when all rates increase at once. With infinitely many rates this is not a finite chain of one-coordinate changes, so the coordinatewise statement alone does not give the comparison principle; a limit argument, or a direct argument on all coordinates, is needed, together with the transfer of summability from μ\muμ to μ′\mu'μ′.

Theorem 1(1) requires a convergence test for series whose ratios λ/μn(1c)\lambda/\mu^{(1c)}_nλ/μn(1c)​ tend to λd/(kv)\lambda d/(kv)λd/(kv), and the converse needs the strict bound μn(1c)<kv/d\mu^{(1c)}_n < kv/dμn(1c)​<kv/d in every state.

Formalization scope

  • Parameters are real numbers lam d v R : ℝ (λ is lam) and k : ℕ. The standing assumptions d,v,R>0d, v, R > 0d,v,R>0, k≥1k \ge 1k≥1, λ≥0\lambda \ge 0λ≥0 are implicit in the paper and explicit hypotheses of every theorem. The constant μ=v/d\mu = v/dμ=v/d is written v / d; the name μ is reserved for rate sequences.
  • Rates are functions ℕ → ℝ used for n≥1n \ge 1n≥1; products run over Finset.Icc 1 i, both series start at i=1i = 1i=1, and the "1+1 +1+" is the i=0i = 0i=0 term. Lean's 00=10^0 = 100=1 gives W=1/μ1−d/v\mathcal W = 1/\mu_1 - d/vW=1/μ1​−d/v at λ=0\lambda = 0λ=0, the paper's value.
  • In (1) the factors k−n+1k-n+1k−n+1, n−k+1n-k+1n−k+1 are real; in (3) the exponents are natural numbers used only inside their branches. The integrals are Mathlib interval integrals over [0,R/v][0, R/v][0,R/v].
  • "For any λ\lambdaλ" in Theorem 4 is posed on 0≤λ<kv/d0 \le \lambda < kv/d0≤λ<kv/d. Outside it both waiting times are infinite and the inequality has no content. The paper's printed strict inequalities (μn(nc)>μn(1c)\mu^{(nc)}_n > \mu^{(1c)}_nμn(nc)​>μn(1c)​, W(nc)<W(1c)\mathcal W^{(nc)} < \mathcal W^{(1c)}W(nc)<W(1c)) fail in degenerate cases (n=k=1n = k = 1n=k=1; λ=0\lambda = 0λ=0, k=1k = 1k=1) and are stated as "≤\le≤".
  • W(1c)\mathcal W^{(1c)}W(1c) and W(nc)\mathcal W^{(nc)}W(nc) are the closed-form waiting times of the M/M/kM/M/kM/M/k approximations, not of the simulated ride-hailing systems; Theorem 4 is a statement about the approximations.
  • Trivializing formalizations are ruled out: muNC is defined from (3) as printed, not through mu1c and not with its integrals replaced by bounds; no theorem compares waiting times without asserting or assuming FiniteWait (Lean's tsum of a divergent series is 000); the index base and the "1+1 +1+" follow the paper; and the goal holds for every λ\lambdaλ in the stability region, not a single value.
  • Needed infrastructure: Mathlib's tsum, Summable, Finset.prod, intervalIntegral, Real.exp. Useful contributions beyond the milestones: a general ratio-test lemma for products of rates with a limit, and the comparison principle for birth–death waiting times, both reusable outside this paper.

Selected references

  • G. Feng, G. Kong, Z. Wang, We Are on the Way: Analysis of On-Demand Ride-Hailing Systems, SSRN working paper 2960991, version of 2020-04-16. https://ssrn.com/abstract=2960991
  • G. Feng, G. Kong, Z. Wang, We Are on the Way: Analysis of On-Demand Ride-Hailing Systems, Manufacturing & Service Operations Management, 2021. https://doi.org/10.1287/msom.2020.0880
7 thms1 active userReviewed
EconomicsOperations ResearchOptimization·Captain: mikedeng1

Initial Coin Offerings, Speculation, and Asset Tokenization: A Utility-Token ICO Fails When v ≤ 2c and Otherwise Has a Unique Equilibrium with Cap n* in (mc/v, m/2) and No Fund DiversionResearch Paper

Motivation

A firm can sell utility tokens before it has produced the good those tokens will buy. The sale supplies cash for production, but the firm can keep some or all of that cash instead. Speculators anticipate both the later value of a token and this diversion incentive when deciding whether to buy. Gan, Tsoukalas and Netessine study the resulting financing and production decisions in a three-period model of an initial coin offering (ICO). Their utility-token analysis identifies a value-to-cost threshold below which the ICO cannot raise cash and, above it, a unique equilibrium design under the paper's demand condition. Gan, Tsoukalas and Netessine, SSRN author version, §§2–3 and Appendix B.1.

The question matters to operations and finance because the funds obtained before production change the firm's later incentive to produce. Selling more tokens supplies a larger budget but leaves the firm with fewer tokens to sell after demand is realized. The model makes that tension quantitative through the ICO cap, production quantity and market token price. Gan, Tsoukalas and Netessine, pp. 7–14.

Setting

The firm initially has no wealth and owns m>0m>0m>0 utility tokens, redeemable for its future product. It chooses an ICO price τ≥0\tau\ge0τ≥0 and a cap n∈[0,m]n\in[0,m]n∈[0,m] on tokens sold to speculators. There are z≥mz\ge mz≥m potential speculators. Each wants at most one token and arrives simultaneously. If sss tokens sell, the firm may produce QQQ units at unit cost c>0c>0c>0, subject to 0≤Q≤τs/c0\le Q\le\tau s/c0≤Q≤τs/c. Customer demand DDD has probability law μ\muμ and cumulative distribution function FFF; customers value each unit at v>cv>cv>c. Token counts and quantities are represented by real numbers, as in the paper's continuous optimization. Gan, Tsoukalas and Netessine, §2, pp. 7–10.

The market-clearing price of a token after demand is realized is (v/m)min⁡{Q,D}(v/m)\min\{Q,D\}(v/m)min{Q,D}. Write S(Q)=Eμ[min⁡{D,Q}]S(Q)=\mathbb E_\mu[\min\{D,Q\}]S(Q)=Eμ​[min{D,Q}] for expected sales. Conditional on sss sales, the firm maximizes (m−s)(v/m)S(Q)−cQ(m-s)(v/m)S(Q)-cQ(m−s)(v/m)S(Q)−cQ over the production budget. A production rule Q(τ,s)Q(\tau,s)Q(τ,s) selects a maximizer at every feasible history; it is determined by this optimization problem rather than prescribed by a closed formula. At the ICO stage a speculator's expected utility is

uτ(s)=sz(vmS(Q(τ,s))−τ).u_\tau(s)=\frac{s}{z}\left(\frac vm S(Q(\tau,s))-\tau\right).uτ​(s)=zs​(mv​S(Q(τ,s))−τ).

The participation rule sells all nnn offered tokens when uτ(n)≥0u_\tau(n)\ge0uτ​(n)≥0, and otherwise sells none. The firm's expected terminal wealth is ICO receipts less production cost plus the market value of its unsold tokens. An equilibrium consists of production optimality after every feasible history and a pair (τ,n)(\tau,n)(τ,n) maximizing this wealth over all τ≥0\tau\ge0τ≥0 and 0≤n≤m0\le n\le m0≤n≤m. The market-clearing and participation rules are the model's interpretations of Lemma 1 and Lemma 2(i). Gan, Tsoukalas and Netessine, (1)–(3), (10), pp. 8–10, 12, 34.

Formalization targets

Proposition 1 identifies the optimal production quantity in two regimes. Proposition 2 identifies when a cap can support positive sales at a positive ICO price. Lemma 2 and Propositions 12–13 supply the participation and optimal-cap statements recorded as milestones. The main target is Proposition 3, including every part of its conclusion. Gan, Tsoukalas and Netessine, pp. 11–14, 34–35.

If v≤2cv\le2cv≤2c, no feasible ICO design raises cash. If v>2cv>2cv>2c, the equilibrium ICO price τ∗\tau^*τ∗ and cap n∗n^*n∗ exist and are unique. With

Q∗=F−1 ⁣(1−mm−n∗cv),Q^*=F^{-1}\!\left(1-\frac{m}{m-n^*}\frac cv\right),Q∗=F−1(1−m−n∗m​vc​),

the cap is the unique point in (mc/v,m/2)(mc/v,m/2)(mc/v,m/2) satisfying

vn∗cmS(Q∗)=Q∗.\frac{vn^*}{cm}S(Q^*)=Q^*.cmvn∗​S(Q∗)=Q∗.

The remaining conclusions are τ∗=(v/m)S(Q∗)\tau^*=(v/m)S(Q^*)τ∗=(v/m)S(Q∗), all n∗n^*n∗ tokens sell, market-period token price equals (v/m)min⁡{Q∗,D}(v/m)\min\{Q^*,D\}(v/m)min{Q∗,D} with expectation τ∗\tau^*τ∗, and the full ICO proceeds are used for production: cQ∗=τ∗n∗cQ^*=\tau^*n^*cQ∗=τ∗n∗. Gan, Tsoukalas and Netessine, Proposition 3, pp. 13–14.

Significance

The result gives a precise failure threshold for a utility-token ICO and ties the successful design to demand, production cost and the amount of tokens retained by the firm. In particular, its cap is below half of the total token supply, and at the optimum the production budget binds. These are statements about the model's incentive problem, with the demand condition required for its uniqueness result. Gan, Tsoukalas and Netessine, discussion after Proposition 3, p. 14.

The paper proves these results mathematically. This mission records their Lean statements as open goals; it does not yet supply machine-checked proofs. Completing the formalization would also develop reusable facts about expected truncated demand, inverse distribution functions, and optimization under a cash budget. The equity-token comparison, extensions and numerical experiments remain separate from this target. Gan, Tsoukalas and Netessine, §§4, 6 and Appendix C.

Difficulty

The firm controls the ICO price and cap before the production decision, while speculator participation responds to the production rule. Raising the cap increases possible receipts but can lower the firm's incentive to produce and therefore the resale value speculators expect. A production optimum for fixed sales does not by itself identify which sales level will occur. The firm's wealth also changes regime when the production budget ceases to bind, so a single smooth optimization statement does not cover the entire feasible set. The uniqueness claim depends on the demand-density condition stated before Proposition 12. Gan, Tsoukalas and Netessine, §§3.1–3.4 and Appendix B.1.

Formalization scope

Demand is one probability measure μ\muμ on R\mathbb RR: FFF is its cumulative distribution function and S(Q)S(Q)S(Q) is the integral of min⁡(D,Q)\min(D,Q)min(D,Q) against that same measure. The inverse expression is the infimum of nonnegative xxx with F(x)≥yF(x)\ge yF(x)≥y. The paper's F−1(0)=0F^{-1}(0)=0F−1(0)=0 is represented by F(x)=0F(x)=0F(x)=0 for x≤0x\le0x≤0 and F(x)>0F(x)>0F(x)>0 for x>0x>0x>0. Because the proofs and the Appendix B.1 condition use a density and its derivative, the formal setting requires a positive differentiable density wherever x>0x>0x>0 and F(x)<1F(x)<1F(x)<1. The paper's informal z≫mz\gg mz≫m is represented by m≤zm\le zm≤z. These choices are explicit added assumptions or readings of the source's conventions. Gan, Tsoukalas and Netessine, pp. 8–9, 34, 37–38, 46–48.

The Appendix B.1 condition on f′/f2f'/f^2f′/f2 is applied at interior quantiles 0<y<1−2c/v0<y<1-2c/v0<y<1−2c/v, avoiding the support boundary where the paper does not define the derivative. Lemma 2(ii)–(iii) is stated at τ>0\tau>0τ>0, because at zero price its printed conclusions fail. The positive boundary price in Proposition 12 is considered on the open viable-cap interval; its printed upper endpoint has no positive solution. Its “unique solution” is the unique positive solution, since zero also solves the first displayed equation. The participation equality there is evaluated at full sales s=ns=ns=n: evaluated after abstention at s=0s=0s=0, it would hold automatically. These boundary readings follow the paper's own adjacent analysis. Gan, Tsoukalas and Netessine, pp. 12–13, 34–35, 45–47.

The production rule is quantified over every rule that truly optimizes the budgeted production subgame. Expected sales is the integral against μ\muμ, never a free function unrelated to FFF. The equilibrium optimizes over all feasible prices and caps; the narrow interval for n∗n^*n∗ is a conclusion. The work needed includes measure-theoretic sales identities, quantile behavior, and the firm's regime-dependent optimization. Contributions establishing these facts or proving the named milestones fit this mission. The model here is the base utility-token, simultaneous-arrival model; equity tokens, sequential arrival, the §6 extensions and numerics are outside its scope.

Selected references

  • J. R. Gan, G. Tsoukalas and S. Netessine, Initial Coin Offerings, Speculation, and Asset Tokenization, SSRN author version 3361121, June 29, 2020; published in Management Science (2020). SSRN author version, DOI.
10 thms1 active userReviewed
Mechanism DesignOperations Research·Captain: mikedeng1

The Power of Opaque Products in Pricing: For Exchangeable Valuations, Opaque Selling Dominates Discriminatory Pricing When All Customers Are Pessimistic or Valuations Take Two ValuesResearch Paper

Motivation

Sellers sometimes offer a customer a choice among individual products and a product whose final item is revealed only after purchase. Airlines, for example, can sell a fixed itinerary alongside a discounted option that leaves one feature unspecified. The seller must decide whether this opaque product earns enough to justify limiting how much individual item prices can differ. Elmachtoub and Hamilton study this question for unit-demand customers who know their own values for the possible items. Their 2021 paper identifies two conditions under which the best menu of opaque products earns at least as much as the best separate item prices.

The comparison matters because the opaque menu uses less item-specific pricing information. The seller gives every item one price and gives opaque products of equal size the same price. The theorem says that this constrained format can still match discriminatory pricing when the valuation law is symmetric and either customers assess opaque products pessimistically or valuations have only two levels. The paper also supplies counterexamples outside those conditions; the symmetry and customer-type hypotheses are substantive.

Setting

There are N≥2N\ge2N≥2 items. A customer has a nonnegative valuation vector v=(v1,…,vN)v=(v_1,\ldots,v_N)v=(v1​,…,vN​), sampled from a probability law FFF. A menu option has a price and a valuation. Its utility is valuation minus price. The customer buys one option of greatest utility when that utility is nonnegative. If several options have greatest utility, the customer buys one with the highest price. Remaining ties at the same price do not affect revenue. A customer may also buy nothing, yielding zero revenue.

In discriminatory pricing (DP), the seller assigns a nonnegative price pip_ipi​ to each item and offers only the individual items. Write RDP(p1,…,pN)R_{\mathrm{DP}}(p_1,\ldots,p_N)RDP​(p1​,…,pN​) for expected revenue and RDPR_{\mathrm{DP}}RDP​ for its optimum over item-price vectors.

In opaque selling (OPQ), the seller offers the items at a common price ppp and also offers every set SSS of at least two items. A set of size kkk costs pkp^kpk, independent of which items it contains. A pessimistic customer values SSS at min⁡i∈Svi\min_{i\in S}v_imini∈S​vi​; a risk-neutral customer values it at ∣S∣−1∑i∈Svi|S|^{-1}\sum_{i\in S}v_i∣S∣−1∑i∈S​vi​. Both value a singleton at its item value. A customer is pessimistic with probability α\alphaα and risk-neutral with probability 1−α1-\alpha1−α, independently of the valuation vector. The expected revenue of a price menu is ROPQα(p,p2,…,pN)R_{\mathrm{OPQ}}^\alpha(p,p^2,\ldots,p^N)ROPQα​(p,p2,…,pN); its optimum is ROPQαR_{\mathrm{OPQ}}^\alphaROPQα​.

A law FFF is exchangeable if permuting the coordinates of vvv leaves its joint distribution unchanged. Coordinates may be correlated, and they need not be independent. A law is two-valued when there are real numbers a<ba<ba<b such that almost every valuation vector has every coordinate in {a,b}\{a,b\}{a,b}. This says the same pair applies to the entire vector, rather than allowing a different pair at each coordinate.

Formalization targets

Theorem 1: revenue dominance

For exchangeable FFF, nonnegative valuations, and 0≤α≤10\le\alpha\le10≤α≤1, the target is

(α=1 or F is two-valued)⟹ROPQα≥RDP.\bigl(\alpha=1\ \text{or}\ F\text{ is two-valued}\bigr) \quad\Longrightarrow\quad R_{\mathrm{OPQ}}^\alpha\ge R_{\mathrm{DP}}.(α=1 or F is two-valued)⟹ROPQα​≥RDP​.

The first case covers any exchangeable law, including discrete laws with ties, when all customers are pessimistic. The second covers every mix of customer types when the valuations take two common values. Neither case requires independent item values. This mission retains both cases in one goal because that is Theorem 1 as stated in the paper.

The milestones trace the two cases. They identify the equal probabilities of valuation-order events, the value of the best opaque subset at a fixed size, the pessimistic dominance case, the three candidate DP price vectors under two values, the conditional revenue identity printed as display (1), the opaque price ladder's revenue for both customer types, and the two-valued dominance case. Each milestone is attached to a passage in the proof of Theorem 1.

Significance

The theorem gives a comparison of optimal revenues, not just a comparison of one chosen menu with one chosen price vector. For a seller with exchangeable valuations, it guarantees that size-based opaque prices can recover at least the revenue available from item-specific prices in either named regime. Outside those regimes, the paper's counterexamples prevent extending that guarantee merely by relaxing the hypotheses.

The result is proved in the cited paper; the goal here is a machine-checked version of its statement and proof. A formal development would establish the choice rule for finite menus, expectations for discrete and continuous valuation laws, permutation symmetry, finite order statistics, and the behavior of size-based menus at utility and price ties. These components are reusable in later formalizations of unit-demand pricing and opaque-product models. The present mission starts with unproved Lean theorem statements; compiling those statements does not certify the paper's argument.

Difficulty

One tempting argument treats the opaque menu as discriminatory pricing with the valuation order known to the seller. That comparison overlooks a restriction: the same price must be attached to each product size, so a price that is useful for the highest-valued item need not be useful for every realized ordering. The paper explicitly notes that the apparent information advantage alone would also suggest dominance in settings where dominance fails.

Ties create another obstacle. The probability formulas displayed for DP and OPQ in the selling-model section use inequalities that can count a tied purchase more than once. The paper sets aside ties for exposition in that section, yet the two-valued case necessarily has them. A faithful proof has to use the stated customer choice rule and the highest-price tie-break, rather than treating those displayed sums as definitions for discrete laws. The two-valued case also requires tracking how revenue changes with the number of high valuations, without assuming independent coordinates.

Formalization scope

Items are represented by Fin N and valuation profiles by functions Fin N → ℝ; the paper's item indices start at one, while Lean's start at zero. The probability law is a measure on profiles with a probability-measure instance and almost-sure nonnegative coordinates. Exchangeability means invariance of this law under every coordinate permutation. Two-valued support is an almost-sure condition on the whole profile. No independence or density assumption is included.

The menu contains every nonempty item set. Singletons stand for the ordinary items at their common price; sets of equal larger size share an opaque price. All seller prices are nonnegative. Revenue is the highest price among options with maximal nonnegative utility, with zero when no option qualifies. This directly represents the rule on page 8, including discrete laws. The pessimistic valuation is a finite minimum and the risk-neutral valuation a finite average. Customer type is mixed independently of the valuation law, so the OPQ expectation is the α\alphaα-weighted sum of the two type-specific expectations.

Expected revenues and optimal revenues use extended nonnegative reals [0,∞][0,\infty][0,∞]. The optima are suprema over nonnegative prices; existence of a maximizing price vector is not assumed. This avoids default values for an unbounded real supremum and records possible infinite revenue. Prices for unused cardinalities are nonnegative but have no effect on the menu. The order statistic sorts the coordinates with multiplicity; it is not defined through the subset identity it is meant to satisfy.

The goal is not implied by a degenerate model: the revenue function charges the highest price among utility maximizers and sells only at nonnegative utility; the optimal revenues range over nonnegative prices; exchangeability is invariance under every coordinate permutation. Contributions toward the finite-menu choice lemmas, permutation-invariant integration, order-statistic identity, two-valued DP structure, and the two cases are within scope.

Selected references

  • A. N. Elmachtoub and M. L. Hamilton, The Power of Opaque Products in Pricing, Management Science, 2021, doi:10.1287/mnsc.2020.3750. This mission uses the author's 54-page manuscript, pinned locally by SHA-256 ccd5b53ae01d607eca3b7df8549f418f52245b99bb2dfa050b6f5c4878a3cffe; its printed and PDF page numbers agree.
9 thms1 active userReviewed
Operations ResearchOptimal TransportStatistics·Captain: mikedeng1

Sample Out-of-Sample Inference Based on Wasserstein Distance: The Rescaled SOS Profile Function for Means Converges in Distribution to an Explicit Dimension-Dependent LimitResearch Paper

Motivation

Stress testing and data-driven decision making often combine two samples: observations X1,…,XnX_1,\dots,X_nX1​,…,Xn​ of the quantity of interest and additional scenarios Y1,…,YmY_1,\dots,Y_mY1​,…,Ym​ (stress scenarios supplied by a regulator, or unlabeled data). Blanchet and Kang (arXiv:1605.01340v4, 2020; later in Operations Research, 2021) propose to measure how far the empirical distribution of the observations is from the set of distributions that live on the merged sample and satisfy a hypothesis about a parameter, such as "the mean is θ\thetaθ". The distance is an optimal-transport (Wasserstein) cost, and the resulting statistic is the sample out-of-sample (SOS) profile function.

The construction is the transport analogue of empirical likelihood (Owen 1988), whose central result is a Wilks-type theorem (Wilks 1938): −2log⁡-2\log−2log of the likelihood ratio at the true mean is asymptotically χ2\chi^2χ2. Empirical likelihood can only reweight the observed points. The SOS function can also move mass to out-of-sample scenarios, which is what makes it usable for stress testing. Confidence regions require the asymptotic law of the statistic at the true parameter. That law is the subject of this mission: the paper's main theorem for means, Theorem 1 (pp. 15–16).

Setting

Let l≥1l\ge1l≥1. Observations X1,X2,…X_1,X_2,\dotsX1​,X2​,… and scenarios Y1,Y2,…Y_1,Y_2,\dotsY1​,Y2​,… are random vectors in Rl\mathbb R^lRl, all mutually independent, with Lebesgue densities fXf_XfX​ and fYf_YfY​. A ratio κ∈[0,∞)\kappa\in[0,\infty)κ∈[0,∞) fixes the number of scenarios m=⌊κn⌋m=\lfloor\kappa n\rfloorm=⌊κn⌋ used with nnn observations. The standing assumptions A1)–A4) (p. 14) are: the samples are i.i.d. with these densities; E∥X1∥22+E∥Y1∥22<∞\mathbb E\|X_1\|_2^2+\mathbb E\|Y_1\|_2^2<\inftyE∥X1​∥22​+E∥Y1​∥22​<∞; fX,fY>0f_X,f_Y>0fX​,fY​>0 everywhere; and, if l≥2l\ge2l≥2, fX,fYf_X,f_YfX​,fY​ are differentiable with bounded gradients.

The merged sample is Zn+m=(Z1,…,Zn+m)=(X1,…,Xn,Y1,…,Ym)\mathcal Z_{n+m}=(Z_1,\dots,Z_{n+m})=(X_1,\dots,X_n,Y_1,\dots,Y_m)Zn+m​=(Z1​,…,Zn+m​)=(X1​,…,Xn​,Y1​,…,Ym​). For θ∈Rl\theta\in\mathbb R^lθ∈Rl the SOS function (Definition 1, in the linear-programming form (21)) is

RnW(θ)=min⁡{∑i=1n∑j=1n+mπ(i,j)∥Xi−Zj∥22: π≥0, ∑jπ(i,j)=1n ∀i, ∑j(∑iπ(i,j))Zj=θ}.R_n^W(\theta)=\min\Big\{\sum_{i=1}^n\sum_{j=1}^{n+m}\pi(i,j)\|X_i-Z_j\|_2^2:\ \pi\ge0,\ \sum_j\pi(i,j)=\tfrac1n\ \forall i,\ \sum_j\Big(\sum_i\pi(i,j)\Big)Z_j=\theta\Big\}.RnW​(θ)=min{i=1∑n​j=1∑n+m​π(i,j)∥Xi​−Zj​∥22​: π≥0, j∑​π(i,j)=n1​ ∀i, j∑​(i∑​π(i,j))Zj​=θ}.

It is the least squared transport cost from the empirical measure μn\mu_nμn​ of the XiX_iXi​ to a probability measure supported on Zn+m\mathcal Z_{n+m}Zn+m​ with mean θ\thetaθ. The statistic of interest is RnW(θ∗)R_n^W(\theta_*)RnW​(θ∗​) at the true mean θ∗=EX1\theta_*=\mathbb E X_1θ∗​=EX1​. Write Var(X)=E[(X1−θ∗)(X1−θ∗)T]\mathrm{Var}(X)=\mathbb E[(X_1-\theta_*)(X_1-\theta_*)^{\mathsf T}]Var(X)=E[(X1​−θ∗​)(X1​−θ∗​)T] and Z~∼N(0,Var(X))\tilde Z\sim N(0,\mathrm{Var}(X))Z~∼N(0,Var(X)).

The limits involve a mixed exponential variable. With the rate Λ(x,λ)=[fX(x+λ/2)+κfY(x+λ/2)] πl/2/Γ(l/2+1)\Lambda(x,\lambda)=[f_X(x+\lambda/2)+\kappa f_Y(x+\lambda/2)]\,\pi^{l/2}/\Gamma(l/2+1)Λ(x,λ)=[fX​(x+λ/2)+κfY​(x+λ/2)]πl/2/Γ(l/2+1), the variable τ(λ)\tau(\lambda)τ(λ) satisfies P[τ(λ)>t]=E[exp⁡(−Λ(X1,λ)t)]\mathbb P[\tau(\lambda)>t]=\mathbb E[\exp(-\Lambda(X_1,\lambda)t)]P[τ(λ)>t]=E[exp(−Λ(X1​,λ)t)]. From τ(0)\tau(0)τ(0) one defines g~(x)=P(τ(0)≤∥x∥22)\tilde g(x)=\mathbb P(\tau(0)\le\|x\|_2^2)g~​(x)=P(τ(0)≤∥x∥22​), η~(x)=E[max⁡(1−τ(0)/∥x∥22,0)]\tilde\eta(x)=\mathbb E[\max(1-\tau(0)/\|x\|_2^2,0)]η~​(x)=E[max(1−τ(0)/∥x∥22​,0)], the solution ρ(z)>0\rho(z)>0ρ(z)>0 of 1/ρ=g~(ρz)1/\rho=\tilde g(\rho z)1/ρ=g~​(ρz), and c0=E[Λ(X1,0)]c_0=\mathbb E[\Lambda(X_1,0)]c0​=E[Λ(X1​,0)].

Formalization targets

Goal: Theorem 1 (pp. 15–16)

As n→∞n\to\inftyn→∞,

l=1:nRnW(θ∗)⇒σ2χ12=law of ∥Z~∥22,σ2=Var(X);l=1:\quad nR_n^W(\theta_*)\Rightarrow\sigma^2\chi_1^2=\text{law of }\|\tilde Z\|_2^2,\qquad \sigma^2=\mathrm{Var}(X);l=1:nRnW​(θ∗​)⇒σ2χ12​=law of ∥Z~∥22​,σ2=Var(X); l=2:nRnW(θ∗)⇒ρ(Z~)(2−η~(ρ(Z~)Z~) ρ(Z~))∥Z~∥22;l=2:\quad nR_n^W(\theta_*)\Rightarrow\rho(\tilde Z)\big(2-\tilde\eta(\rho(\tilde Z)\tilde Z)\,\rho(\tilde Z)\big)\|\tilde Z\|_2^2;l=2:nRnW​(θ∗​)⇒ρ(Z~)(2−η~​(ρ(Z~)Z~)ρ(Z~))∥Z~∥22​; l≥3:n1/2+3/(2l+2)RnW(θ∗)⇒2l+2l+2⋅∥Z~∥21+1/(l+1)c01/(l+1).l\ge3:\quad n^{1/2+3/(2l+2)}R_n^W(\theta_*)\Rightarrow\frac{2l+2}{l+2}\cdot\frac{\|\tilde Z\|_2^{1+1/(l+1)}}{c_0^{1/(l+1)}} .l≥3:n1/2+3/(2l+2)RnW​(θ∗​)⇒l+22l+2​⋅c01/(l+1)​∥Z~∥21+1/(l+1)​​.

The goal is the paper's theorem in all three regimes. The l=2l=2l=2 case is stated with η~(ρZ~)\tilde\eta(\rho\tilde Z)η~​(ρZ~), a correction of the printed η~(Z~)\tilde\eta(\tilde Z)η~​(Z~) (see Formalization scope).

Milestones (in attack order)

  1. (21)–(22), pp. 35–36: LP duality, RnW(0)=max⁡λ,γ≥0{−λTXˉn−1n∑iγi}R_n^W(0)=\max_{\lambda,\gamma\ge0}\{-\lambda^{\mathsf T}\bar X_n-\frac1n\sum_i\gamma_i\}RnW​(0)=maxλ,γ≥0​{−λTXˉn​−n1​∑i​γi​} subject to −λTXi−γi≤inf⁡j{−λTZj+∥Xi−Zj∥22}-\lambda^{\mathsf T}X_i-\gamma_i\le\inf_j\{-\lambda^{\mathsf T}Z_j+\|X_i-Z_j\|_2^2\}−λTXi​−γi​≤infj​{−λTZj​+∥Xi​−Zj​∥22​}.
  2. §7.2.1, p. 38: the closest-point bound, inf⁡a{−λTa+∥x−a∥22}=−λTx−∥λ∥22/4\inf_a\{-\lambda^{\mathsf T}a+\|x-a\|_2^2\}=-\lambda^{\mathsf T}x-\|\lambda\|_2^2/4infa​{−λTa+∥x−a∥22​}=−λTx−∥λ∥22​/4, attained at x+λ/2x+\lambda/2x+λ/2.
  3. (27), p. 41: RnW(0)=max⁡λ{−2λTXˉn−1n∑imax⁡(∥λ∥22−Di(λ),0)}R_n^W(0)=\max_\lambda\{-2\lambda^{\mathsf T}\bar X_n-\frac1n\sum_i\max(\|\lambda\|_2^2-D_i(\lambda),0)\}RnW​(0)=maxλ​{−2λTXˉn​−n1​∑i​max(∥λ∥22​−Di​(λ),0)}, with Di(λ)D_i(\lambda)Di​(λ) the squared distance from Xi+λX_i+\lambdaXi​+λ to the nearest other point of Zn+m\mathcal Z_{n+m}Zn+m​.
  4. Lemma 1, p. 40 (two bounds): lim sup⁡nEsup⁡t≤T∣n−1/2∑i(⋅−E ⋅)∣<∞\limsup_n\mathbb E\sup_{t\le T}|n^{-1/2}\sum_i(\cdot-\mathbb E\,\cdot)|<\inftylimsupn​Esupt≤T​∣n−1/2∑i​(⋅−E⋅)∣<∞ for I(Ti(n,λ)≤t)I(T_i(n,\lambda)\le t)I(Ti​(n,λ)≤t) and for max⁡(t2−Ti(n,λ)α,0)\max(t^2-T_i(n,\lambda)^\alpha,0)max(t2−Ti​(n,λ)α,0), where Ti(n,λ)T_i(n,\lambda)Ti​(n,λ) is the first-arrival time of the counting process of other sample points near Xi+λ/2X_i+\lambda/2Xi​+λ/2.
  5. Lemma 2, p. 41 (first clause): P[Ti(n,λ)≤t]−P[τ(λ)≤t]=O(t1+1/l/n1/l)\mathbb P[T_i(n,\lambda)\le t]-\mathbb P[\tau(\lambda)\le t]=O(t^{1+1/l}/n^{1/l})P[Ti​(n,λ)≤t]−P[τ(λ)≤t]=O(t1+1/l/n1/l).
  6. (31), p. 45: 1/ρ=g~(ρz)1/\rho=\tilde g(\rho z)1/ρ=g~​(ρz) has a unique solution for z≠0z\ne0z=0 (l=2l=2l=2).
  7. §7.2.5, p. 45: max⁡ζ{−2ζTz−∥ζ∥22η~(ζ)}=ρ(z)(2−η~(ρ(z)z)ρ(z))∥z∥22\max_\zeta\{-2\zeta^{\mathsf T}z-\|\zeta\|_2^2\tilde\eta(\zeta)\}=\rho(z)(2-\tilde\eta(\rho(z)z)\rho(z))\|z\|_2^2maxζ​{−2ζTz−∥ζ∥22​η~​(ζ)}=ρ(z)(2−η~​(ρ(z)z)ρ(z))∥z∥22​, attained at −ρ(z)z-\rho(z)z−ρ(z)z.
  8. §7.2.6, pp. 47–48: max⁡ζ{−2ζTz−2Cl+2∥ζ∥2l+2}=2l+2l+2∥z∥21+1/(l+1)/C1/(l+1)\max_\zeta\{-2\zeta^{\mathsf T}z-\frac{2C}{l+2}\|\zeta\|_2^{l+2}\}=\frac{2l+2}{l+2}\|z\|_2^{1+1/(l+1)}/C^{1/(l+1)}maxζ​{−2ζTz−l+22C​∥ζ∥2l+2​}=l+22l+2​∥z∥21+1/(l+1)​/C1/(l+1).

Significance

Theorem 1 calibrates the SOS method. Its quantiles define the radius of a confidence region for the mean, and through the paper's Theorems 2–3 for estimating equations and for stochastic programs whose uncertainty set must cover out-of-sample scenarios. Unlike empirical likelihood, the limit depends on the dimension and on the densities fX,fYf_X,f_YfX​,fY​: more dispersed scenarios give smaller distances and smaller quantiles (§4.1.1). The theorem also shows that the scaling itself changes with lll, from nnn for l≤2l\le2l≤2 to n1/2+3/(2l+2)n^{1/2+3/(2l+2)}n1/2+3/(2l+2) for l≥3l\ge3l≥3.

The result is proved in the paper; nothing here is an open problem. As far as we know, none of it has been formalized. A formal development adds:

  • a checked version of a proof that is informal in places: it passes op(1)o_p(1)op​(1) terms through maxima over unbounded dual variables;
  • the correction of the l=2l=2l=2 limit;
  • reusable pieces: a duality theorem for finite transport problems with a mean constraint, a binomial-to-Poisson estimate for nearest-neighbour counts, and empirical-process bounds for dependent nearest-neighbour statistics.

Difficulty

The obvious approach is a Wilks-type expansion: take the dual of the LP, find the dual multiplier λn≈c Xˉn\lambda_n\approx c\,\bar X_nλn​≈cXˉn​ by a quadratic approximation, and read off a χ2\chi^2χ2 limit. This works only for l=1l=1l=1. In higher dimensions the transport may only move mass to sample points, and the constraint for observation iii is controlled by the distance from Xi+λ/2X_i+\lambda/2Xi​+λ/2 to the nearest other sample point. That distance is of order n−1/ln^{-1/l}n−1/l, so it competes with ∥λ∥2\|\lambda\|^2∥λ∥2 at a dimension-dependent scale. The quadratic approximation therefore fails: the dual objective becomes, after rescaling, an average of max⁡(∥ζ∥2−Ti2/l,0)\max(\|\zeta\|^2-T_i^{2/l},0)max(∥ζ∥2−Ti2/l​,0) over nnn dependent first-arrival times, which share the same merged sample.

Three steps carry the weight. (a) The dependent array Ti(n,λ)T_i(n,\lambda)Ti​(n,λ) needs a uniform n−1/2n^{-1/2}n−1/2 empirical-process bound (Lemma 1); i.i.d. tools such as the Dvoretzky–Kiefer–Wolfowitz inequality do not apply. (b) A quantitative Poisson approximation (Lemma 2) needs the bounded density gradients. (c) The random concave objective must converge to a deterministic one uniformly enough that its maximiser converges (argmax continuity). Step (c) also needs a multivariate central limit theorem for nXˉn\sqrt n\bar X_nn​Xˉn​, which Mathlib does not yet provide.

Formalization scope

Representation. Points live in EuclideanSpace ℝ (Fin l). The samples are infinite sequences on a probability space (Ω,P)(\Omega,\mathbb P)(Ω,P), jointly independent (iIndepFun over N⊔N\mathbb N\sqcup\mathbb NN⊔N), with laws volume.withDensity. The nnn-th statistic uses the first nnn observations and the first ⌊κn⌋\lfloor\kappa n\rfloor⌊κn⌋ scenarios; m=[κn]m=[\kappa n]m=[κn] is read as the floor, and κ=0\kappa=0κ=0 is allowed. The SOS function is a real infimum over couplings π:{1..n}×J→R\pi:\{1..n\}\times J\to\mathbb Rπ:{1..n}×J→R of indexed families. Convergence in distribution is Mathlib's TendstoInDistribution, with limit the image of multivariateGaussian 0 Var(X) under an explicit map φl\varphi_lφl​. The goal is a conjunction of three clauses, l=1l=1l=1, l=2l=2l=2, l≥3l\ge3l≥3.

Conventions and disclosed readings.

  • The l=2l=2l=2 limit uses η~(ρ(Z~)Z~)\tilde\eta(\rho(\tilde Z)\tilde Z)η~​(ρ(Z~)Z~). Theorem 1 prints η~(Z~)\tilde\eta(\tilde Z)η~​(Z~), but the derivation on p. 45 evaluates G(ζ,n)=∥ζ∥2η~(ζ)+o(1)G(\zeta,n)=\|\zeta\|^2\tilde\eta(\zeta)+o(1)G(ζ,n)=∥ζ∥2η~​(ζ)+o(1) at ζn∗=−ρZn\zeta^*_n=-\rho Z_nζn∗​=−ρZn​, and the two laws differ for every Z~≠0\tilde Z\ne0Z~=0. Milestone 7 isolates this computation.
  • When θ∗\theta_*θ∗​ is outside the convex hull of the merged sample, the LP is infeasible; RnWR_n^WRnW​ is then 000 by convention, where the paper has +∞+\infty+∞. The event has probability tending to zero, and no feasibility hypothesis is added to the goal.
  • ρ(0):=0\rho(0):=0ρ(0):=0 (the equation has no solution at 000, which is a null event for Z~\tilde ZZ~), and η~(0)=1\tilde\eta(0)=1η~​(0)=1 is never used.
  • In (27), after λ←2λ\lambda\leftarrow2\lambdaλ←2λ, the nearest-point distance is centred at Xi+λX_i+\lambdaXi​+λ; the page keeps the symbol Ti(n,λ)T_i(n,\lambda)Ti​(n,λ).
  • "Zj≠XiZ_j\ne X_iZj​=Xi​" in the counting process is a condition on values.
  • Lemma 1's lim⁡‾\overline{\lim}lim is a lim sup⁡\limsuplimsup of a lower Lebesgue integral.
  • Lemma 2 is posed at fixed ttt, with the constant allowed to depend on ttt. Its second clause (O(1/nl)O(1/n^l)O(1/nl)) is not posed: it is not supported by the estimate (26) its proof cites. For l=1l=1l=1 Lemma 2 adds the density regularity that A4) requires only for l≥2l\ge2l≥2.
  • In §7.2.6 the sign of GGG is read as +++, and the scaling n1/2+5/(4l+2)n^{1/2+5/(4l+2)}n1/2+5/(4l+2) printed on p. 48 is read as n1/2+3/(2l+2)n^{1/2+3/(2l+2)}n1/2+3/(2l+2) (Theorem 1).

Ruled out. The following are not acceptable formalizations: a feasibility hypothesis on every sample; a limit law left as a parameter or quantified existentially; ρ\rhoρ, g~\tilde gg~​ or η~\tilde\etaη~​ given by closed forms rather than from the law of τ(0)\tau(0)τ(0) and the fixed-point property; the printed η~(Z~)\tilde\eta(\tilde Z)η~​(Z~) presented as proved; Lemma 2 with a constant uniform in ttt; Lemma 1 with a Bochner integral, which would be a default 000 for a non-integrable supremum.

Infrastructure. A full proof needs: finite LP strong duality (Farkas), absent as a packaged theorem; binomial/Poisson estimates for small-ball counts; a pathwise Burkholder–Davis–Gundy inequality (Beiglböck and Siorpaes 2015) or another route to Lemma 1; dominated-convergence differentiation of Emax⁡(∥ζ∥2−τ,0)\mathbb E\max(\|\zeta\|^2-\tau,0)Emax(∥ζ∥2−τ,0); a multivariate CLT; and argmax-continuity for random concave functions. The LP duality and the multivariate CLT are reusable well beyond this mission. Contributions to any milestone are welcome; milestones 1, 2, 6, 7 and 8 are deterministic and independent of the probabilistic core.

Selected references

  • J. Blanchet, Y. Kang, Sample Out-of-Sample Inference Based on Wasserstein Distance, arXiv:1605.01340v4, 2020; Operations Research, 2021. https://arxiv.org/abs/1605.01340
  • J. Blanchet, Y. Kang, K. Murthy, Robust Wasserstein Profile Inference and Applications to Machine Learning, Journal of Applied Probability 56(3), 2019. https://arxiv.org/abs/1610.05627
  • A. B. Owen, Empirical likelihood ratio confidence intervals for a single functional, Biometrika 75(2), 1988. https://doi.org/10.1093/biomet/75.2.237
  • S. S. Wilks, The large-sample distribution of the likelihood ratio for testing composite hypotheses, Annals of Mathematical Statistics 9(1), 1938. https://doi.org/10.1214/aoms/1177732360
  • M. Beiglböck, P. Siorpaes, Pathwise versions of the Burkholder–Davis–Gundy inequality, Bernoulli 21(1), 2015. https://arxiv.org/abs/1305.6188
14 thms1 active userReviewed
Bandit AlgorithmsOperations ResearchStatistics·Captain: mikedeng1

Bayesian Sequential Learning for Clinical Trials of Multiple Correlated Medical Interventions: The cPDE Allocation Policy Is Asymptotically ConsistentResearch Paper

Motivation

A clinical trial that compares several treatments for the same condition must decide, patient by patient, which treatment the next patient receives. When the treatments are related (doses of one drug, drugs of one class), an outcome observed on one arm is informative about the others, and a good allocation rule should exploit that correlation. Chick, Gans and Yapar (SSRN 3184758, version of August 31, 2020; later in Management Science) model this as Bayesian sequential sampling with a correlated Gaussian prior and propose the cPDE allocation policy, which samples the arm with the largest value of an adaptive-lookahead optimal-stopping index. Their one theorem, Theorem 1, says that cPDE is asymptotically consistent: run forever, it learns every arm's mean and identifies the best arm.

Consistency is the minimal guarantee asked of an allocation heuristic: a rule that can stop sampling an arm whose value is still uncertain may settle on the wrong treatment forever. The argument follows the consistency proof of Xie, Frazier and Chick (Operations Research 64(2):542–559, 2016; "XFC") for one-step knowledge-gradient (cKG) policies, extended to an index whose lookahead is itself an optimal stopping time.

Setting

There are M≥2M\ge2M≥2 arms. Arm iii has unknown mean θi\theta_iθi​; an observation of arm iii is Y∣θi∼N(θi,λi)Y\mid\theta_i\sim N(\theta_i,\lambda_i)Y∣θi​∼N(θi​,λi​) with known λi>0\lambda_i>0λi​>0, independent across observations. The prior is θ∼N(μ0,Σ0)\theta\sim N(\mu^0,\Sigma^0)θ∼N(μ0,Σ0) with Σ0\Sigma^0Σ0 positive definite, and sampling arm iii costs ci>0c_i>0ci​>0. After ttt observations the belief is (μt,Σt)(\mu^t,\Sigma^t)(μt,Σt); one observation yyy of arm iii updates it by Bayes' rule (8):

μ′=μ+y−μiλi+Σi,iΣei,Σ′=Σ−Σeiei⊤Σλi+Σi,i.\mu'=\mu+\frac{y-\mu_i}{\lambda_i+\Sigma_{i,i}}\Sigma e_i,\qquad \Sigma'=\Sigma-\frac{\Sigma e_ie_i^\top\Sigma}{\lambda_i+\Sigma_{i,i}}.μ′=μ+λi​+Σi,i​y−μi​​Σei​,Σ′=Σ−λi​+Σi,i​Σei​ei⊤​Σ​.

From a belief (μ,Σ)(\mu,\Sigma)(μ,Σ), imagine sampling arm iii alone. The predictive change Ziτ=μit+τ−μitZ^\tau_i=\mu^{t+\tau}_i-\mu^t_iZiτ​=μit+τ​−μit​ after τ\tauτ samples is a centred Gaussian with variance λiτ/(ni(ni+τ))\lambda_i\tau/(n_i(n_i+\tau))λi​τ/(ni​(ni​+τ)), ni=λi/Σi,in_i=\lambda_i/\Sigma_{i,i}ni​=λi​/Σi,i​ ((9)–(10)). The cPDE allocation index is (EC.8):

νi∗(μ,Σ)=sup⁡Ti≥1E[−Ti+1cimax⁡j{μj+Σi,jΣi,iZiTi}]−1cimax⁡jμj,\nu^*_i(\mu,\Sigma)=\sup_{T_i\ge1}\mathbb E\Big[-T_i+\frac1{c_i}\max_j\Big\{\mu_j+\frac{\Sigma_{i,j}}{\Sigma_{i,i}}Z^{T_i}_i\Big\}\Big]-\frac1{c_i}\max_j\mu_j,νi∗​(μ,Σ)=Ti​≥1sup​E[−Ti​+ci​1​jmax​{μj​+Σi,i​Σi,j​​ZiTi​​}]−ci​1​jmax​μj​,

the supremum over stopping times of the sampling of arm iii. The one-step index νicKG1\nu^{cKG_1}_iνicKG1​​ is the same expression with Ti≡1T_i\equiv1Ti​≡1 ((EC.6)). The cPDE policy allocates observation t+1t+1t+1 to an arm maximizing νi∗(μt,Σt)\nu^*_i(\mu^t,\Sigma^t)νi∗​(μt,Σt); the trial never stops.

Formalization targets

Goal: Theorem 1 (p. 13)

For each arm iii, almost surely,

Σi,iT→0,νi∗(μT,ΣT)→−1,μiT→θi (also in L2),\Sigma^T_{i,i}\to0,\qquad \nu^*_i(\mu^T,\Sigma^T)\to-1,\qquad \mu^T_i\to\theta_i\ \text{(also in }L^2),Σi,iT​→0,νi∗​(μT,ΣT)→−1,μiT​→θi​ (also in L2),

and almost surely θ\thetaθ has a unique maximizer which, for all large TTT, is also the unique maximizer of μT\mu^TμT.

Milestones

  1. Prop. 3 (p. 12): ZiτZ^\tau_iZiτ​ is a uniformly integrable martingale and E[ZiT]=0\mathbb E[Z^{T}_i]=0E[ZiT​]=0 for a.s. finite stopping times.
  2. Lemma EC.1, first inequality (p. ec3): νicKG1≤νi∗\nu^{cKG_1}_i\le\nu^*_iνicKG1​​≤νi∗​.
  3. Lemma EC.3 (p. ec6): if lim inf⁡TνicKG1(μT,ΣT)=−1\liminf_T\nu^{cKG_1}_i(\mu^T,\Sigma^T)=-1liminfT​νicKG1​​(μT,ΣT)=−1 for all iii, then every Σi,iT→0\Sigma^T_{i,i}\to0Σi,iT​→0.
  4. Lemma EC.4 (p. ec7): an arm sampled infinitely often has Σi,iT→0\Sigma^T_{i,i}\to0Σi,iT​→0, and Σi,iT→0\Sigma^T_{i,i}\to0Σi,iT​→0 forces νi∗(μT,ΣT)→−1\nu^*_i(\mu^T,\Sigma^T)\to-1νi∗​(μT,ΣT)→−1.
  5. Lemma EC.5 (p. ec7): if lim inf⁡Tνi∗(μT,ΣT)=−1\liminf_T\nu^*_i(\mu^T,\Sigma^T)=-1liminfT​νi∗​(μT,ΣT)=−1 for all iii, then every Σi,iT→0\Sigma^T_{i,i}\to0Σi,iT​→0.

Significance

Theorem 1 is the only theoretical guarantee the paper gives for cPDE; the rest of the paper is computational (bounds, comparators, experiments). It also covers, through the same sandwich argument, the cheaper variants cPDELower and cPDEUpper (Corollary 1), which are not part of this mission. A consistent policy cannot lock onto a wrong treatment, which is the property a trial designer needs before comparing finite-horizon performance.

The result is proved on paper but has no machine-checked proof. Formalizing it requires a correlated-Gaussian Bayesian learning model, optimal stopping indices defined as suprema over stopping times, and the XFC consistency argument, none of which exist in Lean. The paper's proof also routes Lemma EC.4 through an upper bound that is false as printed (see below), so a checked proof replaces that step with a correct one.

Difficulty

The obvious route bounds νi∗\nu^*_iνi∗​ above by a one-step index with a fixed lookahead τˉ\bar\tauτˉ, as the paper's Lemma EC.1 does. That bound fails: τˉ\bar\tauτˉ of (EC.11) changes under a change of currency while every index is invariant. For M=2M=2M=2, μ=(0,10−3)\mu=(0,10^{-3})μ=(0,10−3), Σ=diag(10−6,10−6)\Sigma=\mathrm{diag}(10^{-6},10^{-6})Σ=diag(10−6,10−6), λ=(10−4,10−4)\lambda=(10^{-4},10^{-4})λ=(10−4,10−4), c=(10−7,10−7)c=(10^{-7},10^{-7})c=(10−7,10−7), the formula gives τˉ=1\bar\tau=1τˉ=1 and ν1cKG1:1≈−1\nu^{cKG_1:1}_1\approx-1ν1cKG1​:1​≈−1, while the lookahead T≡242T\equiv242T≡242 already yields a value above 240240240. Any upper bound on νi∗\nu^*_iνi∗​ must instead be uniform over stopping times of unbounded length, and showing that it tends to −1-1−1 needs (Σi,jT)2/Σi,iT→0(\Sigma^T_{i,j})^2/\Sigma^T_{i,i}\to0(Σi,jT​)2/Σi,iT​→0 along posterior covariances, not only Σi,iT→0\Sigma^T_{i,i}\to0Σi,iT​→0.

The second difficulty is the "finitely sampled arms" half of the argument (Lemmas EC.3, EC.5): an arm whose variance stays positive must keep a one-step value of sampling bounded away from zero, even while correlated arms are learned. This uses almost sure convergence of the posterior means and the structure of the piecewise linear terminal reward.

Formalization scope

Arms are Fin M (0-based) and time ttt counts observations, t=0t=0t=0 being the prior. The trial lives on the canonical space θ∈RM\theta\in\mathbb R^Mθ∈RM times an array of standard normal noises εi,k\varepsilon_{i,k}εi,k​, with law multivariateGaussian µ0 Σ0 times an infinite product of gaussianReal 0 1; the kkk-th sample of arm iii is θi+λi εi,k\theta_i+\sqrt{\lambda_i}\,\varepsilon_{i,k}θi​+λi​​εi,k​. The index νi∗\nu^*_iνi∗​ is built on its own auxiliary space (a prior draw of θi−μi\theta_i-\mu_iθi​−μi​ and i.i.d. noise), not as a conditional expectation on the trial space.

Committed conventions, each a disclosed reading of the page:

  • νi∗\nu^*_iνi∗​ is (EC.8). The printed definition (15) omits −1cimax⁡jμj-\frac1{c_i}\max_j\mu_j−ci​1​maxj​μj​; with (15) Theorem 1 ii) would be false.
  • The supremum in (EC.8) is a real sSup over stopping times of the auxiliary filtration with 1≤Ti≤N1\le T_i\le N1≤Ti​≤N for some NNN; this has the paper's value and keeps every expectation finite.
  • Ties in the cPDE argmax go to the least index (the paper: randomly). Every arm is eligible at every time (Mt=M\mathcal M_t=\mathcal MMt​=M), so Assumption 1 iii) is trivial.
  • Theorem 1 ii) and Lemmas EC.4–EC.5 are read with (μT,ΣT)(\mu^T,\Sigma^T)(μT,ΣT). Theorem 1 iv) is uniqueness of the maximizer of θ\thetaθ plus eventual agreement of the unique maximizer of μT\mu^TμT. The L2L^2L2 claim is MemLp plus eLpNorm → 0.
  • "liminf =−1=-1=−1" in Lemmas EC.3, EC.5 is "for every ε>0\varepsilon>0ε>0, the index is below −1+ε-1+\varepsilon−1+ε infinitely often"; since both indices are ≥−1\ge-1≥−1 this is the printed hypothesis. Lemma EC.4 is stated for every sample path.

Trivializing formalizations are ruled out: νi∗\nu^*_iνi∗​ is defined from (μ,Σ,λi,ci)(\mu,\Sigma,\lambda_i,c_i)(μ,Σ,λi​,ci​), not a parameter, a fixed-lookahead index or a supremum over non-adapted times; the policy is defined, not assumed; μt\mu^tμt and Σt\Sigma^tΣt both come from (8); the L2L^2L2 part does not use an integral that is 000 for non-integrable functions; iv) asserts uniqueness; and the false upper bound of Lemma EC.1 is not posed or used.

A complete development needs Gaussian conditioning for the update (8) (that μt=E[θ∣Ft]\mu^t=\mathbb E[\theta\mid\mathcal F_t]μt=E[θ∣Ft​]), the closed form Σt=(Σ0−1+diag(nkt/λk))−1\Sigma^t=(\Sigma_0^{-1}+\mathrm{diag}(n^t_k/\lambda_k))^{-1}Σt=(Σ0−1​+diag(nkt​/λk​))−1, optional stopping for uniformly integrable martingales, measurability of the index, and the XFC argument. The Gaussian-update and optional-stopping parts are reusable for other Bayesian ranking-and-selection results; contributions of those as separate lemmas are welcome.

Selected references

  • S. E. Chick, N. Gans, Ö. Yapar, Bayesian Sequential Learning for Clinical Trials of Multiple Correlated Medical Interventions, SSRN 3184758 (INSEAD Working Paper 2020/40/TOM/ACGRE), version of August 31, 2020. https://ssrn.com/abstract=3184758
  • J. Xie, P. I. Frazier, S. E. Chick, Bayesian Optimization via Simulation with Pairwise Sampling and Correlated Prior Beliefs, Operations Research 64(2):542–559, 2016.
  • P. I. Frazier, W. B. Powell, S. Dayanik, The Knowledge-Gradient Policy for Correlated Normal Beliefs, INFORMS Journal on Computing 21(4):599–613, 2009.
  • S. E. Chick, P. I. Frazier, Sequential Sampling for Selection with Economics of Selection Procedures, Management Science 58(3):550–569, 2012.
  • D. Williams, Probability with Martingales, Cambridge University Press, 1991.
9 thms1 active userReviewed
Control TheoryOperations ResearchStochastic Systems·Captain: mikedeng1

Dual Sourcing and Smoothing Under Nonstationary Demand Time Series: Reshoring with SpeedFactories: When the Break-Even Price Is Concave, Dual Sourcing with a SpeedFactory Beats Full Off-ShoringResearch Paper

Motivation

Firms that moved production offshore for low unit costs pay for it in lead time: a slow global supplier forces large safety stocks, and the cost of those stocks grows with demand variability. A recurring question in operations management is when it pays to re-shore part of production to a fast, local, more expensive plant. Adidas's "SpeedFactories" are the example that gives this paper its name. Boute, Disney, Gijsbrechts and Van Mieghem (accepted manuscript, 2020; Management Science 2022) answer this for a linear dual-sourcing policy. Their analysis yields an explicit break-even price for the off-shore supplier, together with a condition under which a small local plant that absorbs only demand variability beats full off-shoring, even when local production costs more per unit than off-shore purchasing.

The dual-sourcing literature (Whittemore and Saunders 1977; Song and Zipkin 2009; Janakiraman, Seshadri and Sheopuri 2015) shows that optimal dual-sourcing policies have no simple structure for general lead times. Xin and Goldberg (2018) proved that tailored base-surge (TBS) policies are asymptotically optimal as the global lead time grows. The present paper combines TBS with a smoothing local replenishment rule and prices the local capacity explicitly.

Setting

A firm serves demand dtd_tdt​, t=1,2,…t = 1, 2, \dotst=1,2,…, from finished-goods inventory iti_tit​; excess demand is backlogged. The demands are independent and identically distributed N(μ,σ)N(\mu, \sigma)N(μ,σ), where σ\sigmaσ is the standard deviation. Holding costs hhh and backlog costs bbb per unit and period: Cti=h[it]++b[−it]+C^i_t = h[i_t]^+ + b[-i_t]^+Cti​=h[it​]++b[−it​]+.

Two sources replenish the inventory:

  • a global supplier with lead time L≥2L \ge 2L≥2 and unit price ppp;
  • a local SpeedFactory with lead time one, nominal capacity kkk paid at uuu per unit whether used or not, and overtime at umumum per unit, m>1m > 1m>1: Ctq=uk+um[qt−k]+C^q_t = uk + um[q_t - k]^+Ctq​=uk+um[qt​−k]+.

The TBS-POUT policy with allocation γ∈[0,1]\gamma \in [0,1]γ∈[0,1] and smoothing parameter α∈(−1,1)\alpha \in (-1, 1)α∈(−1,1) orders the constant μ(1−γ)\mu(1-\gamma)μ(1−γ) globally and

qt=μγ+(1−α)(i⋆−it)q_t = \mu\gamma + (1-\alpha)(i^\star - i_t)qt​=μγ+(1−α)(i⋆−it​)

locally, where i⋆i^\stari⋆ is the safety stock. The inventory evolves by it=it−1−dt+μ(1−γ)+qt−1i_t = i_{t-1} - d_t + \mu(1-\gamma) + q_{t-1}it​=it−1​−dt​+μ(1−γ)+qt−1​. The long-run average cost is lim inf⁡n1n∑t=1nE[Cti+pμ(1−γ)+Ctq]\liminf_n \frac1n \sum_{t=1}^n \mathbb E[C^i_t + p\mu(1-\gamma) + C^q_t]liminfn​n1​∑t=1n​E[Cti​+pμ(1−γ)+Ctq​]. The optimized dual-sourcing cost CTBS-POUT,k⋆(γ)C^{\mathrm{TBS\text{-}POUT},k^\star}(\gamma)CTBS-POUT,k⋆(γ) is its infimum over (i⋆,k)∈R2(i^\star, k) \in \mathbb R^2(i⋆,k)∈R2.

The benchmark is full off-shoring with the order-up-to policy: order qtg=(i⋆+Lμ)−(it+∑j=1L−1qt−jg)q^g_t = (i^\star + L\mu) - (i_t + \sum_{j=1}^{L-1} q^g_{t-j})qtg​=(i⋆+Lμ)−(it​+∑j=1L−1​qt−jg​) with inventory it=it−1−dt+qt−Lgi_t = i_{t-1} - d_t + q^g_{t-L}it​=it−1​−dt​+qt−Lg​. Its optimized long-run cost over i⋆i^\stari⋆ is CsC^sCs.

With φ,Φ\varphi, \Phiφ,Φ the standard normal density and distribution function, zi=Φ−1[b/(b+h)]z_i = \Phi^{-1}[b/(b+h)]zi​=Φ−1[b/(b+h)] and zq=Φ−1[(m−1)/m]z_q = \Phi^{-1}[(m-1)/m]zq​=Φ−1[(m−1)/m]. The paper's standard deviations are

σi=σ1−α2,σq=σ1−α1+α,σi,s=σL,\sigma_i = \frac{\sigma}{\sqrt{1-\alpha^2}}, \qquad \sigma_q = \sigma\sqrt{\frac{1-\alpha}{1+\alpha}}, \qquad \sigma_{i,s} = \sigma\sqrt L,σi​=1−α2​σ​,σq​=σ1+α1−α​​,σi,s​=σL​,

and the break-even price (20) is

p−(γ)=h+bμγ(σi−σi,s)φ[zi]+umμγσqφ[zq]+u,γ∈(0,1].p^-(\gamma) = \frac{h+b}{\mu\gamma}(\sigma_i - \sigma_{i,s})\varphi[z_i] + \frac{um}{\mu\gamma}\sigma_q\varphi[z_q] + u, \qquad \gamma \in (0,1].p−(γ)=μγh+b​(σi​−σi,s​)φ[zi​]+μγum​σq​φ[zq​]+u,γ∈(0,1].

Formalization targets

Goal: Proposition 4 (pp. 18–19)

If

h+bum⋅σi,s−σiσq⋅φ[zi]φ[zq]>1,(21)\frac{h+b}{um}\cdot\frac{\sigma_{i,s}-\sigma_i}{\sigma_q}\cdot\frac{\varphi[z_i]}{\varphi[z_q]} > 1, \qquad (21)umh+b​⋅σq​σi,s​−σi​​⋅φ[zq​]φ[zi​]​>1,(21)

then p−p^-p− is concave on (0,1](0,1](0,1] and CTBS-POUT,k⋆(γ)<CsC^{\mathrm{TBS\text{-}POUT},k^\star}(\gamma) < C^sCTBS-POUT,k⋆(γ)<Cs whenever p−(γ)<p<up^-(\gamma) < p < up−(γ)<p<u. Otherwise p−p^-p− is convex on (0,1](0,1](0,1], and whenever p>p−(γ)p > p^-(\gamma)p>p−(γ) for some γ\gammaγ, the allocation γ=1\gamma = 1γ=1 is the unique minimizer of CTBS-POUT,k⋆C^{\mathrm{TBS\text{-}POUT},k^\star}CTBS-POUT,k⋆ on [0,1][0,1][0,1] and beats CsC^sCs. The smoothing parameter α\alphaα is fixed and arbitrary in (−1,1)(-1,1)(−1,1).

Milestones

  1. Lemma 3 (Tsypkin, p. 32): an iid input of variance σx2\sigma_x^2σx2​ through a linear filter ggg has long-run output variance σx2∑tgt2\sigma_x^2 \sum_t g_t^2σx2​∑t​gt2​.
  2. (14) (p. 9): under TBS-POUT, iti_tit​ and qtq_tqt​ are normal, and their variances tend to σi2\sigma_i^2σi2​ and σq2\sigma_q^2σq2​.
  3. Proposition 1 (p. 9): the safety stock σizi\sigma_i z_iσi​zi​ and capacity μγ+σqzq\mu\gamma + \sigma_q z_qμγ+σq​zq​ are the unique optima, and
CTBS-POUT,k⋆=σi(h+b)φ[zi]+σqumφ[zq]+((u−p)γ+p)μ.C^{\mathrm{TBS\text{-}POUT},k^\star} = \sigma_i(h+b)\varphi[z_i] + \sigma_q um\varphi[z_q] + ((u-p)\gamma + p)\mu.CTBS-POUT,k⋆=σi​(h+b)φ[zi​]+σq​umφ[zq​]+((u−p)γ+p)μ.
  1. (18) (p. 11): Cs=σi,s(h+b)φ[zi]+pμC^s = \sigma_{i,s}(h+b)\varphi[z_i] + p\muCs=σi,s​(h+b)φ[zi​]+pμ.
  2. Proposition 3 (p. 11): γ⋆=0\gamma^\star = 0γ⋆=0 if u>pu > pu>p and γ⋆=1\gamma^\star = 1γ⋆=1 if u<pu < pu<p.
  3. (19) (p. 17): the closed form of CTBS-POUT,k⋆−CsC^{\mathrm{TBS\text{-}POUT},k^\star} - C^sCTBS-POUT,k⋆−Cs.
  4. (20) (p. 18): for γ∈(0,1]\gamma \in (0,1]γ∈(0,1], CTBS-POUT,k⋆(γ)<Cs  ⟺  p>p−(γ)C^{\mathrm{TBS\text{-}POUT},k^\star}(\gamma) < C^s \iff p > p^-(\gamma)CTBS-POUT,k⋆(γ)<Cs⟺p>p−(γ).

Significance

Proposition 4 is the paper's strategic prescription. It splits the parameter space by one dimensionless product of a financial factor, a variability factor and a service/flexibility factor. On one side, a small SpeedFactory serving only the demand variability is profitable in a price band p−<p<up^- < p < up−<p<u below the local unit cost. On the other side, re-shoring is all-or-nothing. Since σi,s\sigma_{i,s}σi,s​ grows like L\sqrt LL​ while σi,σq\sigma_i, \sigma_qσi​,σq​ do not depend on LLL, long off-shore lead times push towards the first regime (Figure 6 of the paper: concave at L=6L = 6L=6, convex at L=2L = 2L=2).

The printed proof of Proposition 4 is a single sentence ("readily obtained from the second derivative of p−p^-p−"). The content is in the chain behind it: the inventory under TBS-POUT is an AR(1) process, two Gaussian newsvendor problems give its optimal cost, and the order-up-to benchmark has inventory variance Lσ2L\sigma^2Lσ2. Formalizing the mission machine-checks that chain from the dynamics to the strategic conclusion. None of these results is formalized anywhere to our knowledge. Reusable outputs include the variance of a linear filter of iid noise, the Gaussian newsvendor minimization, and the long-run cost of the order-up-to policy under iid normal demand.

Difficulty

Once the closed forms (13) and (18) are known, the goal is algebra: p−(γ)=u+D/(μγ)p^-(\gamma) = u + D/(\mu\gamma)p−(γ)=u+D/(μγ) with D=(h+b)(σi−σi,s)φ[zi]+umσqφ[zq]D = (h+b)(\sigma_i - \sigma_{i,s})\varphi[z_i] + um\sigma_q\varphi[z_q]D=(h+b)(σi​−σi,s​)φ[zi​]+umσq​φ[zq​], and (21) is equivalent to D<0D < 0D<0. The work is in reaching (13) and (18) from the dynamics:

  • the long-run average is a lim inf⁡\liminfliminf of Cesàro means of expected costs, so one needs the convergence of the laws of iti_tit​ and qtq_tqt​ and the continuity of the Gaussian newsvendor cost in its mean and variance;
  • the infimum over the safety stock and the capacity has to be identified, with uniqueness, through a strictly convex limit;
  • the order-up-to system carries an LLL-period pipeline, and the exact variance Lσ2L\sigma^2Lσ2 holds only once the initial pipeline has been flushed (t≥Lt \ge Lt≥L).

Taking (13) and (18) as definitions would skip all of this and is ruled out.

Formalization scope

Lean namespace SpeedFactory.BreakEven, two definition files and eight theorem statements.

  • Demand. d : ℕ → Ω → ℝ on a probability space, mutually independent, each with law gaussianReal μ (σ²). Periods start at t=1t = 1t=1; d 0 is unused. Independence is assumed, although the printed propositions say "identically, normally distributed": (14), σi,s=σL\sigma_{i,s} = \sigma\sqrt Lσi,s​=σL​ and §5 all need it.
  • Parameters. μ,σ,h,b,u>0\mu, \sigma, h, b, u > 0μ,σ,h,b,u>0, p≥0p \ge 0p≥0, L≥2L \ge 2L≥2. The model takes m>1m > 1m>1: the paper allows m≥1m \ge 1m≥1, but at m=1m = 1m=1, zq=−∞z_q = -\inftyzq​=−∞ and the capacity problem has no minimizer. It also takes α∈(−1,1)\alpha \in (-1, 1)α∈(−1,1): the paper allows α=1\alpha = 1α=1, where σi=∞\sigma_i = \inftyσi​=∞ and (21) divides by σq=0\sigma_q = 0σq​=0.
  • Long-run cost. The paper prints (5) as a pathwise lim inf⁡\liminfliminf of realized costs. The proof of Proposition 1 computes expected costs under the limiting Gaussian law, so the formalization takes the expected Cesàro form lim inf⁡n1n∑t=1nE[⋅]\liminf_n \frac1n\sum_{t=1}^n \mathbb E[\cdot]liminfn​n1​∑t=1n​E[⋅].
  • Linearized model. Safety stock and capacity range over R\mathbb RR, and orders may be negative, as the paper's linear analysis does ("allows capacity k⋆<0k^\star < 0k⋆<0", p. 9).
  • TBS pipeline. The global receipts equal μ(1−γ)\mu(1-\gamma)μ(1−γ) from the first period on, so LLL drops out of the TBS system, as the paper states.
  • OUT benchmark. CsC^sCs is the optimized cost of the order-up-to policy from arbitrary deterministic initial inventory and pipeline. The paper's claim that this policy is optimal among all single-sourcing policies is not formalized.
  • Closed forms. Φ−1\Phi^{-1}Φ−1 is defined as sup⁡{z:Φ(z)≤r}\sup\{z : \Phi(z) \le r\}sup{z:Φ(z)≤r}, since Mathlib has no normal quantile. p−p^-p− is used only on γ∈(0,1]\gamma \in (0,1]γ∈(0,1].
  • Not trivial by construction. The optimized costs are infima of costs defined from the recursions, never the closed forms (13) or (18). The milestones pin them with attained lower bounds (IsLeast) and unique minimizers, so no junk value of sInf or liminf can make a statement true by accident. The critical fractiles are computed from the quantile, not left free.
  • Not formalized. Corollary 1 (α⋆=λ\alpha^\star = \lambdaα⋆=λ) and Proposition 2, because the goal holds for every fixed α\alphaα. Also left out: the AR(1)/IMA results of §6 (Lemmas 1, 2, 4, 5), the numerics of §4 and §7, and Table 4. That table's last column is inconsistent with its own cost columns.

Contributions welcome: proofs of the milestones, in particular a general Gaussian newsvendor lemma and the AR(1) variance recursion, which are reusable beyond this mission.

Selected references

  • R. N. Boute, S. M. Disney, J. Gijsbrechts, J. A. Van Mieghem, Dual Sourcing and Smoothing under Non-Stationary Demand Time Series: Re-shoring with SpeedFactories, accepted manuscript (revised Nov 28, 2020); published in Management Science 68(2), 2022. https://doi.org/10.1287/mnsc.2020.3951
  • L. Xin, D. A. Goldberg, Asymptotic optimality of tailored base-surge policies in dual-sourcing inventory systems, Management Science 64(1), 437–452, 2018. https://doi.org/10.1287/mnsc.2016.2607
  • R. N. Boute, J. A. Van Mieghem, Global dual sourcing and order smoothing: The impact of capacity and lead times, Management Science 61(9), 2080–2099, 2015.
  • G. Janakiraman, S. Seshadri, A. Sheopuri, Analysis of tailored base-surge policies in dual sourcing inventory systems, Management Science 61(7), 1547–1561, 2015.
  • Y. Z. Tsypkin, Sampling Systems Theory and its Application, vol. 2, Pergamon Press, Oxford, 1964.
10 thms1 active userReviewed
Operations ResearchOptimal TransportOptimization·Captain: mikedeng1

Distributionally Robust Mean-Variance Portfolio Selection with Wasserstein Distances: The Robust Problem Is Equivalent to Empirical Variance Minimization with a Norm PenaltyResearch Paper

Motivation

Portfolio selection uses observed asset returns to choose weights that balance expected return against variability. Estimates of both quantities can change sharply when the historical sample changes. A distributionally robust model addresses this uncertainty by allowing the return law to vary around the empirical law, then judging a portfolio by its worst variance and worst expected return. In the model of Blanchet, Chen, and Zhou, the allowed laws form a ball defined by an optimal transportation cost. Theorem 1 asks whether this optimization over probability laws can be expressed using only the sample and a norm of the portfolio weights.

The paper presents this equivalence as its first main contribution. Its later sections use statistical assumptions on a return time series to guide the choice of the ball radius and the return target. This mission concerns the finite-sample equivalence itself; those statistical choices are separate results. The distinction matters because the equivalence holds for a given nonnegative radius and target, without imposing a model for how the observations were generated. Theorem 1 and §3 of the preprint make these two roles explicit.

Setting

There are ddd assets and n≥1n\ge1n≥1 observed return vectors R1,…,Rn∈RdR_1,\ldots,R_n\in\mathbb R^dR1​,…,Rn​∈Rd. A portfolio is a vector φ∈Rd\varphi\in\mathbb R^dφ∈Rd whose weights sum to one. Its return when the asset return vector is xxx is φTx\varphi^TxφTx. The empirical law is Pn=n−1∑i=1nδRiP_n=n^{-1}\sum_{i=1}^n\delta_{R_i}Pn​=n−1∑i=1n​δRi​​, so empirical expectations and variances use the population normalization 1/n1/n1/n. No independence assumption is needed for this finite-sample problem.

Fix 1≤q<∞1\le q<\infty1≤q<∞ and its Hölder conjugate ppp, so 1/p+1/q=11/p+1/q=11/p+1/q=1. The endpoint q=1q=1q=1, p=∞p=\inftyp=∞ is included. Moving a return vector uuu to www costs c(u,w)=∥w−u∥q2c(u,w)=\|w-u\|_q^2c(u,w)=∥w−u∥q2​. For two probability laws P,QP,QP,Q, the transportation cost Dc(P,Q)D_c(P,Q)Dc​(P,Q) is the infimum of expected c(U,W)c(U,W)c(U,W) over joint laws whose first marginal is PPP and second marginal is QQQ. With radius δ≥0\delta\ge0δ≥0, the ambiguity set is

Uδ(Pn)={P a probability law:Dc(P,Pn)≤δ}.U_\delta(P_n)=\{P\text{ a probability law}:D_c(P,P_n)\le\delta\}.Uδ​(Pn​)={P a probability law:Dc​(P,Pn​)≤δ}.

For target return αˉ\bar\alphaαˉ, the feasible region Fδ,αˉ(n)F_{\delta,\bar\alpha}(n)Fδ,αˉ​(n) contains portfolios whose weights sum to one and whose expected return under every P∈Uδ(Pn)P\in U_\delta(P_n)P∈Uδ​(Pn​) is at least αˉ\bar\alphaαˉ. This is the paper's worst-case mean condition. For any such portfolio, the robust objective is the supremum of Var⁡P(φTR)\operatorname{Var}_P(\varphi^TR)VarP​(φTR) over the same ambiguity set. These objects are defined in §§1–2.2 of the preprint.

Formalization targets

Robust mean and mean-constrained second moment

Proposition 1 identifies the minimum expected return over the ball:

min⁡P∈Uδ(Pn)EP(φTR)=EPn(φTR)−δ ∥φ∥p.\min_{P\in U_\delta(P_n)}E_P(\varphi^TR) =E_{P_n}(\varphi^TR)-\sqrt\delta\,\|\varphi\|_p.P∈Uδ​(Pn​)min​EP​(φTR)=EPn​​(φTR)−δ​∥φ∥p​.

Proposition 3 evaluates the maximum second moment subject to a specified mean α\alphaα. Write m=EPn(φTR)m=E_{P_n}(\varphi^TR)m=EPn​​(φTR), N=∥φ∥pN=\|\varphi\|_pN=∥φ∥p​, and v=Var⁡Pn(φTR)v=\operatorname{Var}_{P_n}(\varphi^TR)v=VarPn​​(φTR). When (α−m)2≤δN2(\alpha-m)^2\le\delta N^2(α−m)2≤δN2, its value is

h(α,φ)=EPn[(φTR)2]+2(α−m)m+δN2+2δN2−(α−m)2v.h(\alpha,\varphi)=E_{P_n}[(\varphi^TR)^2]+2(\alpha-m)m+\delta N^2 +2\sqrt{\delta N^2-(\alpha-m)^2}\sqrt v.h(α,φ)=EPn​​[(φTR)2]+2(α−m)m+δN2+2δN2−(α−m)2​v​.

The two displayed steps in the proof of Theorem 1 connect this constrained value to the unconstrained worst variance: h(α,φ)−α2=(v+δN2−(α−m)2)2h(\alpha,\varphi)-\alpha^2=(\sqrt v+\sqrt{\delta N^2-(\alpha-m)^2})^2h(α,φ)−α2=(v​+δN2−(α−m)2​)2, whose maximum over allowed α≥αˉ\alpha\ge\bar\alphaα≥αˉ is (v+δN)2(\sqrt v+\sqrt\delta N)^2(v​+δ​N)2 and is attained at α=m\alpha=mα=m. These milestones retain the paper's constants, feasibility range, and attainment claims. Propositions 1 and 3 and the proof of Theorem 1 give the source statements.

Equivalence of the portfolio problems

Theorem 1 is the goal. It says that the robust problem (2) and the empirical problem (9) have both the same optimal portfolios and the same optimal value:

min⁡φ∈Fδ,αˉ(n)sup⁡P∈Uδ(Pn)Var⁡P(φTR)=min⁡φ∈Fδ,αˉ(n)(Var⁡Pn(φTR)+δ ∥φ∥p)2.\min_{\varphi\in F_{\delta,\bar\alpha}(n)} \sup_{P\in U_\delta(P_n)}\operatorname{Var}_P(\varphi^TR) =\min_{\varphi\in F_{\delta,\bar\alpha}(n)} \left(\sqrt{\operatorname{Var}_{P_n}(\varphi^TR)} +\sqrt\delta\,\|\varphi\|_p\right)^2.φ∈Fδ,αˉ​(n)min​P∈Uδ​(Pn​)sup​VarP​(φTR)=φ∈Fδ,αˉ​(n)min​(VarPn​​(φTR)​+δ​∥φ∥p​)2.

The statement also equates the minimizer sets, rather than equating values alone. The paper calls (9) the dual problem; here that name denotes this regularized empirical optimization problem. Theorem 1, p. 6 is the pinned target.

Significance

The theorem replaces an optimization over probability laws with a finite-dimensional portfolio objective containing a norm penalty. It gives a precise interpretation of that penalty: its size and exponent follow from the Wasserstein radius and transport norm. It also preserves which portfolios are optimal, so the reformulation supports the portfolio decision itself, not only its reported objective value. The target-return constraint still depends on the worst expected return; Proposition 1 gives its explicit empirical form. These consequences are stated in §2.2 of the preprint.

A complete formalization would establish the finite-sample equivalence and the attained extrema that underlie it. It would also supply reusable results about finite empirical measures, transport balls with quadratic norm costs, and portfolio moments under those balls. The pinned results are proved in the paper; the work here is to give their statements and proofs in Lean under a shared, audited model. The current local proposal consists of statements with proof placeholders, not machine-checked proofs of those results.

Difficulty

The adversary changes the whole probability law, while the robust objective subtracts the square of the law's own mean from its second moment. Maximizing the second moment alone does not therefore give the worst variance. The mean constraint in Proposition 3 and the subsequent optimization over α\alphaα are essential to recovering the centered objective. Attainment also matters: the paper states minima and maxima, whereas a bound on an infimum or supremum would be weaker. At the level of Lean definitions, every law in the quadratic transport ball must have sufficiently integrable portfolio returns for the real moment formulas to express the paper's quantities. These are the central obligations of §2.2 and Appendices A–C.

Formalization scope

Return vectors and weights are functions Fin d → ℝ; the sample is indexed by Fin n with n≥1n\ge1n≥1. Exponents are extended nonnegative reals to include p=∞p=\inftyp=∞, while qqq is finite as in the paper's fixed real exponent. The cost takes values in [0,∞][0,\infty][0,∞], and the reused transport definition fixes both coupling marginals. The ambiguity set contains all probability laws within the cost radius, with no added moment restriction. The portfolio moments are real Bochner integrals. Showing that the quadratic-cost ball makes these integrals meaningful is part of proving the results.

The robust variance objective uses an extended-real supremum. This represents an empty or unbounded indexing set without the default value of a real supremum. The feasible mean constraint is written as a universal lower bound over the ball, matching the paper's minimum constraint. The goal compares both minimizer sets and infimum values on that feasible region. It cannot be discharged by defining the robust objective to equal the regularized formula, narrowing the ambiguity set, or assuming the desired moment and value identities. Proposition 1 and Proposition 3 retain attainment through IsLeast and IsGreatest.

This mission covers the finite-sample reformulation in §§1–2.2. The time-series assumptions A1–A3, the data-driven choices of δ\deltaδ and αˉ\bar\alphaαˉ in §3, Theorem 2, and Proposition 4 are outside its scope. Contributions on transport-ball integrability, the two attained extrema, the scalar maximization, and the final equivalence would all support the goal. The general-cost dual formula of Proposition 2 is omitted because the printed formulation has a boundary counterexample at δ=0\delta=0δ=0; a faithful version needs a separate audit of its duality hypotheses.

Selected references

  • José Blanchet, Lin Chen, and Xun Yu Zhou, Distributionally Robust Mean-Variance Portfolio Selection with Wasserstein Distances, arXiv:1802.04885v1, 2018 preprint; published in Management Science, 2022. Pinned preprint.
7 thms1 active userReviewed
Algorithmic Game TheoryMechanism DesignOperations Research·Captain: mikedeng1

Improved Revenue Bounds for Posted-Price and Second-Price Mechanisms: In a Single-Unit Setting Some Sequential Posted-Price Mechanism Earns at Least 0.6543 of the Optimal RevenueResearch Paper

Motivation

A seller with one item and several buyers whose values are private but drawn from known, independent distributions can maximize expected revenue with Myerson's optimal auction (Myerson, 1981). That auction depends on every distribution through virtual values, asks buyers to report their values, and allocates through a computation that buyers must trust. A sequential posted-price (SPP) mechanism is far simpler: the seller fixes one take-it-or-leave-it price per buyer and approaches the buyers one at a time. Posted prices are common in practice, and no buyer reports anything. The natural question is how much revenue this simplicity costs, measured as the worst-case ratio between the best SPP revenue and the optimal revenue.

Timeline. Chawla, Hartline, Malec and Sivan (2010) showed that sequential posted prices earn at least 1−(1−1/n)n≥1−1/e≈0.6321 - (1 - 1/n)^n \ge 1 - 1/e \approx 0.6321−(1−1/n)n≥1−1/e≈0.632 of the optimal revenue in the single-unit setting with nnn independent buyers. Azar, Chiplunkar and Kaplan (2018) improved this to 0.63460.63460.6346 for n≥74n \ge 74n≥74, which gave the best universal bound before the present paper. No SPP mechanism can guarantee more than about 0.7450.7450.745 even for i.i.d. values (Hill and Kertz, 1982; Correa et al., 2017). Beyhaghi, Golrezaei, Paes Leme, Pál and Sivan (Operations Research, 2021; arXiv:1807.03435) proved the universal bound 0.65430.65430.6543, valid for every nnn, together with nnn-dependent, multi-unit and matroid results and a bound of 0.66200.66200.6620 for eager second-price auctions. This mission formalizes their single-unit universal bound.

Setting

There are nnn buyers i∈[n]i \in [n]i∈[n]. Buyer iii has a value viv_ivi​ drawn from a probability distribution μi\mu_iμi​ on R\mathbb RR, independently across buyers; the profile v=(v1,…,vn)v = (v_1,\dots,v_n)v=(v1​,…,vn​) has law μ1⊗⋯⊗μn\mu_1 \otimes \cdots \otimes \mu_nμ1​⊗⋯⊗μn​.

SPP(p)(p)(p). Given prices p=(p1,…,pn)p = (p_1,\dots,p_n)p=(p1​,…,pn​), the mechanism approaches buyers in decreasing order of price, offers pip_ipi​ to buyer iii while the item is unsold, and sells to the first buyer with vi≥piv_i \ge p_ivi​≥pi​ at price pip_ipi​. Its revenue at the profile vvv is therefore max⁡{pi:vi≥pi}\max\{p_i : v_i \ge p_i\}max{pi​:vi​≥pi​} (zero if nobody accepts), and SPP(p)\mathrm{SPP}(p)SPP(p) denotes its expectation.

Threshold mechanisms (the benchmark). By the taxation principle (Lemma 1 of the paper, cited from Nisan et al.), a deterministic truthful mechanism is described by thresholds ti(v−i)≥0t_i(v_{-i}) \ge 0ti​(v−i​)≥0 depending only on the other buyers' values: buyer iii wins and pays tit_iti​ when vi≥tiv_i \ge t_ivi​≥ti​. In the single-unit setting, almost surely at most one buyer weakly exceeds their threshold. The expected revenue is Opt=E[∑iti1{vi≥ti}]\mathrm{Opt} = \mathbb E[\sum_i t_i \mathbf 1\{v_i \ge t_i\}]Opt=E[∑i​ti​1{vi​≥ti​}]. Define

si⋆(τ)=P[vi≥ti≥τ],s⋆(τ)=∑isi⋆(τ),s^\star_i(\tau) = \mathbb P[v_i \ge t_i \ge \tau], \qquad s^\star(\tau) = \sum_i s^\star_i(\tau),si⋆​(τ)=P[vi​≥ti​≥τ],s⋆(τ)=i∑​si⋆​(τ),

so that Opt=∫0∞s⋆(τ) dτ\mathrm{Opt} = \int_0^\infty s^\star(\tau)\,d\tauOpt=∫0∞​s⋆(τ)dτ (Equation (1)).

Two pricing rules. Myersonian prices post to buyer iii the threshold ti′=ti(v−i′)t'_i = t_i(v'_{-i})ti′​=ti​(v−i′​) computed on an independent re-sample of the other buyers' values, freshly for each iii; MP is the resulting expected revenue and m(τ)m(\tau)m(τ) the probability that some buyer has vi≥ti′≥τv_i \ge t'_i \ge \tauvi​≥ti′​≥τ. The uniform price rule posts the same price τ\tauτ to everyone; UP=sup⁡τ≥0SPP(τ,…,τ)\mathrm{UP} = \sup_{\tau \ge 0}\mathrm{SPP}(\tau,\dots,\tau)UP=supτ≥0​SPP(τ,…,τ).

The factor-revealing program (FR). With f(x)=1−e−xf(x) = 1 - e^{-x}f(x)=1−e−x,

FR=max⁡s ∫0∞s(τ) dτs.t.0≤s(τ)≤min⁡(1,1/τ)  ∀τ≥0,∫0∞f(s(τ)) dτ≤1,s weakly decreasing.\mathrm{FR} = \max_{s}\ \int_0^\infty s(\tau)\,d\tau \quad\text{s.t.}\quad 0 \le s(\tau) \le \min(1, 1/\tau)\ \ \forall \tau \ge 0,\quad \int_0^\infty f(s(\tau))\,d\tau \le 1,\quad s \text{ weakly decreasing}.FR=smax​ ∫0∞​s(τ)dτs.t.0≤s(τ)≤min(1,1/τ)  ∀τ≥0,∫0∞​f(s(τ))dτ≤1,s weakly decreasing.

Formalization targets

Goal: Theorem 1 (p. 13)

For every nnn, every independent value distributions and every single-unit threshold mechanism with finite expected revenue Opt\mathrm{Opt}Opt, there is a vector of nonnegative prices ppp with

SPP(p) ≥ 0.6543⋅Opt.\mathrm{SPP}(p) \ \ge\ 0.6543 \cdot \mathrm{Opt}.SPP(p) ≥ 0.6543⋅Opt.

The prices depend on the instance; the constant does not.

Milestones, in the order the proof uses them

  1. Equation (1), p. 12: Opt=∫0∞s⋆(τ) dτ\mathrm{Opt} = \int_0^\infty s^\star(\tau)\,d\tauOpt=∫0∞​s⋆(τ)dτ and s⋆s^\stars⋆ is weakly decreasing.
  2. Inequality (2), p. 14: τ s⋆(τ)≤SPP(τ,…,τ)\tau\, s^\star(\tau) \le \mathrm{SPP}(\tau,\dots,\tau)τs⋆(τ)≤SPP(τ,…,τ) for τ≥0\tau \ge 0τ≥0, hence sup⁡τ≥0τs⋆(τ)≤UP\sup_{\tau \ge 0}\tau s^\star(\tau) \le \mathrm{UP}supτ≥0​τs⋆(τ)≤UP; and s⋆≤1s^\star \le 1s⋆≤1.
  3. Lemma 2, Inequality (5), p. 15: P[Zτ=0]≤(1−s⋆(τ)/n)n≤e−s⋆(τ)\mathbb P[Z_\tau = 0] \le (1 - s^\star(\tau)/n)^n \le e^{-s^\star(\tau)}P[Zτ​=0]≤(1−s⋆(τ)/n)n≤e−s⋆(τ), where ZτZ_\tauZτ​ counts buyers with vi≥ti′≥τv_i \ge t'_i \ge \tauvi​≥ti′​≥τ.
  4. Inequalities (3)–(4), p. 14: MP=∫0∞m(τ) dτ\mathrm{MP} = \int_0^\infty m(\tau)\,d\tauMP=∫0∞​m(τ)dτ and m(τ)≥1−e−s⋆(τ)m(\tau) \ge 1 - e^{-s^\star(\tau)}m(τ)≥1−e−s⋆(τ).
  5. Solving Problem (FR), p. 15: the root τ⋆>1\tau^\star > 1τ⋆>1 of (1−e−1)+∫1τ⋆(1−e−1/τ) dτ=1(1 - e^{-1}) + \int_1^{\tau^\star}(1 - e^{-1/\tau})\,d\tau = 1(1−e−1)+∫1τ⋆​(1−e−1/τ)dτ=1 is unique and lies in [1.6955,1.6965][1.6955, 1.6965][1.6955,1.6965].
  6. Solving Problem (FR), p. 15: the optimal value of (FR) is attained and equals 1+ln⁡τ⋆1 + \ln\tau^\star1+lnτ⋆, which is at most 1.52831.52831.5283.

The intermediate statement max⁡(MP,UP)≥0.6543⋅Opt\max(\mathrm{MP}, \mathrm{UP}) \ge 0.6543 \cdot \mathrm{Opt}max(MP,UP)≥0.6543⋅Opt, the "best of Myersonian and uniform" form shown first in the proof, is context only: Theorem 1 asserts one deterministic price vector, and that is the goal.

Significance

The result. Theorem 1 was the first improvement over 1−1/e1 - 1/e1−1/e for single-unit posted pricing that holds for every number of buyers and for arbitrary (regular or irregular) independent distributions. It quantifies the price of simplicity of posted pricing against the optimal auction, and its method (two pricing rules, each giving one constraint on the revenue curve s⋆s^\stars⋆, and a factor-revealing program over all curves meeting both) is reused in the paper for nnn-dependent, multi-unit, position-auction and eager second-price bounds.

Formalizing it. The result is proved on paper; no machine-checked version exists. A formalization checks the probabilistic core, the solution of an infinite-dimensional program, and the numerical constants. The printed constants carry rounding slips (τ⋆=1.696\tau^\star = 1.696τ⋆=1.696 is a rounding of 1.69572…1.69572\ldots1.69572…; 1+ln⁡τ⋆=1.52810…1 + \ln\tau^\star = 1.52810\ldots1+lnτ⋆=1.52810…, printed as 1.52831.52831.5283), which the milestones state precisely. The paper's normalization max⁡(MP,UP)=1\max(\mathrm{MP}, \mathrm{UP}) = 1max(MP,UP)=1 silently assumes that Opt is finite; the formal statement makes that hypothesis explicit.

Difficulty

The Myersonian rule alone gives only 1−1/e1 - 1/e1−1/e, and the paper's example (values uniform on [1,1+ϵ][1, 1+\epsilon][1,1+ϵ]) shows this is tight for it; the uniform rule alone is also bounded away from 1. The improvement comes only from combining them, and the combination is expressed as an optimization over all decreasing curves sss: the step that the obvious argument misses is the exact solution of (FR), whose optimum must be guessed and certified for an infinite-dimensional program with a nonlinear constraint. The probabilistic step is subtle as well: the true thresholds tit_iti​ are correlated across buyers, and Lemma 2 holds only for the independently re-sampled ti′t'_iti′​. Finally, the theorem claims a single deterministic price vector, so the random Myersonian prices and a possibly unattained best uniform price must both be converted into one concrete ppp, with the slack between 0.65430.65430.6543 and 1/FR=0.65440…1/\mathrm{FR} = 0.65440\ldots1/FR=0.65440… absorbing the approximation.

Formalization scope

Buyers are Fin n; values have law Measure.pi μ with each μ i a probability measure on ℝ; values need not be nonnegative. Revenues and probabilities are in ℝ≥0∞ (lower Lebesgue integrals and measures), never Bochner integrals, so that a non-integrable revenue cannot become a junk zero. The SPP revenue at a profile is the closed form max⁡{pi:vi≥pi}\max\{p_i : v_i \ge p_i\}max{pi​:vi​≥pi​} of the price-ordered sale. A threshold mechanism is a family t i v with t i measurable, nonnegative, independent of coordinate i, and almost surely at most one buyer with t i v ≤ v i. Myersonian prices are read from a fresh copy of the whole profile per buyer, independent of everything else. In (FR) the constraint s(τ)≤min⁡(1,1/τ)s(\tau) \le \min(1, 1/\tau)s(τ)≤min(1,1/τ) is encoded as s(τ)≤1s(\tau) \le 1s(τ)≤1 and τs(τ)≤1\tau s(\tau) \le 1τs(τ)≤1.

Benchmark. The goal is stated for every single-unit threshold mechanism, which is the taxation-principle form of Lemma 1; for the optimal mechanism's thresholds it is the printed theorem, and the paper's proof never uses optimality. Lemma 1 and Myerson's theorem are cited by the paper and are not part of the mission. The one added hypothesis on the goal, finiteness of Opt, is necessary: without it some single-unit threshold mechanisms have infinite revenue while every posted-price revenue is finite.

Ruled out. The SPP revenue is the revenue of the actual sequential sale, not a free function; thresholds may not depend on the buyer's own value (that would allow full surplus extraction) and must be single-unit feasible (otherwise Opt could be nnn times any posted-price revenue); the constant is the explicit 0.65430.65430.6543, not an existential one.

Out of scope. Theorem 2 (nnn-dependent bounds), Theorem 3 and Lemmas 3–4 (multi-unit), Lemmas 5–7 and Theorem 4 (position auctions), the partition-matroid claim and Proposition 1 (matroids), Lemma 8 and Theorems 5–6 (eager second-price auctions), Inequality (6) of Lemma 2, the (1−1/e)(1-1/e)(1−1/e) illustration of §4.1, and the numerical tables.

Infrastructure. The development needs the layer-cake formula for lower integrals, independence under product measures, the AM–GM inequality and (1−y/n)n≤e−y(1 - y/n)^n \le e^{-y}(1−y/n)n≤e−y, all in Mathlib, plus a numerical certificate for the root of a transcendental equation. The threshold-mechanism and SPP definitions are reusable for the multi-unit and matroid results of the same paper. Proofs of any milestone, alternative arguments, and sharper constants are welcome.

Selected references

  • H. Beyhaghi, N. Golrezaei, R. Paes Leme, M. Pál, B. Sivan, Improved Revenue Bounds for Posted-Price and Second-Price Mechanisms, Operations Research 69(6), 2021. arXiv:1807.03435v4. https://arxiv.org/abs/1807.03435
  • R. B. Myerson, Optimal Auction Design, Mathematics of Operations Research 6(1), 1981. https://doi.org/10.1287/moor.6.1.58
  • S. Chawla, J. D. Hartline, D. L. Malec, B. Sivan, Multi-parameter Mechanism Design and Sequential Posted Pricing, STOC 2010. https://doi.org/10.1145/1806689.1806733
  • Y. Azar, A. Chiplunkar, H. Kaplan, Prophet Secretary: Surpassing the 1 − 1/e Barrier, ACM EC 2018, 303–318. https://doi.org/10.1145/3219166.3219182
  • J. Correa, P. Foncea, R. Hoeksma, T. Oosterwijk, T. Vredeveld, Posted Price Mechanisms for a Random Stream of Customers, ACM EC 2017, 169–186. https://doi.org/10.1145/3033274.3085137
  • N. Nisan, T. Roughgarden, É. Tardos, V. V. Vazirani (eds.), Algorithmic Game Theory, Cambridge University Press, 2007, Theorem 9.36.
8 thms1 active userReviewed
Machine LearningStatistics·Captain: mikedeng1

A Robust Spectral Clustering Algorithm for Sub-Gaussian Mixture Models with Outliers 1: Robust-LP Inlier Error ≤ Cn²·max{exp(−Δ²min/64σ²max), log nmin/nmin} with Probability ≥ 1 − 2r/nminResearch Paper

Motivation

Clustering points drawn from a mixture of well-separated distributions is a basic problem of unsupervised learning. Practical data sets also contain outliers, points that belong to no cluster and may be placed anywhere. Srivastava, Sarkar and Hanasusanto (arXiv:1912.07546v3; Operations Research 71(1), 2023) propose a robust spectral clustering algorithm. It thresholds a Gaussian kernel matrix through a linear program, then applies spectral clustering to the result, and they prove that its error decays exponentially in the squared signal-to-noise ratio when the outliers, placed arbitrarily, are a small fraction of the points. The authors report that their bound is tighter than those obtained for related semidefinite relaxations, for example by Mixon, Villar and Ward (arXiv:1602.06612), which use a reference matrix in a similar way (pp. 10–11). This mission formalizes the first of the paper's two main results: the estimation error of the thresholded kernel matrix.

Setting

There are N=n+mN = n + mN=n+m points in Rd\mathbb R^dRd: nnn inliers and mmm outliers. Each inlier iii has a label φi∈{1,…,r}\varphi_i \in \{1,\dots,r\}φi​∈{1,…,r}, and the clusters are Ck={i:φi=k}\mathcal C_k = \{i : \varphi_i = k\}Ck​={i:φi​=k} with sizes nkn_knk​ and nmin⁡=min⁡knkn_{\min} = \min_k n_knmin​=mink​nk​. The inlier observation is

yi=μφi+ξi,y_i = \mu_{\varphi_i} + \xi_i ,yi​=μφi​​+ξi​,

where μ1,…,μr\mu_1,\dots,\mu_rμ1​,…,μr​ are the cluster centres and the noise vectors ξi\xi_iξi​ are independent, identically distributed within each cluster, and sub-gaussian with parameter σφi\sigma_{\varphi_i}σφi​​: for every ν∈Rd\nu \in \mathbb R^dν∈Rd, E[eν⊤ξi]≤eσφi2∥ν∥2/2\mathbb E[e^{\nu^\top\xi_i}] \le e^{\sigma_{\varphi_i}^2\|\nu\|^2/2}E[eν⊤ξi​]≤eσφi​2​∥ν∥2/2 (Definition 3). The outliers z1,…,zmz_1,\dots,z_mz1​,…,zm​ are arbitrary random vectors, with no independence or distributional assumption. Write Δkl=∥μk−μl∥\Delta_{kl} = \|\mu_k - \mu_l\|Δkl​=∥μk​−μl​∥, Δmin⁡=min⁡k≠lΔkl\Delta_{\min} = \min_{k\ne l}\Delta_{kl}Δmin​=mink=l​Δkl​ and σmax⁡=max⁡kσk\sigma_{\max} = \max_k\sigma_kσmax​=maxk​σk​.

The algorithm forms the Gaussian kernel Kij=exp⁡(−∥yi−yj∥2/2θ2)K_{ij} = \exp(-\|y_i - y_j\|^2/2\theta^2)Kij​=exp(−∥yi​−yj​∥2/2θ2) over all NNN points and solves the Robust-LP

max⁡X ⟨K−γEN,X⟩subject to0≤Xij≤1,\max_X\ \langle K - \gamma E_N, X\rangle \quad\text{subject to}\quad 0 \le X_{ij} \le 1 ,Xmax​ ⟨K−γEN​,X⟩subject to0≤Xij​≤1,

where ⟨A,B⟩=∑ijAijBij\langle A, B\rangle = \sum_{ij}A_{ij}B_{ij}⟨A,B⟩=∑ij​Aij​Bij​ and ENE_NEN​ is the all-ones matrix. Its solution has the closed form X^ij=1\hat X_{ij} = 1X^ij​=1 if Kij>γK_{ij} > \gammaKij​>γ and 000 otherwise. The true clustering matrix X0X^0X0 has Xij0=1X^0_{ij} = 1Xij0​=1 exactly when iii and jjj are inliers with the same label. The error is measured in the entrywise norm ∥A∥1=∑ij∣Aij∣\|A\|_1 = \sum_{ij}|A_{ij}|∥A∥1​=∑ij​∣Aij​∣, and AIA_{\mathcal I}AI​ denotes the inlier block.

Formalization targets

Goal: Theorem 1 (with corrected thresholds)

Fix κ>0\kappa > 0κ>0 and υ∈(0,1)\upsilon \in (0,1)υ∈(0,1), and set θ=κΔmin⁡\theta = \kappa\Delta_{\min}θ=κΔmin​, τin=e−5Δmin⁡2/64θ2\tau_{\mathrm{in}} = e^{-5\Delta_{\min}^2/64\theta^2}τin​=e−5Δmin2​/64θ2, τout=e−Δmin⁡2/4θ2\tau_{\mathrm{out}} = e^{-\Delta_{\min}^2/4\theta^2}τout​=e−Δmin2​/4θ2 and γ=υτin+(1−υ)τout\gamma = \upsilon\tau_{\mathrm{in}} + (1-\upsilon)\tau_{\mathrm{out}}γ=υτin​+(1−υ)τout​. There are constants C,C′,C′′>0C, C', C'' > 0C,C′,C′′>0, depending only on κ\kappaκ and υ\upsilonυ, such that if nmin⁡>rn_{\min} > rnmin​>r, Δmin⁡≥8σmax⁡d\Delta_{\min} \ge 8\sigma_{\max}\sqrt dΔmin​≥8σmax​d​ and m<nm < nm<n, then with probability at least 1−2r/nmin⁡1 - 2r/n_{\min}1−2r/nmin​,

∥X^I−XI0∥1≤Cn2max⁡{e−Δmin⁡2/64σmax⁡2,log⁡nmin⁡nmin⁡}\|\hat X_{\mathcal I} - X^0_{\mathcal I}\|_1 \le Cn^2\max\Big\{e^{-\Delta^2_{\min}/64\sigma^2_{\max}}, \frac{\log n_{\min}}{n_{\min}}\Big\}∥X^I​−XI0​∥1​≤Cn2max{e−Δmin2​/64σmax2​,nmin​lognmin​​}

and

∥X^−X0∥1∥X0∥1≤C′r e−Δmin⁡2/64σmax⁡2+C′′rmax⁡{log⁡nmin⁡nmin⁡,mn}.\frac{\|\hat X - X^0\|_1}{\|X^0\|_1} \le C'r\,e^{-\Delta^2_{\min}/64\sigma^2_{\max}} + C''r\max\Big\{\frac{\log n_{\min}}{n_{\min}}, \frac{m}{n}\Big\}.∥X0∥1​∥X^−X0∥1​​≤C′re−Δmin2​/64σmax2​+C′′rmax{nmin​lognmin​​,nm​}.

The constants are left existential, as on the page.

Milestones

The milestones follow the structure of the paper's proof:

  • the closed form of X^\hat XX^ (§3);
  • Lemmas 1–3, which compare Robust-LP with a reference problem built from the reference matrix RRR of display (6);
  • the counting bound (E.2);
  • the sub-gaussian tail bounds (E.5), (E.6), (E.7)–(E.8) and (E.9)–(E.10);
  • the Bernstein inequalities for one- and two-sample U-statistics (E.11) and (E.14), and their consequences (E.13) and (E.15);
  • the threshold gap (E.18).

Significance

Theorem 1 is the quantitative core of the paper. It says that thresholding a Gaussian kernel at a level between two explicit values recovers the cluster structure among the inliers up to an error that is exponentially small in Δmin⁡2/σmax⁡2\Delta^2_{\min}/\sigma^2_{\max}Δmin2​/σmax2​, plus a sampling term of order log⁡nmin⁡/nmin⁡\log n_{\min}/n_{\min}lognmin​/nmin​, whatever the outliers do. The paper's Theorem 2 feeds this bound into Davis–Kahan perturbation theory to bound the fraction of misclassified points after spectral rounding, and Proposition 2 uses it to identify the outliers.

Formalizing it produces more than the theorem. As printed, Theorem 1 is false: the kernel uses 2θ22\theta^22θ2 while the thresholds use θ2\theta^2θ2, a factor-two mismatch that the proof silently drops. A mechanically checked statement settles the corrected thresholds. The proof also rests on three cited probabilistic results that are not proved in the paper and are not in Mathlib: the Hsu–Kakade–Zhang tail bound for quadratic forms of sub-gaussian vectors, and Bernstein inequalities for one- and two-sample U-statistics (the two-sample version is asserted "following the arguments" of earlier work, with no proof). None of these results has a machine-checked proof at present.

Difficulty

The deterministic part (Lemmas 1–3, (E.2), (E.18)) is elementary linear algebra on [0,1][0,1][0,1]-matrices. The difficulty is probabilistic. The kernel entries KijK_{ij}Kij​ are not independent: each point appears in n−1n - 1n−1 entries. So the natural first idea, a Chernoff or Hoeffding bound on the number of threshold violations treated as a sum of independent indicators, does not apply. The paper handles the dependence by writing the counts of violations as U-statistics and invoking Bernstein-type bounds for them. Those bounds need Hoeffding's representation of a U-statistic as an average of sums of independent terms, which is not available in Mathlib. A second obstacle is the Hsu–Kakade–Zhang inequality: the norm of a sub-gaussian vector whose coordinates may be dependent concentrates at the scale ddd, not d2d^2d2, and the coordinate-independent Hanson–Wright inequality does not give it.

Formalization scope

  • Points and labels. Points are indexed by Fin n ⊕ Fin m (inliers, then outliers) and vectors live in EuclideanSpace ℝ (Fin d). The labels φ\varphiφ are fixed. The paper draws them from a multinomial distribution, but its proof conditions on them, and nmin⁡n_{\min}nmin​ appears in the probability bound.
  • Sub-gaussian parameter. σk\sigma_kσk​ is the Definition 3 parameter, encoded through Mathlib's HasSubgaussianMGF of every projection ν⊤ξ\nu^\top\xiν⊤ξ. The proof uses this parameter, not the "largest eigenvalue of the second moment matrix" mentioned in §2.2.
  • Minima and maxima. Δmin⁡\Delta_{\min}Δmin​, nmin⁡n_{\min}nmin​, σmax⁡\sigma_{\max}σmax​ and τout\tau_{\mathrm{out}}τout​ are finite minima or maxima over nonempty index sets; r≥2r \ge 2r≥2 and d≥1d \ge 1d≥1 are hypotheses.
  • Corrections and additions. The thresholds are corrected from the printed e−5Δmin⁡2/32θ2e^{-5\Delta^2_{\min}/32\theta^2}e−5Δmin2​/32θ2 and e−Δkl2/2θ2e^{-\Delta^2_{kl}/2\theta^2}e−Δkl2​/2θ2 to e−5Δmin⁡2/64θ2e^{-5\Delta^2_{\min}/64\theta^2}e−5Δmin2​/64θ2 and e−Δkl2/4θ2e^{-\Delta^2_{kl}/4\theta^2}e−Δkl2​/4θ2, which are the values for which (E.7)–(E.10) hold. "θ=Θ(Δmin⁡)\theta = \Theta(\Delta_{\min})θ=Θ(Δmin​)" is read as θ=κΔmin⁡\theta = \kappa\Delta_{\min}θ=κΔmin​, as on p. 42. m<nm < nm<n is added for (12), as the proof uses it at (E.20).
  • Probability. The failure event's outer measure is bounded, and outliers need not be measurable.
  • Ruling out trivial statements. The goal does not mention the reference matrix, the corruption counts, U-statistics or ρmin⁡\rho_{\min}ρmin​, and its constants are quantified before every model datum. With CCC fixed, letting nmin⁡n_{\min}nmin​ and the signal-to-noise ratio grow forces ∥X^I−XI0∥1/n2→0\|\hat X_{\mathcal I} - X^0_{\mathcal I}\|_1/n^2 \to 0∥X^I​−XI0​∥1​/n2→0, so (11) is not implied by the trivial bound ∥X^I−XI0∥1≤n2\|\hat X_{\mathcal I} - X^0_{\mathcal I}\|_1 \le n^2∥X^I​−XI0​∥1​≤n2.
  • Reusable work. Proofs of the sub-gaussian tail bounds and the U-statistic Bernstein inequalities are welcome as standalone developments. They are reusable well beyond this mission, for example in kernel methods, random geometric graphs and two-sample testing.

Selected references

  • P. R. Srivastava, P. Sarkar, G. A. Hanasusanto, A Robust Spectral Clustering Algorithm for Sub-Gaussian Mixture Models with Outliers, Operations Research 71(1), 2023; arXiv:1912.07546v3, 2021. https://arxiv.org/abs/1912.07546
  • D. Hsu, S. Kakade, T. Zhang, A tail inequality for quadratic forms of subgaussian random vectors, Electronic Communications in Probability 17, 2012. https://doi.org/10.1214/ECP.v17-2079
  • W. Hoeffding, Probability inequalities for sums of bounded random variables, Journal of the American Statistical Association 58, 1963. https://doi.org/10.1080/01621459.1963.10500830
  • M. A. Arcones, A Bernstein-type inequality for U-statistics and U-processes, Statistics & Probability Letters 22(3):239–247, 1995 (as cited in arXiv:1912.07546v3, reference list). https://arxiv.org/abs/1912.07546
  • Y. Pitcan, A note on concentration inequalities for U-statistics, arXiv:1712.06160, 2017. https://arxiv.org/abs/1712.06160
  • D. G. Mixon, S. Villar, R. Ward, Clustering subgaussian mixtures by semidefinite programming, Information and Inference 6(4), 2017; arXiv:1602.06612. https://arxiv.org/abs/1602.06612
16 thms1 active userReviewed
Operations ResearchRandom Matrix TheoryStatistics·Captain: mikedeng1

The Risk of Expected Utility Under Parameter Uncertainty I: Closed-Form Out-of-Sample Utility Variance of the Shrinkage Mean-Variance PortfolioResearch Paper

Why out-of-sample utility risk matters

A portfolio chosen from historical returns can look attractive in the data used to construct it and deliver a different result when evaluated under the population return distribution. The gap is driven by estimation error in both the mean return and covariance matrix. A mean of future utility values describes average performance, but it does not describe how much that utility varies across possible estimation samples. Lassance, Martín-Utrera and Simaan study this variation for a portfolio that blends an estimated mean-variance choice with an estimated global-minimum-variance choice. Their closed-form variance makes the sampling risk of that blend an explicit function of market and sample parameters. The source for this mission is the authors’ SSRN manuscript, version dated August 18, 2023, including its Internet Appendix; appendix pages are cited as IA pages.

The market and the estimated portfolio

There are NNN risky stocks. Their excess-return vector has population mean μ∈RN\mu\in\mathbb R^Nμ∈RN and positive-definite population covariance Σ∈RN×N\Sigma\in\mathbb R^{N\times N}Σ∈RN×N. The investor has risk aversion γ>0\gamma>0γ>0. A portfolio is a vector of weights www whose entries sum to one. Its population mean-variance utility is

U(w)=w⊤μ−γ2w⊤Σw.U(w)=w^\top\mu-\frac{\gamma}{2}w^\top\Sigma w.U(w)=w⊤μ−2γ​w⊤Σw.

Write eee for the vector of ones. The global-minimum-variance portfolio and its variance are wg=Σ−1e(e⊤Σ−1e)−1w_g=\Sigma^{-1}e(e^\top\Sigma^{-1}e)^{-1}wg​=Σ−1e(e⊤Σ−1e)−1 and σg2=(e⊤Σ−1e)−1\sigma_g^2=(e^\top\Sigma^{-1}e)^{-1}σg2​=(e⊤Σ−1e)−1. Its mean is μg=μ⊤Σ−1e σg2\mu_g=\mu^\top\Sigma^{-1}e\,\sigma_g^2μg​=μ⊤Σ−1eσg2​. The population matrix B=Σ−1(I−ewg⊤)B=\Sigma^{-1}(I-ew_g^\top)B=Σ−1(I−ewg⊤​) maps the mean vector to a zero-cost portfolio. The variance of that portfolio is ψ2=μ⊤Σ−1μ−μg2/σg2\psi^2=\mu^\top\Sigma^{-1}\mu-\mu_g^2/\sigma_g^2ψ2=μ⊤Σ−1μ−μg2​/σg2​. These quantities are computed from μ\muμ and Σ\SigmaΣ; they are not independent inputs. These definitions are equations (3)–(7) of the manuscript.

The investor observes TTT independent return vectors r1,…,rTr_1,\ldots,r_Tr1​,…,rT​, each with multivariate Gaussian law N(μ,Σ)\mathcal N(\mu,\Sigma)N(μ,Σ). The sample mean is μ^=T−1∑trt\hat\mu=T^{-1}\sum_t r_tμ^​=T−1∑t​rt​, and the sample covariance is Σ^=T−1∑t(rt−μ^)(rt−μ^)⊤\hat\Sigma=T^{-1}\sum_t(r_t-\hat\mu)(r_t-\hat\mu)^\topΣ^=T−1∑t​(rt​−μ^​)(rt​−μ^​)⊤. Replacing the population parameters in wgw_gwg​ and BBB by these estimates gives the sample GMV portfolio w^g\hat w_gw^g​ and the sample zero-cost portfolio w^z=B^μ^\hat w_z=\hat B\hat\muw^z​=B^μ^​. The estimated mean-variance portfolio is w^⋆=w^g+γ−1w^z\hat w^\star=\hat w_g+\gamma^{-1}\hat w_zw^⋆=w^g​+γ−1w^z​. At shrinkage intensity κ∈[0,1]\kappa\in[0,1]κ∈[0,1], the portfolio under study is w^⋆(κ)=(1−κ)w^g+κw^⋆\hat w^\star(\kappa)=(1-\kappa)\hat w_g+\kappa\hat w^\starw^⋆(κ)=(1−κ)w^g​+κw^⋆. Its out-of-sample utility, U(w^⋆(κ))U(\hat w^\star(\kappa))U(w^⋆(κ)), is evaluated using population μ\muμ and Σ\SigmaΣ. It is random because the weights depend on the sample. These choices are fixed by equations (9)–(12) and Assumption 2 of the manuscript.

Formalization targets

The goal is Proposition 1, equations (13)–(19). Under N≥2N\ge2N≥2 and T>N+7T>N+7T>N+7, it asserts that the utility has a finite second moment and that

V[U(w^⋆(κ))]=V[U(w^g)]+Δ(κ),Δ(κ)=a1κ4+a2κ3+a3κ2+a4κ.\mathbb V[U(\hat w^\star(\kappa))] =\mathbb V[U(\hat w_g)]+\Delta(\kappa),\qquad \Delta(\kappa)=a_1\kappa^4+a_2\kappa^3+a_3\kappa^2+a_4\kappa.V[U(w^⋆(κ))]=V[U(w^g​)]+Δ(κ),Δ(κ)=a1​κ4+a2​κ3+a3​κ2+a4​κ.

The coefficients a1,…,a4a_1,\ldots,a_4a1​,…,a4​ and V[U(w^g)]\mathbb V[U(\hat w_g)]V[U(w^g​)] are the exact rational expressions in T,N,γ,σg2,ψ2T,N,\gamma,\sigma_g^2,\psi^2T,N,γ,σg2​,ψ2 on pp. 14–15 of the manuscript. In particular, the result covers every κ∈[0,1]\kappa\in[0,1]κ∈[0,1], including the sample GMV portfolio at κ=0\kappa=0κ=0 and the sample mean-variance portfolio at κ=1\kappa=1κ=1.

The milestones follow the quantities named in the authors’ proof, Section IA.10. Equation (IA43) gives the covariance matrix of the random weights, and (IA44) gives the variance of their population mean return. The jointly distributed mean-return and return-variance pair in (IA45)–(IA48) supplies the setting for two auxiliary moment identities, (IA53) and (IA57). Equations (IA49), (IA59), and (IA60) give the first and second moments needed for return variance and mean return; (IA68) gives their covariance. All are statements from the Internet Appendix, pp. IA35–IA39.

What the result supplies

The expression tells an investor how the sampling risk of utility changes with the shrinkage intensity, sample size, stock count, risk aversion, and the two population variance parameters. The later comparison of variance-minimizing, mean-maximizing, and robustness-maximizing intensities uses this formula. Without Proposition 1, those comparisons have no explicit OOSU risk function to analyze. The result is already proved in the paper; this mission asks for its Lean proof and for reusable formal statements of the associated Gaussian portfolio and moment identities. No machine-checked proof of this paper’s proposition is included in the draft.

Where the difficulty lies

The sample covariance is random and its inverse appears in the estimated weights. Taking the variance of utility therefore involves inverse-covariance terms and moments beyond those needed for expected utility. The requirement T>N+7T>N+7T>N+7 is tied to existence of the fourth negative moment of a chi-squared variable in (IA57). A direct calculation that replaces random weights by their population counterparts loses the dependence on the estimation sample. A calculation from separate marginal laws also does not determine the covariance between mean return and return variance; the representation in (IA47)–(IA48) describes them jointly.

Formalization scope

The Lean market uses vectors Fin N → ℝ, matrices indexed by Fin N, and returns in EuclideanSpace ℝ (Fin N) to use Mathlib’s multivariate Gaussian law. The sample has Fin T observations on a probability space. Its covariance divisor is exactly TTT, as in (9). The standing assumptions are N≥2N\ge2N≥2, T>N+7T>N+7T>N+7, Σ≻0\Sigma\succ0Σ≻0, γ>0\gamma>0γ>0, and independent, identically distributed Gaussian returns. Every theorem about a variance or covariance explicitly asserts finite second moments; expectation statements explicitly assert integrability. The source’s finite-variance convention is thereby represented without Lean’s default value for a nonintegrable variance.

The left side of the goal is the probability-theory variance of utility under the sampling law. It is not defined by the closed formula. Likewise σg2\sigma_g^2σg2​, μg\mu_gμg​, and ψ2\psi^2ψ2 are computed from the market; independence is part of the sampling hypothesis. In the auxiliary representation, the chi-squared law is the squared norm of independent standard normals. The zero-degree cases and footnote 6’s N=2,3N=2,3N=2,3 rules are encoded explicitly. Mathlib’s matrix inverse and real division are total operations; their values on singular sample covariances or zero denominators do not alter the distributions because those events have probability zero under the stated assumptions. Contributions establishing the sample-covariance distribution, the joint representation, the negative moments, and the final variance identity are within scope and can support other Gaussian portfolio results.

Selected references

  • N. Lassance, A. Martín-Utrera, and M. Simaan, The Risk of Expected Utility under Parameter Uncertainty, authors’ SSRN manuscript dated August 18, 2023, subsequently in Management Science (2024). SSRN 3855546; DOI 10.1287/mnsc.2023.00178.
12 thms1 active userReviewed
Convex OptimizationOperations ResearchStochastic Systems·Captain: mikedeng1

Real-Time Spatial–Intertemporal Pricing and Relocation in a Ride-Hailing Network: Near-Optimal Policies and the Value of Dynamic Pricing 1: Arc Balancing Control Has Loss O(log(1+n)^{3/2}/n^{2/3})Research Paper

Why spatial pricing needs a dynamic control

A ride-hailing firm chooses prices while drivers move through a city. A trip from one region uses a driver immediately and returns that driver at another region only after a travel delay. Prices affect which trips are requested, but the actual requests fluctuate. A plan that ignores these fluctuations can run out of available drivers at a busy origin even when its average flows fit the fleet. Chen, Lei, and Jasin study how closely a feasible pricing control can approach a deterministic revenue benchmark in this setting (accepted manuscript, published article).

Their Arc Balancing Control (ABC) responds to realized trips. It groups each directed arc's periods into batches and adjusts the next batch's demand rates according to the previous batch's cumulative surplus or deficit. The paper proves a loss rate for ABC that is smaller than the rate established there for its Static Price Control. The target here is the paper's Theorem 3, with the model and parameters of its authors' accepted manuscript. Its e-companion contains the paper's proofs.

The pricing network and benchmark

The network has N≥1N\ge1N≥1 regions and a horizon of T≥1T\ge1T≥1 periods. Every ordered pair i→ji\to ji→j, including i=ji=ji=j, is an arc. At most one customer can request an arc's trip in a period. A served trip takes one server from region iii for τij≥1\tau_{ij}\ge1τij​≥1 periods, then returns it at jjj. Region iii starts with Ci≥NC_i\ge NCi​≥N servers. A control must respect the available server count before it serves each period's realized requests.

The decision is a demand rate λt,ij∈[0,1]\lambda_{t,ij}\in[0,1]λt,ij​∈[0,1] rather than its inverse price. The expected revenue from that arc and period is rt,ij(λt,ij)r_{t,ij}(\lambda_{t,ij})rt,ij​(λt,ij​). The revenue functions are nonnegative and strongly concave, vanish at zero, and have bounded first and second derivatives. The bound on those derivatives is Ψ3>0\Psi_3>0Ψ3​>0. The paper's price-side assumptions imply these rate-side conditions; the Lean model states the conditions directly.

The deterministic convex program DCP(ζ)\mathrm{DCP}(\zeta)DCP(ζ) chooses rates to maximize the sum of expected revenues. Its flow constraints replace realized trips with their rates, while its cushion requires ζ≤λt,ij≤1−ζ\zeta\le\lambda_{t,ij}\le1-\zetaζ≤λt,ij​≤1−ζ. Write J(ζ)J(\zeta)J(ζ) for its value and λζ,D\lambda^{\zeta,D}λζ,D for an optimal schedule. In the nnnth system, the fleet, horizon, and travel times are multiplied by nnn; each original period's revenue curve is used for nnn consecutive scaled periods. Thus Tn=nTT^n=nTTn=nT and τ‾n=nmin⁡i,jτij\underline\tau^n=n\min_{i,j}\tau_{ij}τ​n=nmini,j​τij​. For a control π\piπ, the per-period benchmark loss is

Lπ(n)=J(n,0)−Eπ[Rπ(n)]Tn.L^\pi(n)=\frac{J(n,0)-\mathbb E^\pi[R^\pi(n)]}{T^n}.Lπ(n)=TnJ(n,0)−Eπ[Rπ(n)]​.

ABC uses λn,ζ,D\lambda^{n,\zeta,D}λn,ζ,D as its baseline. Its batches have a target sum bbb of the baseline variance weights λt,ijn,ζ,D(1−λt,ijn,ζ,D)\lambda^{n,\zeta,D}_{t,ij}(1-\lambda^{n,\zeta,D}_{t,ij})λt,ijn,ζ,D​(1−λt,ijn,ζ,D​). Within a batch, the control subtracts a weighted share of the preceding batch's realized surplus δˉijk=∑t∈Tijk(Dt,ij−λt,ijABC)\bar\delta^k_{ij}=\sum_{t\in\mathcal T^k_{ij}}(D_{t,ij}-\lambda^{ABC}_{t,ij})δˉijk​=∑t∈Tijk​​(Dt,ij​−λt,ijABC​). A rate is projected to [0,1][0,1][0,1], and an origin with too few available servers is turned off. The finite path law uses the rates posted along each path.

Formalization targets

The principal target is Theorem 3. With bn=(τ‾n)2/3b_n=(\underline\tau^n)^{2/3}bn​=(τ​n)2/3 and

ϵn=(τ‾n)−1(1+162(bn+1)log⁡Tn),ζn=2ϵn,\epsilon_n=(\underline\tau^n)^{-1}\left(1+16\sqrt{2(b_n+1)\log T^n}\right),\qquad \zeta_n=2\epsilon_n,ϵn​=(τ​n)−1(1+162(bn​+1)logTn​),ζn​=2ϵn​,

there is a constant Ψ~2\widetilde\Psi_2Ψ2​, independent of nnn, for which every integer n≥1n\ge1n≥1 satisfies

LABC(n)≤Ψ~2log⁡(1+n)3/2n2/3.L^{ABC}(n)\le\widetilde\Psi_2\frac{\log(1+n)^{3/2}}{n^{2/3}}.LABC(n)≤Ψ2​n2/3log(1+n)3/2​.

The milestone list follows the source results used in the analysis: Lemma 2 bounds the deterministic cost of a small cushion and gives feasibility; Lemma EC.1 bounds a maximal cumulative Bernoulli deviation; and the two parts of Lemma 4 characterize ABC on a batchwise good event A\mathcal AA and bound that event's probability. Theorem 3 fixes the complete control and these exact parameters; its constant may depend on the fixed network instance and Ψ3\Psi_3Ψ3​.

What the result establishes

The theorem gives a finite-nnn loss bound relative to J(n,0)J(n,0)J(n,0), with the stated n−2/3n^{-2/3}n−2/3 rate and logarithmic factor. In the paper, Lemma 1 shows that J(n,0)J(n,0)J(n,0) bounds the optimal admissible policy's revenue. Theorem 3 therefore bounds ABC's distance from a benchmark that is at least as high as the unknown stochastic optimum. The paper compares this rate with its n−1/2n^{-1/2}n−1/2 result for Static Price Control and explains the resulting value of adapting prices to realized demand (Theorems 2–3).

The published mathematical result and its manuscript proof are known; the statements in this mission are Lean targets with proofs still to be formalized. A complete development would also make reusable definitions for finite controlled path laws, delayed network flow balance, and cushioned deterministic relaxations. The batch and feedback definitions isolate the paper-specific control from that general infrastructure.

Main difficulty

The deterministic program controls expected flows, while ABC must keep every realized path feasible. An unadjusted baseline can accumulate demand fluctuations over the whole horizon; bounding those fluctuations alone does not describe the effect of ABC's feedback. The control's posted rate in a period depends on earlier realized requests, which changes the distribution of later requests. The manuscript's proof of Lemma 4 bounds a variance proxy from its deterministic baseline when treating the adaptive posted rates; that step needs separate justification in a formal proof. Lemma EC.1, as posed here, covers deterministic Bernoulli rates and does not by itself close that gap. The finite-horizon claim, including small scaling factors for which the cushioned DCP may be infeasible, must still be established.

Formalization scope

Lean represents regions by Fin N, periods by natural numbers 1,…,T1,\ldots,T1,…,T, travel times and server counts by naturals, demand by Boolean paths, and revenue and rates by real numbers. The DCP(n,ζ)\mathrm{DCP}(n,\zeta)DCP(n,ζ) program uses a real supremum; statements involving positive ζ\zetaζ assert feasibility, and J(n,0)J(n,0)J(n,0) has the zero schedule as a feasible point. ABC's path law is the product of sequential Bernoulli factors at its own posted rates. Its rate history is built by structural recursion, and the event A\mathcal AA checks every nonempty batch and all its partial sums.

The accepted manuscript leaves the final incomplete batch unspecified, so the last short batch is merged with its predecessor. The printed ABC box turns an origin off at Ct,i≤NC_{t,i}\le NCt,i​≤N, while Lemma 4 and its base case require service when Ct,i=NC_{t,i}=NCt,i​=N; the formal control uses Ct,i≥NC_{t,i}\ge NCt,i​≥N. If the cushioned DCP has no optimum, the baseline is zero. The sample-path part of Lemma 4 adds the hypothesis τ‾≤T\underline\tau\le Tτ​≤T, which the manuscript does not print. Without it the part is false: when the horizon is shorter than every travel time, no server returns, and a single realized trip can leave a region below NNN servers on the good event. These conventions are recorded with the source discrepancies in the moderation notes. The rate model drops assumptions A1 and A3(i), which concern only the price parametrization, and writes A2, A3(ii), r≥0r\ge0r≥0, and r(0)=0r(0)=0r(0)=0 directly. It assumes a twice continuously differentiable extension of revenue to R\mathbb RR. No statement substitutes expected demand for the realized batch surplus or lets the theorem's parameters depend on nnn by existential choice. Contributions toward the source milestones, the adaptive concentration step, and the reusable model definitions are all within scope.

Selected references

  • Q. Chen, Y. Lei, and S. Jasin, Real-Time Spatial-Intertemporal Pricing and Relocation in a Ride-Hailing Network: Near Optimal Heuristics and The Value of Dynamic Pricing, authors' accepted manuscript, LBS Research Online eprint 2845, pp. 7–25 and ec2–ec6; published as Real-Time Spatial–Intertemporal Pricing and Relocation in a Ride-Hailing Network: Near-Optimal Policies and the Value of Dynamic Pricing, Operations Research 72(5):2097–2118, 2024. Manuscript, DOI.
7 thms1 active userReviewed
Algorithmic Game TheoryEconomicsOperations Research·Captain: mikedeng1

Information Design for Congested Social Services: Optimal Need-Based Persuasion 2: With Only Low-Need Users, Full Information Is Within a Factor of the Optimal Signaling MechanismResearch Paper

Motivation

Social services such as public housing, food banks and free clinics are congested: the users who arrive wait in a queue they cannot see, and some of them could do almost as well elsewhere. A provider who sees the queue can decide what to tell arriving users about it. Telling low-need users that the queue is long sends them to their outside option and shortens the wait for everyone else; telling them too much may send away users the provider would rather serve. Information design (Bayesian persuasion, Kamenica and Gentzkow 2011) studies what a sender who commits to a signaling rule in advance can achieve.

Anunrojwong, Iyer and Manshadi (2023) build on the queueing persuasion model of Lingenbrink and Iyer (2019) and ask how much welfare information design can add in a congested service. Their answer depends on whether users are heterogeneous. This mission formalizes the homogeneous case, Proposition 1: when every user is low-need, the simplest policy, full disclosure of the queue length, is already within an explicit factor of the best signaling mechanism.

Setting

A single server with exponential service times of rate one serves users first come, first served. Low-need users (type L) arrive as a Poisson process of rate λL>0\lambda_L > 0λL​>0 and may leave for an outside option worth 000; high-need users (type H, rate λH≥0\lambda_H \ge 0λH​≥0) always join. The total rate is λ=λL+λH≤1\lambda = \lambda_L + \lambda_H \le 1λ=λL​+λH​≤1. A type-iii user who joins with nnn users present receives utility ui(n)u_i(n)ui​(n), where ui(0)>0u_i(0) > 0ui​(0)>0, uiu_iui​ is strictly decreasing, ui(k)<0u_i(k) < 0ui​(k)<0 for large kkk, and the increments uL(n)−uL(n+1)u_L(n) - u_L(n+1)uL​(n)−uL​(n+1) are non-increasing (Assumption 1).

By Lemma 1 of the paper, an obedient signaling mechanism is described by the stationary distribution π\piπ of the queue length on N0\mathbb N_0N0​ it induces. The admission policies ΠAP\Pi_{AP}ΠAP​ are the probability distributions with

λHπn≤πn+1≤λπn(n≥0),\lambda_H \pi_n \le \pi_{n+1} \le \lambda \pi_n \qquad (n \ge 0),λH​πn​≤πn+1​≤λπn​(n≥0),

and the signaling mechanisms ΠSM\Pi_{SM}ΠSM​ are those that also satisfy the obedience constraints

J(π)=∑n≥0(πn+1−λHπn) uL(n)≥0,L(π)=∑n≥0(λπn−πn+1) uL(n)≤0.J(\pi) = \sum_{n\ge0}(\pi_{n+1} - \lambda_H\pi_n)\,u_L(n) \ge 0, \qquad L(\pi) = \sum_{n\ge0}(\lambda\pi_n - \pi_{n+1})\,u_L(n) \le 0 .J(π)=n≥0∑​(πn+1​−λH​πn​)uL​(n)≥0,L(π)=n≥0∑​(λπn​−πn+1​)uL​(n)≤0.

The welfare of low-need users is WL(π)=J(π)W_L(\pi) = J(\pi)WL​(π)=J(π).

A threshold policy πx\pi_xπx​ with x=m+ax = m + ax=m+a, m∈N0m \in \mathbb N_0m∈N0​, a∈[0,1)a \in [0,1)a∈[0,1), admits L-users below queue length mmm, admits them with probability aaa at mmm, and turns them away above mmm. WL(x)W_L(x)WL​(x) denotes WL(πx)W_L(\pi_x)WL​(πx​). The full-information threshold mfim_{fi}mfi​ is the least kkk with uL(k)<0u_L(k) < 0uL​(k)<0, and the full-information mechanism is fi=πmfi\mathrm{fi} = \pi_{m_{fi}}fi=πmfi​​. The homogeneous setting is λH=0\lambda_H = 0λH​=0.

Formalization targets

Goal: Proposition 1 (p. 16)

In the homogeneous setting, with

βfi=∑n=0mfi−1λLn∑n=0mfiλLn,\beta_{fi} = \frac{\sum_{n=0}^{m_{fi}-1}\lambda_L^n}{\sum_{n=0}^{m_{fi}}\lambda_L^n},βfi​=∑n=0mfi​​λLn​∑n=0mfi​−1​λLn​​,

the goal asserts three facts:

  1. every signaling mechanism π∈ΠSM\pi \in \Pi_{SM}π∈ΠSM​ satisfies WL(fi)≥βfi WL(π)W_L(\mathrm{fi}) \ge \beta_{fi}\, W_L(\pi)WL​(fi)≥βfi​WL​(π);
  2. βfi≥1−1/(mfi+1)\beta_{fi} \ge 1 - 1/(m_{fi}+1)βfi​≥1−1/(mfi​+1);
  3. full information maximizes WLW_LWL​ over ΠSM\Pi_{SM}ΠSM​ if and only if WL(mfi)≥WL(mfi−1)W_L(m_{fi}) \ge W_L(m_{fi}-1)WL​(mfi​)≥WL​(mfi​−1).

The paper phrases parts 1 and 3 for "the optimal signaling mechanism sm". They are stated here for every element of ΠSM\Pi_{SM}ΠSM​, which implies the paper's form for any maximizer and requires no existence result.

Milestones

  • Lemma 3 (p. 36), the detailed-balance sandwich λHk−qπq≤πk≤λk−qπq\lambda_H^{k-q}\pi_q \le \pi_k \le \lambda^{k-q}\pi_qλHk−q​πq​≤πk​≤λk−qπq​.
  • Theorem 1 (p. 14): every Pareto-efficient signaling mechanism has a threshold structure with threshold at most mfim_{fi}mfi​.
  • Lemma 5(2) (p. 37): WL(x)W_L(x)WL​(x) is monotone between consecutive integers and unimodal on R+\mathbb R_+R+​.
  • From the proof of Proposition 1 (p. 40):
    • full information is a feasible signaling mechanism with WL(fi)>0W_L(\mathrm{fi}) > 0WL​(fi)>0;
    • every obedient threshold is at least mfi−1m_{fi}-1mfi​−1;
    • βfi WL(mfi−1)≤WL(fi)\beta_{fi}\,W_L(m_{fi}-1) \le W_L(\mathrm{fi})βfi​WL​(mfi​−1)≤WL​(fi);
    • βfi≥1−1/(mfi+1)\beta_{fi} \ge 1 - 1/(m_{fi}+1)βfi​≥1−1/(mfi​+1) for all λL≤1\lambda_L \le 1λL​≤1.

Significance

Proposition 1 bounds what persuasion can do when users are alike. The ratio 1/βfi1/\beta_{fi}1/βfi​ is at most 1+1/mfi1 + 1/m_{fi}1+1/mfi​, so the gain from optimal signaling is small whenever full information already admits a few users. The paper contrasts this with the heterogeneous setting (§4.2), where information design can Pareto-improve on full information substantially. The homogeneous bound is the baseline for that comparison: it shows that the gains there come from the presence of two types, not from pooling queue states.

The result is proved in the paper. Nothing in this model has been machine-checked so far. A formal proof would check the reduction from an arbitrary signaling mechanism to a threshold mechanism (Theorem 1), the unimodality of threshold welfare (Lemma 5(2)), and the obedience calculation on p. 40. It would also confirm that these hold at λL=1\lambda_L = 1λL​=1, where the stationary distribution of the always-join policy no longer exists. The model layer defined here is shared with three sibling missions on Theorems 1 and 6 and Proposition 3 of the same paper.

Difficulty

The two finite inequalities, the bound on βfi\beta_{fi}βfi​ and the comparison of WL(mfi−1)W_L(m_{fi}-1)WL​(mfi​−1) with WL(fi)W_L(\mathrm{fi})WL​(fi), are elementary. The difficulty lies in part 1 for an arbitrary π∈ΠSM\pi \in \Pi_{SM}π∈ΠSM​, which is a distribution on an infinite state space. Restricting attention to threshold policies is not free: Theorem 1 only says that Pareto-efficient mechanisms are thresholds. An arbitrary obedient π\piπ must either be shown to be dominated by a threshold mechanism, or bounded directly. The sets ΠSM\Pi_{SM}ΠSM​ are not compact in the obvious topology at λL=1\lambda_L = 1λL​=1, so the existence of an optimal mechanism cannot simply be assumed.

Part 3 needs a strict comparison. When WL(mfi−1)>WL(mfi)W_L(m_{fi}-1) > W_L(m_{fi})WL​(mfi​−1)>WL​(mfi​), the proof must exhibit an obedient threshold policy strictly better than full information, which requires the obedience constraint to hold for thresholds slightly below mfim_{fi}mfi​.

Formalization scope

Lean represents queue lengths as ℕ and distributions as functions ℕ → ℝ, inside the namespace NeedPersuasion.Homog. Every statement carries the standing assumptions M.Standing: λL>0\lambda_L > 0λL​>0, λH≥0\lambda_H \ge 0λH​≥0, λ≤1\lambda \le 1λ≤1, ui(0)>0u_i(0) > 0ui​(0)>0, ui(k)<0u_i(k) < 0ui​(k)<0 for some kkk, uiu_iui​ strictly decreasing, and uL(n)−uL(n+1)u_L(n) - u_L(n+1)uL​(n)−uL​(n+1) non-increasing. The homogeneous statements add λH=0\lambda_H = 0λH​=0. The type-H utility uHu_HuH​ stays a field of the model so that the definitions coincide with those of the sibling missions; with λH=0\lambda_H = 0λH​=0 it does not enter WLW_LWL​.

Conventions the formalization commits to:

  • Membership in ΠAP\Pi_{AP}ΠAP​ includes summability of πnuL(n)\pi_n u_L(n)πn​uL​(n) and πnuH(n)\pi_n u_H(n)πn​uH​(n). The paper treats welfare as finite, and Lean's infinite sum of a non-summable series is 000.
  • WLW_LWL​ is the series ∑n(πn+1−λHπn)uL(n)\sum_n (\pi_{n+1} - \lambda_H\pi_n) u_L(n)∑n​(πn+1​−λH​πn​)uL​(n), not the form of (2) that divides by λLπn\lambda_L\pi_nλL​πn​.
  • The threshold policy πx\pi_xπx​ is defined by its unnormalized weights and normalized by their sum. WL(fi)W_L(\mathrm{fi})WL​(fi) and WL(x)W_L(x)WL​(x) are computed from these distributions, never from a closed-form expression.
  • λL≤1\lambda_L \le 1λL​≤1 is allowed, as on the page; no λ<1\lambda < 1λ<1 is assumed.

The following trivializations are excluded: defining "sm" as fi\mathrm{fi}fi, defining it through sSup of an unbounded set or Classical.choose, computing WL(fi)W_L(\mathrm{fi})WL​(fi) from Lemma 5's closed form, or stating part 3 with both sides about the same threshold. A sorry-free sanity check exhibits a model satisfying all hypotheses (λL=1/2\lambda_L = 1/2λL​=1/2, uL(k)=1−3(k+1)/10u_L(k) = 1 - 3(k+1)/10uL​(k)=1−3(k+1)/10, mfi=3m_{fi} = 3mfi​=3, βfi=14/15\beta_{fi} = 14/15βfi​=14/15) with fi∈ΠSM\mathrm{fi} \in \Pi_{SM}fi∈ΠSM​.

A complete development needs:

  • finite geometric sums and ratios of affine functions, which are in Mathlib;
  • monotonicity of ratios of affine functions;
  • summation arguments for distributions on ℕ.

The perturbation arguments behind Theorem 1 and the lemmas about threshold policies are reusable in the sibling missions. Proofs of any milestone are welcome. So are intermediate lemmas, such as closed forms for WL(x)W_L(x)WL​(x) and L(x)L(x)L(x) of a threshold policy in the homogeneous case.

Selected references

  • J. Anunrojwong, K. Iyer, V. Manshadi, Information Design for Congested Social Services: Optimal Need-Based Persuasion, Management Science, 2023; read as arXiv:2005.07253v3 (2022). https://arxiv.org/abs/2005.07253
  • B. Lingenbrink, K. Iyer, Optimal Signaling Mechanisms in Unobservable Queues, Operations Research 67(5), 2019. https://doi.org/10.1287/opre.2019.1856
  • E. Kamenica, M. Gentzkow, Bayesian Persuasion, American Economic Review 101(6), 2011. https://doi.org/10.1257/aer.101.6.2590
  • P. Naor, The Regulation of Queue Size by Levying Tolls, Econometrica 37(1), 1969. https://doi.org/10.2307/1909200
10 thms1 active userReviewed
Algorithmic Game TheoryEconomicsOperations Research·Captain: mikedeng1

Information Design for Congested Social Services: Optimal Need-Based Persuasion 4: Signaling Achieves the First-Best Admission Policy for All Welfare Weights Above a ThresholdResearch Paper

Motivation

Social services such as public health clinics, legal aid offices and food banks are congested: users wait in a queue whose length they cannot see. Some users have no alternative (high need) and others have an outside option (low need). A provider cannot price access, but can decide what to tell arriving users about the queue. Anunrojwong, Iyer and Manshadi (Management Science, 2023; read as arXiv:2005.07253v3) study how far such information design can raise the welfare of both groups. Their model builds on the obedient signaling framework of Lingenbrink and Iyer (Operations Research 67(5), 2019). This mission formalizes their Section 5. It compares the best signaling mechanism with the first-best: the admission policy a provider would choose if it could see user types and order users to join or leave.

Setting

A single server with exponential service times of rate one serves an unobservable first-come-first-served queue. High-need users (type H) arrive at Poisson rate λH≥0\lambda_H\ge0λH​≥0 and always join. Low-need users (type L) arrive at rate λL>0\lambda_L>0λL​>0 and either join or take an outside option worth 000. Write λ=λL+λH≤1\lambda=\lambda_L+\lambda_H\le1λ=λL​+λH​≤1. A type-iii user who joins behind nnn users gets utility ui(n)u_i(n)ui​(n). The standing assumptions are ui(0)>0u_i(0)>0ui​(0)>0, lim⁡kui(k)<0\lim_k u_i(k)<0limk​ui​(k)<0, each uiu_iui​ strictly decreasing, and the increments uL(n)−uL(n+1)u_L(n)-u_L(n+1)uL​(n)−uL​(n+1) non-increasing (Assumption 1).

An admission policy is a distribution π\piπ on N0\mathbb N_0N0​ with

λHπn≤πn+1≤λπn(n≥0);\lambda_H\pi_n\le\pi_{n+1}\le\lambda\pi_n\qquad(n\ge0);λH​πn​≤πn+1​≤λπn​(n≥0);

their set is ΠAP\Pi_{AP}ΠAP​. It is the steady state of a queue in which a low-need user arriving at length nnn is admitted with probability (πn+1−λHπn)/(λLπn)(\pi_{n+1}-\lambda_H\pi_n)/(\lambda_L\pi_n)(πn+1​−λH​πn​)/(λL​πn​). A signaling mechanism is an admission policy that low-need users obey when it is announced as a join/leave recommendation. The obedience constraints are

J(π)=∑n(πn+1−λHπn)uL(n)≥0,L(π)=∑n(λπn−πn+1)uL(n)≤0,J(\pi)=\sum_n(\pi_{n+1}-\lambda_H\pi_n)u_L(n)\ge0,\qquad L(\pi)=\sum_n(\lambda\pi_n-\pi_{n+1})u_L(n)\le0,J(π)=n∑​(πn+1​−λH​πn​)uL​(n)≥0,L(π)=n∑​(λπn​−πn+1​)uL​(n)≤0,

and the signaling mechanisms form ΠSM⊆ΠAP\Pi_{SM}\subseteq\Pi_{AP}ΠSM​⊆ΠAP​. The welfares are WL(π)=J(π)W_L(\pi)=J(\pi)WL​(π)=J(π) and WH(π)=λH∑nπnuH(n)W_H(\pi)=\lambda_H\sum_n\pi_nu_H(n)WH​(π)=λH​∑n​πn​uH​(n). For a weight θ∈[0,1]\theta\in[0,1]θ∈[0,1], W(π,θ)=θWL(π)+(1−θ)WH(π)W(\pi,\theta)=\theta W_L(\pi)+(1-\theta)W_H(\pi)W(π,θ)=θWL​(π)+(1−θ)WH​(π). Then sm(θ)\mathrm{sm}(\theta)sm(θ) is the set of maximizers of W(⋅,θ)W(\cdot,\theta)W(⋅,θ) over ΠSM\Pi_{SM}ΠSM​, and ap(θ)\mathrm{ap}(\theta)ap(θ) the set of maximizers over ΠAP\Pi_{AP}ΠAP​. Under no information (ni) every low-need user joins with the same equilibrium probability pnip^{ni}pni. Finally,

g(x)=∑k≥0(1−x)xkuL(k)g(x)=\sum_{k\ge0}(1-x)x^ku_L(k)g(x)=k≥0∑​(1−x)xkuL​(k)

is a low-need user's expected utility in an M/M/1 queue with load xxx. It has a unique root ΛˉH∈(0,1)\bar\Lambda_H\in(0,1)ΛˉH​∈(0,1).

Formalization targets

Goal: Theorem 6 (Achieving first-best)

  1. If λH≥ΛˉH\lambda_H\ge\bar\Lambda_HλH​≥ΛˉH​, then sm(θ)=ap(θ)\mathrm{sm}(\theta)=\mathrm{ap}(\theta)sm(θ)=ap(θ) for every θ∈[0,1]\theta\in[0,1]θ∈[0,1].
  2. If λH<ΛˉH\lambda_H<\bar\Lambda_HλH​<ΛˉH​, there is θˉ=θ(λL,λH)∈(0,1]\bar\theta=\theta(\lambda_L,\lambda_H)\in(0,1]θˉ=θ(λL​,λH​)∈(0,1] with
sm(θ)=ap(θ)(θ>θˉ),sm(θ)=sm(θ′)(θ,θ′<θˉ).\mathrm{sm}(\theta)=\mathrm{ap}(\theta)\quad(\theta>\bar\theta),\qquad \mathrm{sm}(\theta)=\mathrm{sm}(\theta')\quad(\theta,\theta'<\bar\theta).sm(θ)=ap(θ)(θ>θˉ),sm(θ)=sm(θ′)(θ,θ′<θˉ).

The goal does not fix θˉ\bar\thetaθˉ or give it in closed form. It asserts only that θˉ\bar\thetaθˉ exists, that θˉ\bar\thetaθˉ is positive, and that the optimal signaling mechanism splits into these two regimes.

Milestones

In the order the proof in Appendix D uses them:

  • Lemma 3, the detailed-balance sandwich λHk−qπq≤πk≤λk−qπq\lambda_H^{k-q}\pi_q\le\pi_k\le\lambda^{k-q}\pi_qλHk−q​πq​≤πk​≤λk−qπq​.
  • Theorems 1 and 2: every Pareto-efficient signaling mechanism or admission policy has a threshold structure with threshold at most mfim_{fi}mfi​.
  • Lemma 2, existence of maximizers.
  • Lemma 4, the no-information equilibrium.
  • ggg strictly decreasing with a unique root.
  • Theorem 5 and Proposition 3(3): ni is Pareto-dominated by a signaling mechanism iff λH<ΛˉH\lambda_H<\bar\Lambda_HλH​<ΛˉH​.
  • Lemma 5(1): WHW_HWH​ of the threshold policy decreases in the threshold.
  • Theorem 3: when (LEAVE) is slack, efficiency in ΠSM\Pi_{SM}ΠSM​ and in ΠAP\Pi_{AP}ΠAP​ coincide.
  • Lemma 6, the shape of LLL along threshold policies.

Significance

Theorem 6 says that over a large range of weights the obedience constraints cost nothing. The optimal signaling mechanism coincides with the first-best admission policy, even though that benchmark presumes the provider sees user types and can enforce decisions. In that range the signal acts only as a coordination device, and no user is indifferent between the recommended action and the alternative. This differs from typical Bayesian persuasion, where optimal schemes make some receiver indifferent. The two regimes of the theorem coincide with those of Theorem 5. Where no information is already Pareto-efficient, efficient signaling and efficient admission agree for all weights. Elsewhere, signaling falls short of the first-best only for weights that favour high-need users.

The paper proves these results on paper. To our knowledge none of them has a machine-checked proof. The mission produces a formal model of obedient binary signaling in an unobservable queue, with admission policies, obedience constraints, welfare and Pareto efficiency within a class. It also formalizes the structural theorems and the first-best comparison. Formal proofs would also check the steps the paper calls "straightforward" or leaves out, such as Theorem 2's omitted proof and the coupling arguments.

Difficulty

Each step of the proof uses a different tool. The threshold monotonicity in θ\thetaθ is a revealed-preference argument, and it needs WHW_HWH​ of threshold policies to be strictly decreasing. Obedience propagates upward only along threshold policies, through the shape of LLL in Lemma 6. The positivity of θˉ\bar\thetaθˉ needs the optimal admission policy for small θ\thetaθ to violate (LEAVE). Below θˉ\bar\thetaθˉ, the binding (LEAVE) constraint must pin a single threshold. The tempting shortcut is to argue only with the scalarized problem max⁡W(⋅,θ)\max W(\cdot,\theta)maxW(⋅,θ), and it fails for two reasons. At θ∈{0,1}\theta\in\{0,1\}θ∈{0,1} maximizers need not be Pareto-efficient (footnote 12). Also, sm(θ)\mathrm{sm}(\theta)sm(θ) and ap(θ)\mathrm{ap}(\theta)ap(θ) are sets, so set equality needs both inclusions and non-emptiness. Existence (Lemma 2) rests on a weak-compactness argument over distributions on N0\mathbb N_0N0​, which is delicate at λ=1\lambda=1λ=1.

Formalization scope

  • A model is a structure with fields λL,λH∈R\lambda_L,\lambda_H\in\mathbb RλL​,λH​∈R and uL,uH:N→Ru_L,u_H:\mathbb N\to\mathbb RuL​,uH​:N→R. Queue lengths are natural numbers, and everything is real-valued.
  • Membership in ΠAP\Pi_{AP}ΠAP​ requires ∑nπnuL(n)\sum_n\pi_nu_L(n)∑n​πn​uL​(n) and ∑nπnuH(n)\sum_n\pi_nu_H(n)∑n​πn​uH​(n) to converge, since the paper treats welfare as finite.
  • WLW_LWL​ is written without the division by λLπn\lambda_L\pi_nλL​πn​.
  • Thresholds are reals x≥0x\ge0x≥0, and threshold ∞\infty∞ is a separate case.
  • sm(θ)\mathrm{sm}(\theta)sm(θ) and ap(θ)\mathrm{ap}(\theta)ap(θ) are sets of maximizers (footnote 14). At θ=0\theta=0θ=0 (θ=1\theta=1θ=1) they maximize WHW_HWH​ then WLW_LWL​ (WLW_LWL​ then WHW_HWH​), as in footnote 12.
  • The no-information equilibrium is defined by the low-need user's best-response condition, not by a closed form.
  • ΛˉH\bar\Lambda_HΛˉH​ enters as any root of ggg in (0,1)(0,1)(0,1), which is unique.

Hypotheses added relative to the page, all stated in the items:

  • Lemma 2's hypothesis (λ<1\lambda<1λ<1, or uL,uH→−∞u_L,u_H\to-\inftyuL​,uH​→−∞) and convergence of ∑kλHkuH(k)\sum_k\lambda_H^ku_H(k)∑k​λHk​uH​(k) on the goal, so the compared optimizer sets are non-empty.
  • Convergence of ∑kλHkuH(k)\sum_k\lambda_H^ku_H(k)∑k​λHk​uH​(k) on Lemma 2, Lemma 5(1), and the goal, and convergence of ∑krkuH(k)\sum_kr^ku_H(k)∑k​rkuH​(k) for every r∈[0,1)r\in[0,1)r∈[0,1) on the existence part of Lemma 4. Under the finite-welfare convention, ΠAP\Pi_{AP}ΠAP​ is empty without the first when λH>0\lambda_H>0λH​>0 and uHu_HuH​ falls fast enough.
  • λH>0\lambda_H>0λH​>0 in Lemma 5(1), which is false for λH=0\lambda_H=0λH​=0.

Encodings that would trivialize the goal are excluded:

  • sm(θ)\mathrm{sm}(\theta)sm(θ) and ap(θ)\mathrm{ap}(\theta)ap(θ) are full sets of maximizers, not single chosen maximizers. The goal's finiteness and existence hypotheses rule out empty optimizer sets.
  • Pareto efficiency of ni is tested against all of ΠAP\Pi_{AP}ΠAP​.
  • (LEAVE) is never read off a divergent series.

The work needs:

  • stochastic-dominance arguments for distributions on N0\mathbb N_0N0​;
  • monotonicity of ratios of affine functions;
  • a compactness argument for existence.

The model layer and the threshold structural theorems can be reused by the companion missions of this series. Proofs of individual milestones are welcome independently.

Selected references

  • J. Anunrojwong, K. Iyer, V. Manshadi, Information Design for Congested Social Services: Optimal Need-Based Persuasion, Management Science, 2023; preprint arXiv:2005.07253v3, 2022. https://arxiv.org/abs/2005.07253v3
  • D. Lingenbrink, K. Iyer, Optimal Signaling Mechanisms in Unobservable Queues, Operations Research 67(5):1397–1416, 2019.
  • Y.-K. Che, J. Kim, F. Kojima, C. T. Ryan, Characterizing Pareto Optima: Sequential Utilitarian Welfare Maximization, arXiv:2008.10819, 2020. https://arxiv.org/abs/2008.10819
14 thms1 active userReviewed
Algorithmic Game TheoryEconomicsOperations Research·Captain: mikedeng1

Information Design for Congested Social Services: Optimal Need-Based Persuasion 1: Every Pareto-Efficient Signaling Mechanism Has a Threshold Structure at or Below the Full-Information ThresholdResearch Paper

Why information design for congested public services

Many social services are rationed by waiting rather than by price: public health clinics, food banks, shelters, legal aid. Users who need the service most often have no outside option, while users with less need could be served elsewhere but join the queue anyway and lengthen everyone's wait. The provider usually cannot force anyone to leave, and charging money defeats the purpose of the service. What the provider does control is information: it observes the queue and decides what to tell arriving users about it.

Anunrojwong, Iyer and Manshadi (2023) model this as a Bayesian persuasion problem in a queue, building on the revenue-maximization model of Lingenbrink and Iyer (2019, Operations Research 67(5):1397–1416). Their first structural result, Theorem 1, says that whatever trade-off the provider makes between the two user groups, the efficient way to signal is a simple threshold rule, and that the threshold never exceeds the one under full information. This mission formalizes Theorem 1 and the steps of its proof.

Setting

A single server serves users first-come-first-served; service times are exponential with rate one. Users of two types arrive by independent Poisson processes: low-need users (type L\mathsf LL) at rate λL>0\lambda_{\mathsf L}>0λL​>0 and high-need users (type H\mathsf HH) at rate λH≥0\lambda_{\mathsf H}\ge0λH​≥0, with total rate λ=λL+λH≤1\lambda=\lambda_{\mathsf L}+\lambda_{\mathsf H}\le 1λ=λL​+λH​≤1. A type-iii user who joins with n∈N0n\in\mathbb N_0n∈N0​ users already present receives utility ui(n)u_i(n)ui​(n), where ui(0)>0u_i(0)>0ui​(0)>0, ui(n)<0u_i(n)<0ui​(n)<0 for large nnn, uiu_iui​ is strictly decreasing, and the increments uL(n)−uL(n+1)u_{\mathsf L}(n)-u_{\mathsf L}(n+1)uL​(n)−uL​(n+1) are non-increasing (Assumption 1). High-need users always join; low-need users take an outside option worth 000 unless joining is at least as good in expectation.

By the revelation principle and Lemma 1 of the paper, an obedient binary signaling mechanism is identified with its steady-state distribution π=(πn)n≥0\pi=(\pi_n)_{n\ge0}π=(πn​)n≥0​ of the queue length. The admission policies ΠAP\Pi_{\mathsf{AP}}ΠAP​ are the probability distributions on N0\mathbb N_0N0​ satisfying the detailed-balance constraints

λHπn≤πn+1≤λπn(n≥0).\lambda_{\mathsf H}\pi_n\le\pi_{n+1}\le\lambda\pi_n\qquad(n\ge0).λH​πn​≤πn+1​≤λπn​(n≥0).

The signaling mechanisms ΠSM⊆ΠAP\Pi_{\mathsf{SM}}\subseteq\Pi_{\mathsf{AP}}ΠSM​⊆ΠAP​ additionally satisfy the obedience constraints

J(π)=∑n≥0(πn+1−λHπn) uL(n)≥0,L(π)=∑n≥0(λπn−πn+1) uL(n)≤0.J(\pi)=\sum_{n\ge0}(\pi_{n+1}-\lambda_{\mathsf H}\pi_n)\,u_{\mathsf L}(n)\ge0,\qquad L(\pi)=\sum_{n\ge0}(\lambda\pi_n-\pi_{n+1})\,u_{\mathsf L}(n)\le0 .J(π)=n≥0∑​(πn+1​−λH​πn​)uL​(n)≥0,L(π)=n≥0∑​(λπn​−πn+1​)uL​(n)≤0.

The welfare of the two groups is WL(π)=J(π)W_{\mathsf L}(\pi)=J(\pi)WL​(π)=J(π) and WH(π)=λH∑nπnuH(n)W_{\mathsf H}(\pi)=\lambda_{\mathsf H}\sum_n\pi_nu_{\mathsf H}(n)WH​(π)=λH​∑n​πn​uH​(n). A mechanism π^\hat\piπ^ Pareto-dominates π\piπ if Wi(π^)≥Wi(π)W_i(\hat\pi)\ge W_i(\pi)Wi​(π^)≥Wi​(π) for both iii, with at least one strict inequality; π∈ΠSM\pi\in\Pi_{\mathsf{SM}}π∈ΠSM​ is Pareto-efficient within ΠSM\Pi_{\mathsf{SM}}ΠSM​ if no π^∈ΠSM\hat\pi\in\Pi_{\mathsf{SM}}π^∈ΠSM​ dominates it (Definition 1).

A distribution has a threshold structure (Definition 2) if for some m∈N0∪{∞}m\in\mathbb N_0\cup\{\infty\}m∈N0​∪{∞}, πk+1=λπk\pi_{k+1}=\lambda\pi_kπk+1​=λπk​ for k<mk<mk<m and πk+1=λHπk\pi_{k+1}=\lambda_{\mathsf H}\pi_kπk+1​=λH​πk​ for k>mk>mk>m: low-need users are told to join below mmm, to leave above mmm, and to join with probability a=(πm+1−λHπm)/(λLπm)∈[0,1]a=(\pi_{m+1}-\lambda_{\mathsf H}\pi_m)/(\lambda_{\mathsf L}\pi_m)\in[0,1]a=(πm+1​−λH​πm​)/(λL​πm​)∈[0,1] at mmm. The threshold is x=m+ax=m+ax=m+a. The full-information threshold mfim_{\mathsf{fi}}mfi​ is the smallest kkk with uL(k)<0u_{\mathsf L}(k)<0uL​(k)<0: when the queue is revealed, low-need users join exactly below it.

Formalization targets

Goal: Theorem 1 (p. 14)

For every π∈ΠSM\pi\in\Pi_{\mathsf{SM}}π∈ΠSM​ that is Pareto-efficient within ΠSM\Pi_{\mathsf{SM}}ΠSM​, there is a finite m∈N0m\in\mathbb N_0m∈N0​ such that

πk+1=λπk (k<m),πk+1=λHπk (k>m),m+πm+1−λHπmλLπm≤mfi.\pi_{k+1}=\lambda\pi_k\ (k<m),\qquad \pi_{k+1}=\lambda_{\mathsf H}\pi_k\ (k>m),\qquad m+\frac{\pi_{m+1}-\lambda_{\mathsf H}\pi_m}{\lambda_{\mathsf L}\pi_m}\le m_{\mathsf{fi}} .πk+1​=λπk​ (k<m),πk+1​=λH​πk​ (k>m),m+λL​πm​πm+1​−λH​πm​​≤mfi​.

No weight θ\thetaθ is fixed: the statement covers the whole Pareto frontier.

Milestones

  1. Lemma 3 (p. 36): for π∈ΠAP\pi\in\Pi_{\mathsf{AP}}π∈ΠAP​ and k>qk>qk>q, λHk−qπq≤πk≤λk−qπq\lambda_{\mathsf H}^{k-q}\pi_q\le\pi_k\le\lambda^{k-q}\pi_qλHk−q​πq​≤πk​≤λk−qπq​; if λH>0\lambda_{\mathsf H}>0λH​>0, every πk∈(0,1)\pi_k\in(0,1)πk​∈(0,1).
  2. Step 1 (App. A, p. 34): if π∈ΠSM\pi\in\Pi_{\mathsf{SM}}π∈ΠSM​ has πm+1<λπm\pi_{m+1}<\lambda\pi_mπm+1​<λπm​ and λHπm+1<πm+2\lambda_{\mathsf H}\pi_{m+1}<\pi_{m+2}λH​πm+1​<πm+2​ for some mmm, some π^∈ΠSM\hat\pi\in\Pi_{\mathsf{SM}}π^∈ΠSM​ Pareto-dominates π\piπ.
  3. Structure dichotomy (App. A, p. 35): if πm+1<λπm\pi_{m+1}<\lambda\pi_mπm+1​<λπm​ always forces πm+2=λHπm+1\pi_{m+2}=\lambda_{\mathsf H}\pi_{m+1}πm+2​=λH​πm+1​, then π\piπ has threshold ∞\infty∞ or a finite threshold structure.
  4. Step 2 (App. A, pp. 35–36): a π∈ΠSM\pi\in\Pi_{\mathsf{SM}}π∈ΠSM​ with a threshold structure whose threshold is ∞\infty∞ or exceeds mfim_{\mathsf{fi}}mfi​ is Pareto-dominated within ΠSM\Pi_{\mathsf{SM}}ΠSM​.

Significance

Theorem 1 reduces an infinite-dimensional design problem (a distribution on N0\mathbb N_0N0​ subject to obedience) to a one-parameter family of threshold policies x∈[0,mfi]x\in[0,m_{\mathsf{fi}}]x∈[0,mfi​]. Every later result of the paper is computed in that family: the bound of the optimal signaling mechanism against full information when all users are low-need (Proposition 1), the condition under which information design Pareto-dominates full information (Theorem 4 and Proposition 3), and the conditions under which signaling achieves the first-best admission policy (Theorems 5 and 6). The bound x≤mfix\le m_{\mathsf{fi}}x≤mfi​ also has a direct reading: efficient persuasion only ever discourages low-need users relative to full information, never encourages them.

The theorem is proved in the paper (App. A). This mission produces a machine-checked version of the proof, on top of a reusable Lean model of obedient signaling in an M/M/1M/M/1M/M/1-type queue with two user types. To our knowledge no part of this model or of its results has been formalized before.

Difficulty

The statement quantifies over the full Pareto frontier, so it cannot be obtained from first-order conditions of a single weighted objective. Both steps are perturbation arguments on infinite sequences: each must build an explicit competitor, keep it inside the detailed-balance constraints and both obedience constraints at once, and show weak improvement for one group with strict improvement for the other. The obedience constraint (LEAVE) cuts against (JOIN): moving mass toward shorter queues raises JJJ but can raise LLL, and the bound on LLL is where λ≤1\lambda\le1λ≤1 is needed. Since λ=1\lambda=1λ=1 is allowed, the geometric tails are not automatically summable and the infinite sums have to be handled with care. The threshold ∞\infty∞ must be excluded explicitly; it is not ruled out by the structure argument alone.

Formalization scope

All definitions live in NeedPersuasion.Threshold (file Def_NeedPersuasion_Threshold_Model). Queue lengths are ℕ, everything else is ℝ; the two types are two fields of a Model structure. The standing assumptions are the predicate Model.Standing: λL>0\lambda_{\mathsf L}>0λL​>0, λH≥0\lambda_{\mathsf H}\ge0λH​≥0, λ≤1\lambda\le1λ≤1, ui(0)>0u_i(0)>0ui​(0)>0, ui(k)<0u_i(k)<0ui​(k)<0 for some kkk (equivalent to lim⁡ui<0\lim u_i<0limui​<0 for a strictly decreasing sequence, and allowing −∞-\infty−∞), both utilities strictly decreasing, and the L\mathsf LL-increments non-increasing. Every theorem assumes it; no other hypothesis is added, and in particular λ<1\lambda<1λ<1 is not assumed.

Committed conventions:

  • ΠAP\Pi_{\mathsf{AP}}ΠAP​ requires ∑nπn=1\sum_n\pi_n=1∑n​πn​=1 as a HasSum, and finite welfare: (πnuL(n))(\pi_nu_{\mathsf L}(n))(πn​uL​(n)) and (πnuH(n))(\pi_nu_{\mathsf H}(n))(πn​uH​(n)) summable. The paper treats the welfare sums as real numbers; without this, a policy with welfare −∞-\infty−∞ (possible at λ=1\lambda=1λ=1) would receive welfare 000 from Lean's tsum.
  • WLW_{\mathsf L}WL​ is the right-hand form of (2), which has no division by λLπn\lambda_{\mathsf L}\pi_nλL​πn​.
  • The threshold ∞\infty∞ is the first disjunct of HasThresholdStructure; HasThresholdAtMost π y asks for a finite mmm with m+a≤ym+a\le ym+a≤y.
  • Pareto efficiency is Definition 1 (ParetoEfficientIn), never maximization of θWL+(1−θ)WH\theta W_{\mathsf L}+(1-\theta)W_{\mathsf H}θWL​+(1−θ)WH​.

Trivializing encodings are ruled out: the threshold-structure predicate is not satisfied by every admission policy, efficiency is taken over all of ΠSM\Pi_{\mathsf{SM}}ΠSM​ (including the competitors the proof builds), and no sum whose value is compared can be a divergent tsum. A sorry-free sanity file exhibits a model satisfying Standing whose full-information mechanism lies in ΠSM\Pi_{\mathsf{SM}}ΠSM​, so the hypotheses are not vacuous.

A complete development needs manipulation of tsum over shifted and truncated sequences, geometric tails, and first-order stochastic dominance for sequences on N0\mathbb N_0N0​; these pieces are reusable for the other missions of this series (missions 2–4 restate the same model). Contributions of proofs of individual milestones, or of general lemmas on detailed-balance sequences, are welcome.

Selected references

  • J. Anunrojwong, K. Iyer, V. Manshadi, Information Design for Congested Social Services: Optimal Need-Based Persuasion, Management Science, 2023; read as arXiv:2005.07253v3 (2022). https://arxiv.org/abs/2005.07253
  • D. Lingenbrink, K. Iyer, Optimal Signaling Mechanisms in Unobservable Queues, Operations Research 67(5):1397–1416, 2019 (cited as reference in arXiv:2005.07253v3, p. 32).
  • E. Kamenica, M. Gentzkow, Bayesian Persuasion, American Economic Review 101(6), 2011. https://doi.org/10.1257/aer.101.6.2590
  • P. Naor, The Regulation of Queue Size by Levying Tolls, Econometrica 37(1), 1969. https://doi.org/10.2307/1909200
6 thms1 active userReviewed
Machine LearningOperations ResearchStatistics·Captain: mikedeng1

Subset Selection with Shrinkage: Sparse Linear Modeling When the SNR Is Low V: When the SNR Is Low, Best Subset Selection Has Prediction Error at Least of Order σ²γ²k log(ep/k)Research Paper

Why a lower bound for best subsets

Best subset selection fits a linear model by least squares using at most kkk of ppp available predictors. It is the classical ℓ0\ell_0ℓ0​-constrained estimator of sparse regression. Mixed-integer optimization now computes it for problems with thousands of variables (Bertsimas, King & Mazumder 2016), so its statistical behaviour matters in practice. Simulations in Hastie, Tibshirani & Tibshirani 2020 found that it predicts worse than shrinkage methods such as the Lasso when the signal-to-noise ratio (SNR) is low.

Mazumder, Radchenko and Dedieu (arXiv:1708.03288v4; Operations Research, 2023) add an ℓ1\ell_1ℓ1​ or ℓ2\ell_2ℓ2​ shrinkage penalty to best subsets. They prove upper bounds of two kinds: slow rates, linear in the tuning parameter λ\lambdaλ, and fast rates, quadratic in λ\lambdaλ. Theorems 2 and 3 of the paper give them for the shrunken estimators. For best subsets, the paper only has a fast-rate bound, of order σ2klog⁡(ep/k)\sigma^2 k\log(ep/k)σ2klog(ep/k). When ∥β∗∥/σ\|\beta^*\|/\sigma∥β∗∥/σ is small, the shrunken estimators' slow rate beats that bound. Theorem 4, the goal of this mission, shows that the comparison is meaningful: in that regime best subsets really does incur an error of the order of its upper bound.

Setting

The data follow the linear model

y=Xβ∗+ϵ,y,ϵ∈Rn, β∗∈Rp,y = X\beta^* + \epsilon ,\qquad y,\epsilon\in\mathbb R^n,\ \beta^*\in\mathbb R^p ,y=Xβ∗+ϵ,y,ϵ∈Rn, β∗∈Rp,

where the design XXX is a deterministic real n×pn\times pn×p matrix whose columns have unit Euclidean norm. The noise ϵ\epsilonϵ has independent N(0,σ2)N(0,\sigma^2)N(0,σ2) entries, with σ>0\sigma>0σ>0. Write ∥v∥\|v\|∥v∥ for the Euclidean norm, ∥v∥1=∑i∣vi∣\|v\|_1=\sum_i|v_i|∥v∥1​=∑i​∣vi​∣, supp⁡(v)={i:vi≠0}\operatorname{supp}(v)=\{i:v_i\ne0\}supp(v)={i:vi​=0} and ∥v∥0=∣supp⁡(v)∣\|v\|_0=|\operatorname{supp}(v)|∥v∥0​=∣supp(v)∣. The ℓ0\ell_0ℓ0​ ball is B0(s)={u∈Rp:∥u∥0≤s}B_0(s)=\{u\in\mathbb R^p:\|u\|_0\le s\}B0​(s)={u∈Rp:∥u∥0​≤s}. The minimal sss-sparse eigenvalue of XXX is

γs=min⁡u≠0, u∈B0(s)∥Xu∥∥u∥.\gamma_s=\min_{u\ne0,\ u\in B_0(s)}\frac{\|Xu\|}{\|u\|} .γs​=u=0, u∈B0​(s)min​∥u∥∥Xu∥​.

A best-subsets solution of size kkk is any minimizer

β^ℓ0∈arg⁡min⁡ ∥y−Xβ∥2s.t.∥β∥0≤k.(1)\hat\beta_{\ell_0}\in\arg\min\ \|y-X\beta\|^2\quad\text{s.t.}\quad\|\beta\|_0\le k .\tag{1}β^​ℓ0​​∈argmin ∥y−Xβ∥2s.t.∥β∥0​≤k.(1)

A constant is universal if it does not depend on n,p,k,X,σ,β∗n,p,k,X,\sigma,\beta^*n,p,k,X,σ,β∗. The symbols ≲\lesssim≲ and ≳\gtrsim≳ mean ≤\le≤ and ≥\ge≥ up to positive universal factors. Logarithms are natural, and e=exp⁡(1)e=\exp(1)e=exp(1).

Formalization targets

Goal: Theorem 4 (p. 18)

There are universal constants c0,c,C>0c_0,c,C>0c0​,c,C>0 such that the following holds. Suppose 2≤k≤p2\le k\le p2≤k≤p, ∥β∗∥0≤k\|\beta^*\|_0\le k∥β∗∥0​≤k and ∥β∗∥≤c0σγkklog⁡(ep/k)\|\beta^*\|\le c_0\sigma\gamma_k\sqrt{k\log(ep/k)}∥β∗∥≤c0​σγk​klog(ep/k)​. Then, with probability at least 1−2(ep/k)−cγk2k−(ep/k)−k1-2(ep/k)^{-c\gamma_k^2k}-(ep/k)^{-k}1−2(ep/k)−cγk2​k−(ep/k)−k, every best-subsets solution satisfies

∥Xβ^ℓ0−Xβ∗∥2 ≥ C σ2γk2 klog⁡(ep/k).\|X\hat\beta_{\ell_0}-X\beta^*\|^2\ \ge\ C\,\sigma^2\gamma_k^2\,k\log(ep/k).∥Xβ^​ℓ0​​−Xβ∗∥2 ≥ Cσ2γk2​klog(ep/k).

The constants are left unfixed, as in the paper. The statement asserts the order of the error, which is what the comparison with the upper bounds needs.

Milestones

  1. Maurey approximation (p. 44). For unit columns and every β∗\beta^*β∗, some β~∗\tilde\beta^*β~​∗ with ∥β~∗∥0≤k/2\|\tilde\beta^*\|_0\le k/2∥β~​∗∥0​≤k/2 has ∥Xβ∗−Xβ~∗∥2≤2∥β∗∥12/k\|X\beta^*-X\tilde\beta^*\|^2\le2\|\beta^*\|_1^2/k∥Xβ∗−Xβ~​∗∥2≤2∥β∗∥12​/k.
  2. (39) (p. 44). With Δ∗=Xβ∗−Xβ~∗\Delta^*=X\beta^*-X\tilde\beta^*Δ∗=Xβ∗−Xβ~​∗, define G(β)=∥y−Xβ∥2−∥y−Xβ~∗∥2G(\beta)=\|y-X\beta\|^2-\|y-X\tilde\beta^*\|^2G(β)=∥y−Xβ∥2−∥y−Xβ~​∗∥2 and H(u)=∥Xu∥2−2(Δ∗+ϵ)⊤XuH(u)=\|Xu\|^2-2(\Delta^*+\epsilon)^\top XuH(u)=∥Xu∥2−2(Δ∗+ϵ)⊤Xu. Then G(β)G(\beta)G(β) equals ∥Xβ−Xβ~∗∥2+2(Δ∗+ϵ)⊤(Xβ~∗−Xβ)\|X\beta-X\tilde\beta^*\|^2+2(\Delta^*+\epsilon)^\top(X\tilde\beta^*-X\beta)∥Xβ−Xβ~​∗∥2+2(Δ∗+ϵ)⊤(Xβ~​∗−Xβ), and min⁡∥β∥0≤kG≤min⁡H(u)\min_{\|\beta\|_0\le k}G\le\min H(u)min∥β∥0​≤k​G≤minH(u) over uuu supported off supp⁡(β~∗)\operatorname{supp}(\tilde\beta^*)supp(β~​∗) with ∥u∥0≤k/2\|u\|_0\le k/2∥u∥0​≤k/2.
  3. (40) (p. 45). min⁡∥β∥0≤kG(β)≤−max⁡v∈Vk∣(Δ∗+ϵ)⊤Xv∣2\min_{\|\beta\|_0\le k}G(\beta)\le-\max_{v\in\mathcal V_k}|(\Delta^*+\epsilon)^\top Xv|^2min∥β∥0​≤k​G(β)≤−maxv∈Vk​​∣(Δ∗+ϵ)⊤Xv∣2, where Vk\mathcal V_kVk​ is the set of vvv supported off supp⁡(β~∗)\operatorname{supp}(\tilde\beta^*)supp(β~​∗) with ∥v∥0≤k/2\|v\|_0\le k/2∥v∥0​≤k/2 and ∥Xv∥=1\|Xv\|=1∥Xv∥=1.
  4. Lemma 1 (p. 44). Let SSS have qqq elements and let s∈[1,q]s\in[1,q]s∈[1,q] be an integer. Then max⁡∣ϵ⊤Xv∣≳σγ2sslog⁡(eq/s)\max|\epsilon^\top Xv|\gtrsim\sigma\gamma_{2s}\sqrt{s\log(eq/s)}max∣ϵ⊤Xv∣≳σγ2s​slog(eq/s)​ over supp⁡(v)⊆S\operatorname{supp}(v)\subseteq Ssupp(v)⊆S, ∥v∥0≤s\|v\|_0\le s∥v∥0​≤s, ∥Xv∥=1\|Xv\|=1∥Xv∥=1, with probability at least 1−2(eq/s)−c~γ2s2s1-2(eq/s)^{-\tilde c\gamma_{2s}^2s}1−2(eq/s)−c~γ2s2​s.

Significance

Theorem 4 matches the fast-rate upper bound for best subsets from Corollary 1 of the paper. When γk\gamma_kγk​ is bounded below, the prediction error of best subsets is of exact order σ2klog⁡(ep/k)\sigma^2k\log(ep/k)σ2klog(ep/k) in the low-SNR regime (Remark 10). This is the theoretical counterpart of the empirical observation that best subsets overfits at low SNR. It is also what makes the paper's case for shrinkage a theorem rather than a comparison of upper bounds. The paper's Proposition 4 applies it to a concrete correlated design, where it gives a ratio of prediction errors between best subsets and its ℓ2\ell_2ℓ2​-shrunken version that grows with kkk.

The result is proved in the paper; this mission formalizes it. No formalization of this result exists on the platform. The mission also produces a reusable Gaussian lower bound on sparse correlations (Lemma 1) and a deterministic reduction of the best-subsets objective, (39)–(40), which are of independent use for ℓ0\ell_0ℓ0​ estimators.

Difficulty

An upper bound on best subsets follows from comparing β^ℓ0\hat\beta_{\ell_0}β^​ℓ0​​ with β∗\beta^*β∗ in a basic inequality. A lower bound must hold for every minimizer, and β∗\beta^*β∗ itself gives no lower bound. The first idea is to compare with the noiseless fit. It fails because the noise can be fitted by sparse directions, which is exactly the effect to be quantified. The difficult step is Lemma 1. It requires a large set of sparse directions that remain separated after multiplication by XXX, a lower bound on the expected maximum of a Gaussian process over that set, and concentration of that maximum around its mean, with all constants universal. A second subtlety is that the statement is about every minimizer of a nonconvex problem, so the proof has to bound the objective rather than the estimator.

Formalization scope

Vectors are Fin n → ℝ and Fin p → ℝ, with indices starting at 0. The design is X : Matrix (Fin n) (Fin p) ℝ. The Euclidean norm, ℓ1\ell_1ℓ1​ norm, support and ∥⋅∥0\|\cdot\|_0∥⋅∥0​ are explicit definitions, since Mathlib's norm on Fin m → ℝ is the sup norm. γs\gamma_sγs​ is an infimum over the nonzero vectors with ∥u∥0≤s\|u\|_0\le s∥u∥0​≤s; this set is nonempty when p,s≥1p,s\ge1p,s≥1, and every ratio is ≥0\ge0≥0. The noise law is the product of nnn centred Gaussians of variance σ2\sigma^2σ2 on Rn\mathbb R^nRn. "With probability at least 1−η1-\eta1−η" means that a measurable set of noise vectors of probability at least 1−η1-\eta1−η exists on which the conclusion holds. Best subsets is an argmin predicate: Theorem 4 quantifies over every minimizer, and existence is neither assumed nor needed. Universal constants are quantified before every other parameter. "∥⋅∥0≤k/2\|\cdot\|_0\le k/2∥⋅∥0​≤k/2" is a comparison of real numbers.

The model consists of XXX, β∗\beta^*β∗ (with f∗=Xβ∗f^*=X\beta^*f∗=Xβ∗), ϵ\epsilonϵ and its Gaussian law, the estimator (1), and the minimal sparse eigenvalue γs\gamma_sγs​.

Two hypotheses are pinned relative to the printed theorem:

  • k≥2k\ge2k≥2, because the statement is false for k=1k=1k=1. Take every column equal to e1e_1e1​ and β∗=0\beta^*=0β∗=0: the error is ϵ12\epsilon_1^2ϵ12​, while the bound demands order σ2log⁡(ep)\sigma^2\log(ep)σ2log(ep) with probability tending to one.
  • ∥β∗∥0≤k\|\beta^*\|_0\le k∥β∗∥0​≤k, because the proof uses ∥β∗∥1\|\beta^*\|_1∥β∗∥1​, which the printed ℓ2\ell_2ℓ2​ hypothesis controls only for sparse β∗\beta^*β∗, and §3.4 places the theorem in Corollary 1's setting β∗∈B0(k)\beta^*\in B_0(k)β∗∈B0​(k).

The milestones carry neither pin. Because the constants are existential, a formalization in which c0c_0c0​, ccc or CCC may depend on ppp, kkk or XXX would be trivial, and is ruled out by quantifying them first. A version asserting the bound for some minimizer would be weaker than the theorem.

Not formalized here: the upper bounds (Theorems 2, 3, 5), the noise events of Theorem 1, Corollaries 1–6, Propositions 3–4, and the numerical experiments. The proof of the goal also uses Theorem 1's event G2k\mathcal G_{2k}G2k​, which belongs to a separate mission of this series. Proved platform results that a proof of Lemma 1 can use include Sudakov minoration for separated Gaussian families (HighDimProb.RandomProcesses.sudakov_minoration_finite_separated), Gaussian Lipschitz concentration (HighDimStat.TailBounds.lipschitz_gaussian_concentration) and a lower bound on Gaussian maxima (HighDimProb.RandomProcesses.standard_gaussian_max_lower_bound). Contributions are welcome on any milestone, and on supporting lemmas: a packing of sparse sign vectors (Gilbert–Varshamov type), and the approximation of ℓ1\ell_1ℓ1​-bounded vectors by sparse ones.

Selected references

  • R. Mazumder, P. Radchenko, A. Dedieu, Subset Selection with Shrinkage: Sparse Linear Modeling When the SNR Is Low, Operations Research, 2023; preprint arXiv:1708.03288v4, 2022. https://arxiv.org/abs/1708.03288v4
  • D. Bertsimas, A. King, R. Mazumder, Best subset selection via a modern optimization lens, Annals of Statistics 44(2), 2016. https://doi.org/10.1214/15-AOS1388
  • T. Hastie, R. Tibshirani, R. Tibshirani, Best Subset, Forward Stepwise or Lasso? Analysis and Recommendations Based on Extensive Comparisons, Statistical Science 35(4), 2020. https://doi.org/10.1214/19-STS733
  • P. C. Bellec, G. Lecué, A. B. Tsybakov, Slope meets Lasso: improved oracle bounds and optimality, Annals of Statistics 46(6B), 2018. https://doi.org/10.1214/17-AOS1670
  • R. Vershynin, High-Dimensional Probability, Cambridge University Press, 2018. https://doi.org/10.1017/9781108231596
8 thms1 active userReviewed
Algorithmic Game TheoryMechanism DesignOperations Research·Captain: mikedeng1

Contextual Standard Auctions with Budgets: Revenue Equivalence and Efficiency Guarantees 3: Pacing-Based Equilibria Have Liquid Welfare Price of Anarchy at Least 1/2Research Paper

Motivation

Advertising platforms repeatedly sell opportunities to show an advertisement. The opportunity's features, such as audience and placement, affect each advertiser differently, while advertisers limit their average spending. This makes the efficiency of an auction a question about both heterogeneous values and budgets. The paper by Balseiro, Kroer, and Kumar studies this setting with a continuum of buyer types and context-dependent values. It constructs equilibria by value pacing, in which a buyer scales values before bidding. This mission asks how much attainable value those equilibria preserve when budgets cap useful value.

Earlier efficiency bounds for second-price auctions concern a different, non-Bayesian multi-item model. In particular, the half guarantee discussed by Azar and coauthors, as discussed in §5 of the paper uses a no-overbudgeting condition that the present pacing equilibria need not satisfy. Balseiro, Kroer, and Kumar prove their bound under their own contextual and in-expectation budget model; the distinction matters when interpreting the constant. Their existence and revenue-equivalence results also make it possible to speak about the same pacing-based allocation across standard auction payment rules. These claims and the limitation of the earlier assumption are explained in §5 of the paper.

Setting

An item has a context α∈A⊂Rd\alpha\in A\subset\mathbb R^dα∈A⊂Rd, with d≥2d\ge2d≥2. There are n≥2n\ge2n≥2 buyers. Each buyer has a private type θ=(w,B)\theta=(w,B)θ=(w,B): the vector w∈(0,U)dw\in(0,U)^dw∈(0,U)d assigns a positive weight to every context feature, and B∈(Bmin⁡,U)B\in(B_{\min},U)B∈(Bmin​,U) is the buyer's budget, where 0<Bmin⁡<U0<B_{\min}<U0<Bmin​<U. The buyer's value for the item is w⊤αw^\top\alphaw⊤α. A context is drawn from a distribution FFF and buyers' types are drawn independently from a distribution GGG; both distributions have densities. The context and type ranges are the spaces AAA and Θ=(0,U)d×(Bmin⁡,U)\Theta=(0,U)^d\times(B_{\min},U)Θ=(0,U)d×(Bmin​,U) of §2. The reserve price is zero throughout the efficiency section.

A pacing function μ:Θ→R≥0\mu:\Theta\to\mathbb R_{\ge0}μ:Θ→R≥0​ scales type θ\thetaθ's value to w⊤α/(1+μ(θ))w^\top\alpha/(1+\mu(\theta))w⊤α/(1+μ(θ)). Let Hαμ(x)H^\mu_\alpha(x)Hαμ​(x) be the probability that the maximum paced value among n−1n-1n−1 independent competitors is at most xxx. The first-price bid βμ(θ,α)\beta^\mu(\theta,\alpha)βμ(θ,α) applies the paper's bid function σαμ\sigma^\mu_\alphaσαμ​ to the paced value. A buyer may choose a context-dependent bid, subject to an in-expectation budget: expected payment over contexts and competitors must not exceed BBB. The buyer's dual function qμ(θ,t)q^\mu(\theta,t)qμ(θ,t) is the supremum of its Lagrangian over such bids at multiplier t≥0t\ge0t≥0.

The relevant pacing equilibria are fixed points of the correspondence C0∗C_0^*C0∗​. Its domain X0X_0X0​ consists of measurable, integrable pacing functions in [0,ω/Bmin⁡][0,\omega/B_{\min}][0,ω/Bmin​] with total variation at most V0=(d+1)Ud+1ω/Bmin⁡V_0=(d+1)U^{d+1}\omega/B_{\min}V0​=(d+1)Ud+1ω/Bmin​, where ω\omegaω is the supremum of possible values. For a given μ∈X0\mu\in X_0μ∈X0​, C0∗(μ)C_0^*(\mu)C0∗​(μ) minimizes the average dual objective Eθ∼G[qμ(θ,μ^(θ))]\mathbb E_{\theta\sim G}[q^\mu(\theta,\widehat\mu(\theta))]Eθ∼G​[qμ(θ,μ​(θ))] over μ^∈X0\widehat\mu\in X_0μ​∈X0​. These are the objects of §3.3, including its multivariate definition of total variation.

An allocation xxx assigns nonnegative shares xi(α,θ1,…,θn)x_i(\alpha,\theta_1,\ldots,\theta_n)xi​(α,θ1​,…,θn​) summing to one. For buyer iii, keep θi=(wi,Bi)\theta_i=(w_i,B_i)θi​=(wi​,Bi​) fixed while averaging wi⊤α xiw_i^\top\alpha\,x_iwi⊤​αxi​ over the context and the other types. The buyer's liquid welfare is the smaller of this interim value and BiB_iBi​; total liquid welfare LW(x)\mathrm{LW}(x)LW(x) sums its expectation over each buyer's own type. For an equilibrium pacing function μ∈C0∗(μ)\mu\in C_0^*(\mu)μ∈C0∗​(μ), xμx^\muxμ allocates to a highest paced bidder. Definitions 5 and 6 use this allocation to define the worst equilibrium liquid welfare relative to the optimal allocation.

Formalization targets

The main target is the Theorem 3 guarantee, stated for every measurable allocation xxx and every equilibrium pacing function μ\muμ:

LW(xμ)≥12 LW(x).\mathrm{LW}(x^\mu)\ge\tfrac12\,\mathrm{LW}(x).LW(xμ)≥21​LW(x).

The universal comparison expresses the paper's price-of-anarchy bound PoA≥1/2\mathrm{PoA}\ge1/2PoA≥1/2. It leaves no optimum or worst equilibrium as an arbitrary variable. The two milestones are the dual derivative identity and the first-order budget and complementary-slackness conditions stated in §3.2:

∂tqμ(θ,t)=B−payment(bt),payment(bt∗)≤B,t∗(B−payment(bt∗))=0.\partial_t q^\mu(\theta,t)=B-\mathrm{payment}(b_t),\qquad \mathrm{payment}(b_{t^*})\le B,\quad t^*(B-\mathrm{payment}(b_{t^*}))=0.∂t​qμ(θ,t)=B−payment(bt​),payment(bt∗​)≤B,t∗(B−payment(bt∗​))=0.

Here bt(α)=σαμ(w⊤α/(1+t))b_t(\alpha)=\sigma^\mu_\alpha(w^\top\alpha/(1+t))bt​(α)=σαμ​(w⊤α/(1+t)) and t∗t^*t∗ minimizes qμ(θ,⋅)q^\mu(\theta,\cdot)qμ(θ,⋅) over nonnegative multipliers. The milestones are posed in the zero-reserve setting used by the main target.

Significance

The theorem gives an explicit welfare guarantee despite private budgets, correlated values across buyers through a shared context, and equilibrium bidding that depends on those budgets. It applies to the allocation induced by pacing in every standard auction covered by the paper, since those auctions have the same equilibrium allocation even when payment rules differ Balseiro, Kroer, and Kumar, §4–5. Without an efficiency bound, existence of a pacing equilibrium alone would give no quantitative assurance about the allocation it produces.

The result has a mathematical proof in the paper's online appendix; it is an open formalization target here. A complete development would connect measurable Bayesian auctions, the buyer's nonconvex dual problem, the bounded-variation fixed-point domain, and interim liquid welfare. The definitions of interim welfare, dual expenditure, and multivariate variation can support other budgeted auction results. The source PDF held for this mission contains the published main article but not its online appendix, where the proofs and auxiliary lemmas appear.

Difficulty

The expected budget applies to each buyer type after averaging over contexts and opponents, while liquid welfare caps that same type's interim value before averaging over types. Moving the budget cap across the expectation changes the quantity. The natural first-price tie convention also awards the good to every highest bidder for the buyer optimization problem, while a welfare allocation must give total share one. The proof must reconcile the dual account of expenditure with a feasible allocation and preserve these different orders of expectation. The dual objective is a supremum over bids, so its derivative cannot be obtained merely by differentiating a chosen closed formula without justifying that formula and the maximizing bid.

Formalization scope

Lean represents a context by Fin d → ℝ and a buyer type by its weight vector and real budget. Value is the explicit finite sum ∑iwiαi\sum_i w_i\alpha_i∑i​wi​αi​. The model requires a nonempty bounded measurable context set in the positive orthant, absolutely continuous probability laws FFF and GGG supported on their stated domains, measurable reserves and pacing, n≥2n\ge2n≥2, and 0<Bmin⁡<U0<B_{\min}<U0<Bmin​<U. Boundedness of AAA makes ω\omegaω finite; these conditions make the domain and integrals implicit in the article well posed. The welfare target assumes zero reserves as §5 does.

Admissible bids are measurable and nonnegative on AAA, with integrable expected payment. The dual is defined by its supremum over bids. Total variation is extended-real-valued and uses Euclidean lengths of test fields; thus infinite variation cannot satisfy the X0X_0X0​ bound through a real supremum's default value. Allocations are measurable and simplex-valued on the model domain. At a tied highest bid, xμx^\muxμ splits the item equally; ties have probability zero under these equilibria, as the paper states by its appendix Lemma 6(d). The closed simplex permits zero shares, correcting the paper's notation R+\mathbb R_+R+​ in Definition 5, which §2.1 otherwise calls strictly positive. The universal welfare inequality avoids default real infimum and supremum values on pathological sets. Contributions to the measurable bidding, variation, duality, and welfare infrastructure are all useful within this scope.

Selected references

  • Santiago R. Balseiro, Christian Kroer, and Rachitesh Kumar, Contextual Standard Auctions with Budgets: Revenue Equivalence and Efficiency Guarantees, Management Science, Articles in Advance, 2023. DOI: 10.1287/mnsc.2023.4719.
6 thms1 active userReviewed
Algorithmic Game TheoryFunctional AnalysisMechanism Design+1·Captain: mikedeng1

Contextual Standard Auctions with Budgets: Revenue Equivalence and Efficiency Guarantees 1: A Value Pacing-Based Symmetric First Price Equilibrium ExistsResearch Paper

Motivation

Most online advertising is sold by auction, one impression at a time, to advertisers who manage a budget over many auctions rather than a value for a single one. Since 2019 the major ad exchanges have moved from second price to first price auctions, in which the winner pays its own bid. A budget-constrained advertiser in a first price auction must therefore shade its bids twice: once because of the budget, and once because of the payment rule. Whether these two effects admit a coherent equilibrium, and of what form, is the question behind Balseiro, Kroer and Kumar, Contextual Standard Auctions with Budgets: Revenue Equivalence and Efficiency Guarantees (Management Science, 2023, doi:10.1287/mnsc.2023.4719).

Earlier work on pacing equilibria treated second price auctions, where truthful bidding is dominant without budgets (Balseiro, Besbes and Weintraub 2015; Conitzer et al. 2022). In first price auctions with a single deterministic item, an equilibrium may fail to exist (the paper's Online Appendix A). The paper restores existence by letting items carry a random context and values depend on it linearly. This mission formalizes its first main result, Theorem 1: an equilibrium of a simple, interpretable form exists.

Setting

An item has a context α∈A⊂R+d\alpha\in A\subset\mathbb R^d_{+}α∈A⊂R+d​, a feature vector drawn from a law FFF with a density. There are nnn buyers. Each buyer has a type θ=(w,B)\theta=(w,B)θ=(w,B), a weight vector www and a budget BBB, drawn independently from a law GGG with a density on

Θ=(0,U)d×(Bmin⁡,U).\Theta=(0,U)^d\times(B_{\min},U).Θ=(0,U)d×(Bmin​,U).

Buyer (w,B)(w,B)(w,B) values item α\alphaα at wTαw^T\alphawTα, and ω:=sup⁡(w,B)∈Θ,α∈AwTα\omega:=\sup_{(w,B)\in\Theta,\alpha\in A}w^T\alphaω:=sup(w,B)∈Θ,α∈A​wTα. The seller reveals α\alphaα, then runs a first price auction with a public reserve price r(α)r(\alpha)r(α): a bid b(α)b(\alpha)b(α) wins if b(α)≥max⁡(r(α),{β(θi,α)}i)b(\alpha)\ge\max(r(\alpha),\{\beta(\theta_i,\alpha)\}_i)b(α)≥max(r(α),{β(θi​,α)}i​), where β\betaβ is the strategy of the n−1n-1n−1 competitors, and the winner pays its bid. A budget only needs to hold in expectation over α\alphaα and the competitors' types.

A strategy β∗:Θ×A→R≥0\beta^*:\Theta\times A\to\mathbb R_{\ge0}β∗:Θ×A→R≥0​ is a symmetric first price equilibrium (SFPE, Definition 1) if for GGG-almost every type (w,B)(w,B)(w,B) the bid function β∗(w,B,⋅)\beta^*(w,B,\cdot)β∗(w,B,⋅) solves

max⁡b:A→R≥0 E[(wTα−b(α))1{b(α)≥max⁡(r(α),{β∗(θi,α)}i)}]  s.t.  E[b(α)1{⋯ }]≤B.\max_{b:A\to\mathbb R_{\ge0}}\ \mathbb E\big[(w^T\alpha-b(\alpha))\mathbb 1\{b(\alpha)\ge\max(r(\alpha),\{\beta^*(\theta_i,\alpha)\}_i)\}\big]\ \ \text{s.t.}\ \ \mathbb E\big[b(\alpha)\mathbb 1\{\cdots\}\big]\le B .b:A→R≥0​max​ E[(wTα−b(α))1{b(α)≥max(r(α),{β∗(θi​,α)}i​)}]  s.t.  E[b(α)1{⋯}]≤B.

A pacing function μ:Θ→R≥0\mu:\Theta\to\mathbb R_{\ge0}μ:Θ→R≥0​ scales values down to paced values wTα/(1+μ(w,B))w^T\alpha/(1+\mu(w,B))wTα/(1+μ(w,B)). Let HαμH^\mu_\alphaHαμ​ be the distribution function of the highest of n−1n-1n−1 i.i.d. paced values, and for x≥r(α)x\ge r(\alpha)x≥r(α) let

σαμ(x)=x−∫r(α)xHαμ(s)Hαμ(x) ds,\sigma^\mu_\alpha(x)=x-\int_{r(\alpha)}^x\frac{H^\mu_\alpha(s)}{H^\mu_\alpha(x)}\,ds,σαμ​(x)=x−∫r(α)x​Hαμ​(x)Hαμ​(s)​ds,

with σαμ(x)=0\sigma^\mu_\alpha(x)=0σαμ​(x)=0 if Hαμ(x)=0H^\mu_\alpha(x)=0Hαμ​(x)=0 and σαμ(x)=x\sigma^\mu_\alpha(x)=xσαμ​(x)=x if x<r(α)x<r(\alpha)x<r(α). The value pacing–based strategy (Definition 2) is

βμ(w,B,α)=σαμ(wTα1+μ(w,B)).\beta^\mu(w,B,\alpha)=\sigma^\mu_\alpha\Big(\frac{w^T\alpha}{1+\mu(w,B)}\Big).βμ(w,B,α)=σαμ​(1+μ(w,B)wTα​).

For a buyer facing competitors who use βμ\beta^\muβμ, the dual function is qμ(w,B,t)=sup⁡bE[(wTα−(1+t)b(α))1{⋯ }]+tBq^\mu(w,B,t)=\sup_b\mathbb E[(w^T\alpha-(1+t)b(\alpha))\mathbb 1\{\cdots\}]+tBqμ(w,B,t)=supb​E[(wTα−(1+t)b(α))1{⋯}]+tB for t≥0t\ge0t≥0.

Formalization targets

Goal: Theorem 1 (p. 7)

∃ μ:Θ→R≥0such thatβμ is an SFPE.\exists\,\mu:\Theta\to\mathbb R_{\ge0}\quad\text{such that}\quad \beta^\mu\ \text{is an SFPE}.∃μ:Θ→R≥0​such thatβμ is an SFPE.

Milestones

  • Lemma 1 (p. 8): for every type and every t≥0t\ge0t≥0, σαμ(wTα/(1+t))\sigma^\mu_\alpha(w^T\alpha/(1+t))σαμ​(wTα/(1+t)) maximizes the Lagrangian E[(wTα/(1+t)−b(α))1{⋯ }]\mathbb E[(w^T\alpha/(1+t)-b(\alpha))\mathbb 1\{\cdots\}]E[(wTα/(1+t)−b(α))1{⋯}].
  • Proposition 1 (p. 8): on a set Θ′\Theta'Θ′ of full GGG-measure, for every μ\muμ, a minimizer t∗t^*t∗ of qμ(w,B,⋅)q^\mu(w,B,\cdot)qμ(w,B,⋅) over t≥0t\ge0t≥0 makes σαμ(wTα/(1+t∗))\sigma^\mu_\alpha(w^T\alpha/(1+t^*))σαμ​(wTα/(1+t∗)) optimal for the budget-constrained problem Qμ(w,B)Q^\mu(w,B)Qμ(w,B).
  • Lemma 2 (p. 10): the smallest minimizer ℓμ(w,B)\ell^\mu(w,B)ℓμ(w,B) of qμ(w,B,⋅)q^\mu(w,B,\cdot)qμ(w,B,⋅) over [0,ω/Bmin⁡][0,\omega/B_{\min}][0,ω/Bmin​] is nondecreasing in each wiw_iwi​ and nonincreasing in BBB.
  • Lemma 3 (p. 10): ℓμ∈L1(Θ)\ell^\mu\in L_1(\Theta)ℓμ∈L1​(Θ) and its total variation is at most V0=(d+1)max⁡(Ud,Ud+1) ω/Bmin⁡V_0=(d+1)\max(U^d,U^{d+1})\,\omega/B_{\min}V0​=(d+1)max(Ud,Ud+1)ω/Bmin​. The paper prints (d+1)Ud+1ω/Bmin⁡(d+1)U^{d+1}\omega/B_{\min}(d+1)Ud+1ω/Bmin​, which agrees when U≥1U\ge1U≥1. For U<1U<1U<1 the printed bound fails by a rescaling argument, and the mission corrects the exponent to ddd there.
  • Lemma 4 (p. 10): X0={μ∈L1(Θ):0≤μ≤ω/Bmin⁡, V(μ,Θ)≤V0}\mathcal X_0=\{\mu\in L_1(\Theta):0\le\mu\le\omega/B_{\min},\ V(\mu,\Theta)\le V_0\}X0​={μ∈L1​(Θ):0≤μ≤ω/Bmin​, V(μ,Θ)≤V0​} is nonempty, compact and convex in L1(Θ)L_1(\Theta)L1​(Θ); f(μ,μ^)=EG[qμ(w,B,μ^(w,B))]f(\mu,\hat\mu)=\mathbb E_G[q^\mu(w,B,\hat\mu(w,B))]f(μ,μ^​)=EG​[qμ(w,B,μ^​(w,B))] is continuous; C0∗(μ)=arg⁡min⁡μ^∈X0f(μ,μ^)C^*_0(\mu)=\arg\min_{\hat\mu\in\mathcal X_0}f(\mu,\hat\mu)C0∗​(μ)=argminμ^​∈X0​​f(μ,μ^​) is upper hemicontinuous with nonempty, convex, compact values.
  • Lemma 5 (p. 10): a fixed point μ∈C0∗(μ)\mu\in C^*_0(\mu)μ∈C0∗​(μ) satisfies μ(w,B)∈arg⁡min⁡t∈[0,ω/Bmin⁡]qμ(w,B,t)\mu(w,B)\in\arg\min_{t\in[0,\omega/B_{\min}]}q^\mu(w,B,t)μ(w,B)∈argmint∈[0,ω/Bmin​]​qμ(w,B,t) almost surely.
  • Proposition 2 (p. 9): some μ\muμ satisfies μ(w,B)∈arg⁡min⁡t≥0qμ(w,B,t)\mu(w,B)\in\arg\min_{t\ge0}q^\mu(w,B,t)μ(w,B)∈argmint≥0​qμ(w,B,t) almost surely.

The paper states that Proposition 2 combined with Proposition 1 implies Theorem 1.

Significance

Theorem 1 says that budget-constrained buyers in contextual first price auctions have an equilibrium that is described by one scalar per type: shade the value by the factor 1/(1+μ(w,B))1/(1+\mu(w,B))1/(1+μ(w,B)), then bid as an unconstrained buyer would in an i.i.d. first price auction against paced competitors. This is the object on which the rest of the paper is built: revenue equivalence across standard auctions (Theorem 2) and a liquid-welfare price of anarchy of at least 1/21/21/2 (Theorem 3) are both statements about pacing-based equilibria. Practically, it supports the use of pacing for budget management in first price auctions, with pacing multipliers computed as for second price auctions.

The result is proved in the paper; every proof is in its online appendix, which is not part of the source this mission cites. Nothing in the paper is formalized. A formalization would check a strong-duality argument for a non-convex problem and an infinite-dimensional fixed-point argument whose topology the authors chose with care, and it would produce reusable pieces: multivariate total variation, compactness of bounded-variation sets in L1L_1L1​, and the first price equilibrium bid against an arbitrary (possibly atomic) value distribution with a reserve price.

Difficulty

The buyer's problem Qμ(w,B)Q^\mu(w,B)Qμ(w,B) is not convex in the bid function, so standard convex duality does not give strong duality; Proposition 1 needs a direct argument combining Lagrangian optimality (Lemma 1), differentiability of qμq^\muqμ in ttt, and complementary slackness. Lemma 1 itself cannot quote the textbook first price equilibrium, because the paced value distribution λαμ\lambda^\mu_\alphaλαμ​ may have atoms and the reserve price creates discontinuities.

The fixed point lives in an infinite-dimensional space of pacing functions. The obvious topologies fail: the set X\mathcal XX of bounded pacing functions is not compact in any LpL_pLp​, and in the weak-* topology of L∞L_\inftyL∞​ upper hemicontinuity of the best-response correspondence is hard to obtain; restricting to Lipschitz functions loses the property that best responses stay in the set. The paper restricts to functions of bounded total variation, which requires the monotonicity of Lemma 2 and the variation bound of Lemma 3.

Formalization scope

Contexts are Fin d → ℝ, types are pairs (w, B) : (Fin d → ℝ) × ℝ, and wTαw^T\alphawTα is the explicit sum ∑iwiαi\sum_i w_i\alpha_i∑i​wi​αi​. A single Model structure holds d,n,U,Bmin⁡,A,F,G,rd,n,U,B_{\min},A,F,G,rd,n,U,Bmin​,A,F,G,r, and Model.Standing collects the assumptions of §2.2 together with four that the paper uses without stating them: n≥2n\ge2n≥2, Bmin⁡<UB_{\min}<UBmin​<U, AAA nonempty and bounded (so ω<∞\omega<\inftyω<∞), and measurability of AAA and rrr. Pacing functions are measurable functions on Rd×R\mathbb R^d\times\mathbb RRd×R that are nonnegative on Θ\ThetaΘ; values off Θ\ThetaΘ are irrelevant since G(Θc)=0G(\Theta^c)=0G(Θc)=0.

The conventions fixed in Lean:

  • Ties win: the indicator is b(α)≥max⁡(⋯ )b(\alpha)\ge\max(\cdots)b(α)≥max(⋯), as in the paper's tie-breaking rule.
  • An admissible bid function is measurable, nonnegative on AAA, and has an integrable payment integrand. The problem of Definition 1, QμQ^\muQμ, qμq^\muqμ and the Lagrangian of Lemma 1 all range over this class.
  • An SFPE is jointly measurable in (θ,α)(\theta,\alpha)(θ,α), so Theorem 1 must produce a measurable strategy.
  • qμq^\muqμ is the supremum over bids of p. 7, not the closed form of p. 8. σαμ\sigma^\mu_\alphaσαμ​ has an explicit branch for Hαμ(x)=0H^\mu_\alpha(x)=0Hαμ​(x)=0.
  • Argmins are stated as membership plus IsMinOn, over t≥0t\ge0t≥0 in Propositions 1–2 and over [0,ω/Bmin⁡][0,\omega/B_{\min}][0,ω/Bmin​] in Lemma 5 and ℓμ\ell^\muℓμ, as printed.
  • V(u,Ω)V(u,\Omega)V(u,Ω) takes values in EReal, with Euclidean ∥ϕ(x)∥≤1\|\phi(x)\|\le1∥ϕ(x)∥≤1 and test fields supported in Ω\OmegaΩ; L1(Θ)L_1(\Theta)L1​(Θ) is with respect to Lebesgue measure. Lemma 4 is stated on the image of X0\mathcal X_0X0​ in Lp ℝ 1 (volume.restrict Θ), in its norm topology.

A formalization in which qμq^\muqμ is defined by its closed form, in which non-integrable payments count as zero, or in which the strategy need not be measurable would make the milestones easier without proving the paper's theorem; these readings are excluded by the definitions above.

A complete development needs: first price equilibrium bids for general atomic distributions with reserves; measurability of parametric integrals; a duality argument for one-dimensional Lagrangian relaxations; multivariate bounded variation and its compact embedding in L1L_1L1​ of a box; and the Kakutani–Fan–Glicksberg theorem in a locally convex space. The bounded-variation and fixed-point parts are reusable beyond this mission. Proofs of any milestone, and alternative arguments, are welcome.

Selected references

  • S. R. Balseiro, C. Kroer, R. Kumar, Contextual Standard Auctions with Budgets: Revenue Equivalence and Efficiency Guarantees, Management Science, Articles in Advance, 2023. https://doi.org/10.1287/mnsc.2023.4719
  • S. R. Balseiro, O. Besbes, G. Y. Weintraub, Repeated Auctions with Budgets in Ad Exchanges: Approximations and Design, Management Science 61(4), 2015. https://doi.org/10.1287/mnsc.2014.2022
  • V. Conitzer, C. Kroer, E. Sodomka, N. E. Stier-Moses, Multiplicative Pacing Equilibria in Auction Markets, Operations Research 70(2), 2022. https://doi.org/10.1287/opre.2021.2167
  • V. Krishna, Auction Theory, 2nd ed., Academic Press, 2009.
  • L. C. Evans, R. F. Gariepy, Measure Theory and Fine Properties of Functions, revised ed., CRC Press, 2015.
12 thms1 active userReviewed
Algorithmic Game TheoryMechanism DesignOperations Research·Captain: mikedeng1

Contextual Standard Auctions with Budgets: Revenue Equivalence and Efficiency Guarantees 2: Value Pacing Is an Equilibrium of Every Standard Auction with the Same Expected PaymentsResearch Paper

Motivation

Online advertising sells billions of impressions a day through auctions, and most advertisers do not bid by hand: they hand a budget to the platform, which paces their bids so that spending over a campaign matches the budget. Over the last decade the major ad exchanges moved from second price to first price auctions. An advertiser who has tuned a pacing policy for one format wants to know whether it still works after the switch, and a platform wants to know whether its revenue changes.

Classical revenue equivalence (Myerson 1981; Riley and Samuelson 1981) says that, with independent private values, risk-neutral buyers and no budgets, every standard auction yields the same expected revenue. Che and Gale (1998) showed that this fails when budgets are hard: first price auctions then raise more revenue than second price auctions. Balseiro, Kroer and Kumar (2023) study budgets that bind in expectation over many auctions, with values driven by item features, and show that equivalence is restored. This mission formalizes that result, their Theorem 2.

Setting

An item has a context α∈A⊂R+d\alpha\in A\subset\mathbb R^d_+α∈A⊂R+d​ drawn from a law FFF with a density. Each of nnn buyers has a type θ=(w,B)\theta=(w,B)θ=(w,B) drawn independently from a law GGG with a density on Θ=(0,U)d×(Bmin⁡,U)\Theta=(0,U)^d\times(B_{\min},U)Θ=(0,U)d×(Bmin​,U). Here www are feature weights and BBB is a budget. The buyer's value for the item is wTαw^T\alphawTα, and ω\omegaω is the supremum of all values. The seller discloses α\alphaα, then runs an auction with reserve price r(α)r(\alpha)r(α). A buyer must keep its expected payment, over contexts and competitors' types, at most BBB.

A pacing function μ:Θ→R≥0\mu:\Theta\to\mathbb R_{\ge0}μ:Θ→R≥0​ shades every value to the paced value wTα/(1+μ(w,B))w^T\alpha/(1+\mu(w,B))wTα/(1+μ(w,B)). Let λαμ\lambda^\mu_\alphaλαμ​ be the law of the paced value when (w,B)∼G(w,B)\sim G(w,B)∼G, and Hαμ(x)=λαμ((−∞,x])n−1H^\mu_\alpha(x)=\lambda^\mu_\alpha((-\infty,x])^{n-1}Hαμ​(x)=λαμ​((−∞,x])n−1 the c.d.f. of the highest of n−1n-1n−1 independent paced values. In first price auctions the paper's candidate strategy is βμ(w,B,α)=σαμ(wTα/(1+μ(w,B)))\beta^\mu(w,B,\alpha)=\sigma^\mu_\alpha(w^T\alpha/(1+\mu(w,B)))βμ(w,B,α)=σαμ​(wTα/(1+μ(w,B))), where σαμ\sigma^\mu_\alphaσαμ​ is the first-price equilibrium bid against paced competitors.

The buyer's dual function qμ(w,B,t)q^\mu(w,B,t)qμ(w,B,t) is the supremum over bid functions b(⋅)b(\cdot)b(⋅) of the Lagrangian E[(wTα−(1+t)b(α))1{win}]+tB\mathbb E[(w^T\alpha-(1+t)b(\alpha))\mathbb 1\{\text{win}\}]+tBE[(wTα−(1+t)b(α))1{win}]+tB, with competitors bidding βμ\beta^\muβμ. The set X0\mathcal X_0X0​ collects pacing functions with values in [0,ω/Bmin⁡][0,\omega/B_{\min}][0,ω/Bmin​] and total variation at most V0=(d+1)Ud+1ω/Bmin⁡V_0=(d+1)U^{d+1}\omega/B_{\min}V0​=(d+1)Ud+1ω/Bmin​. The correspondence C0∗(μ)C^*_0(\mu)C0∗​(μ) is the set of minimizers over X0\mathcal X_0X0​ of μ^↦EG[qμ(w,B,μ^(w,B))]\hat\mu\mapsto\mathbb E_G[q^\mu(w,B,\hat\mu(w,B))]μ^​↦EG​[qμ(w,B,μ^​(w,B))]. A fixed point μ∈C0∗(μ)\mu\in C^*_0(\mu)μ∈C0∗​(μ) is an equilibrium pacing function for first price auctions.

An anonymous standard auction Aα=(r(α),Mα)\mathcal A_\alpha=(r(\alpha),M_\alpha)Aα​=(r(α),Mα​) gives the item to the highest bid at or above the reserve. Its payment Mα(b,{bi})M_\alpha(b,\{b_i\})Mα​(b,{bi​}) for a bid bbb against competing bids bib_ibi​ does not depend on the buyers' identities. An oracle returns, for each atomless value law H\mathcal HH on [0,ω][0,\omega][0,ω], a single-auction equilibrium bid ψαH\psi^{\mathcal H}_\alphaψαH​ that is nondecreasing, meets the reserve exactly for values above it, gives a zero-value bidder zero payoff, and has an atomless bid law. The candidate strategy is

Ψμ(w,B,α)=ψαλαμ(wTα1+μ(w,B)).\Psi^\mu(w,B,\alpha)=\psi^{\lambda^\mu_\alpha}_\alpha\Big(\frac{w^T\alpha}{1+\mu(w,B)}\Big).Ψμ(w,B,α)=ψαλαμ​​(1+μ(w,B)wTα​).

Formalization targets

Goal: Theorem 2 (Revenue and Pacing Equivalence)

For every family {Aα}\{\mathcal A_\alpha\}{Aα​} of anonymous standard auctions with such an oracle and every μ∈X0\mu\in\mathcal X_0μ∈X0​ with μ∈C0∗(μ)\mu\in C^*_0(\mu)μ∈C0∗​(μ):

  1. Ψμ\Psi^\muΨμ is a symmetric equilibrium of the family (Definition 4): almost every buyer type maximizes expected utility subject to its expected budget by bidding Ψμ\Psi^\muΨμ when everyone else does;
  2. for every θ∈Θ\theta\in\Thetaθ∈Θ,
E[Mα(Ψμ(θ,α),{Ψμ(θi,α)}i)]=E[βμ(θ,α) 1{βμ(θ,α)≥max⁡(r(α),{βμ(θi,α)}i)}].\mathbb E\big[M_\alpha(\Psi^\mu(\theta,\alpha),\{\Psi^\mu(\theta_i,\alpha)\}_i)\big]=\mathbb E\big[\beta^\mu(\theta,\alpha)\,\mathbb 1\{\beta^\mu(\theta,\alpha)\ge\max(r(\alpha),\{\beta^\mu(\theta_i,\alpha)\}_i)\}\big].E[Mα​(Ψμ(θ,α),{Ψμ(θi​,α)}i​)]=E[βμ(θ,α)1{βμ(θ,α)≥max(r(α),{βμ(θi​,α)}i​)}].

Milestones

  • §3.2, p. 8, last display. For t≥0t\ge0t≥0 and xα=wTα/(1+t)x_\alpha=w^T\alpha/(1+t)xα​=wTα/(1+t),
qμ(w,B,t)=(1+t) Eα[1{xα≥r(α)}∫r(α)xαHαμ(s) ds]+tB.q^\mu(w,B,t)=(1+t)\,\mathbb E_\alpha\Big[\mathbb 1\{x_\alpha\ge r(\alpha)\}\int_{r(\alpha)}^{x_\alpha}H^\mu_\alpha(s)\,ds\Big]+tB.qμ(w,B,t)=(1+t)Eα​[1{xα​≥r(α)}∫r(α)xα​​Hαμ​(s)ds]+tB.
  • §4, p. 12, key step. When the competitors bid Ψμ\Psi^\muΨμ, the dual function of the buyer's problem in any standard auction family equals qμ(w,B,t)q^\mu(w,B,t)qμ(w,B,t).

Significance

Theorem 2 says that pacing is portable: an equilibrium pacing function computed for first price auctions is an equilibrium pacing function for second price auctions, all-pay auctions, or any context-dependent mixture of standard formats. Each buyer's expected spend is the same in all of them, so the seller's expected revenue is too. This explains why budget-management systems built for second price auctions keep working after a move to first price. It also separates in-expectation budgets, where equivalence holds, from the hard budgets of Che and Gale, where it fails. Theorem 3 of the same paper, a liquid-welfare price of anarchy of 1/21/21/2, uses Theorem 2 to argue in the second price auction.

The paper's proofs are in its online appendix, which is not part of the source this mission cites. To our knowledge no part of the argument has been machine-checked. Classical revenue equivalence for incentive-compatible direct mechanisms without budgets exists on the platform as other items, but none has pacing, budgets or contexts. The mission asks for a machine-checked proof of Theorem 2 and of the two steps the authors name.

Difficulty

The classical argument pins payments by the allocation rule and the envelope formula, value by value. Here a buyer's decision is a whole bid function of the context, coupled across contexts by one budget constraint, and the budget-constrained problem is not convex. So the envelope argument cannot be applied to the constrained problem directly. The obvious move, Lagrangian relaxation, needs strong duality for a non-convex problem, and it needs the dual of each standard auction to agree with the first-price dual for every multiplier, not only at the optimum. The paced value laws λαμ\lambda^\mu_\alphaλαμ​ need not have densities, so the textbook single-auction results, which assume a positive density, do not apply verbatim. Finally, the fixed-point hypothesis ties all buyer types together, and optimality must be recovered almost surely in the type.

Formalization scope

Everything is defined in the namespace ContextualAuctions.RevEquiv. Contexts are Fin d → ℝ, types are (Fin d → ℝ) × ℝ, and wTαw^T\alphawTα is an explicit sum. The joint law of the context and the n−1n-1n−1 competitor types is F.prod (Measure.pi fun _ => G). Ties win, as in the paper's tie-breaking rule. The standing assumptions are d≥2d\ge2d≥2, FFF and GGG probability measures with densities concentrated on AAA and Θ\ThetaΘ, AAA in the open positive orthant, and rrr measurable. Disclosed additions, implicit in the paper:

  • n≥2n\ge2n≥2 and 0<Bmin⁡<U0<B_{\min}<U0<Bmin​<U;
  • AAA measurable, nonempty and bounded, so that ω<∞\omega<\inftyω<∞;
  • pacing functions and bids are measurable; first-price bids are nonnegative on AAA, and standard-auction bids are nonnegative for almost every context, because the oracle constrains Ψμ\Psi^\muΨμ only where λαμ\lambda^\mu_\alphaλαμ​ is atomless;
  • the payment rule is measurable along measurable bid maps on any measurable space;
  • the strategy Ψμ\Psi^\muΨμ is jointly measurable, since the oracle is a black box.

The dual function is the supremum over admissible bids, which are measurable with integrable payment. Its closed form is a milestone and never its definition. Total variation (Definition 3) is extended-real valued and uses the Euclidean norm of the test field. In oracle property 1, expected payments are taken in [0,∞][0,\infty][0,∞], so a bid with infinite expected payment can never look free. The payment identity of Theorem 2 is stated in [0,∞][0,\infty][0,∞], so it also asserts that the left side is finite. This rules out the trivializing readings: a closed-form definition of qμq^\muqμ that makes the first milestone hold by definition, Bochner junk zeros for non-integrable payments, and a vacuous variation bound.

A complete development needs single-auction revenue equivalence for i.i.d. atomless value laws without densities. It also needs the measure-theoretic interchange of a supremum over bid functions with the expectation over contexts, and strong duality for the budget-constrained problem. These pieces, notably the single-auction equivalence for general atomless laws, are reusable beyond this mission. Contributions toward any of them are welcome.

Selected references

  • S. Balseiro, C. Kroer, R. Kumar, Contextual Standard Auctions with Budgets: Revenue Equivalence and Efficiency Guarantees, Management Science, Articles in Advance, 2023. https://doi.org/10.1287/mnsc.2023.4719
  • Y.-K. Che, I. Gale, Standard Auctions with Financially Constrained Bidders, Review of Economic Studies 65(1), 1998. https://doi.org/10.1111/1467-937X.00041
  • R. Myerson, Optimal Auction Design, Mathematics of Operations Research 6(1), 1981. https://doi.org/10.1287/moor.6.1.58
  • J. Riley, W. Samuelson, Optimal Auctions, American Economic Review 71(3), 1981. https://www.jstor.org/stable/1802779
  • V. Krishna, Auction Theory, 2nd ed., Academic Press, 2009. https://doi.org/10.1016/C2009-0-22474-3
  • S. Balseiro, Y. Gur, Learning in Repeated Auctions with Budgets: Regret Minimization and Equilibrium, Management Science 65(9), 2019. https://doi.org/10.1287/mnsc.2018.3174
6 thms1 active userReviewed
Graph TheoryLinear OptimizationOperations Research+1·Captain: mikedeng1

Linear Programming and Community Detection 6: The LP Relaxation Fails w.h.p. in the Very Dense and Dense SBM Above Explicit ThresholdsResearch Paper

Motivation

Community detection asks for a hidden partition of the nodes of a graph from its edges. Its simplest probabilistic benchmark is the two-community stochastic block model (SBM), in which the minimum bisection problem is the maximum a posteriori estimator. That problem is NP-hard, so convex relaxations are used in its place, and the question is when a relaxation still returns the hidden partition exactly. For semidefinite relaxations this question has sharp answers (Abbe, Bandeira, Hall 2016; Bandeira 2018). Del Pia, Khajavirad and Kunisky (arXiv:2006.03213v2) ask the same question for the most common linear programming relaxation of minimum bisection, the metric relaxation. They show that it recovers the planted bisection in very dense graphs when p−q>1/2p-q>1/2p−q>1/2 (their Theorem 6). This mission is their converse in the very dense and dense regimes, Theorem 7: above explicit thresholds the LP fails with high probability.

Setting

The node set is V={1,…,n}V=\{1,\dots,n\}V={1,…,n}, nnn even. A bisection is a partition V=V1⊔V2V=V_1\sqcup V_2V=V1​⊔V2​ with ∣V1∣=∣V2∣=n/2|V_1|=|V_2|=n/2∣V1​∣=∣V2​∣=n/2. The cut vector xˉ∈{0,1}(n2)\bar x\in\{0,1\}^{\binom n2}xˉ∈{0,1}(2n​) of V1,V2V_1,V_2V1​,V2​ has xˉij=1\bar x_{ij}=1xˉij​=1 exactly when iii and jjj lie on different sides. For a graph GGG with adjacency indicators aija_{ij}aij​, the metric relaxation is

(LP-P)min⁡x∑i<jaijxijs.t.xij≤xik+xjk,xij+xik+xjk≤2,∑i<jxij=n24,\text{(LP-P)}\qquad \min_x \sum_{i<j}a_{ij}x_{ij}\quad\text{s.t.}\quad x_{ij}\le x_{ik}+x_{jk},\quad x_{ij}+x_{ik}+x_{jk}\le2,\quad \sum_{i<j}x_{ij}=\frac{n^2}{4},(LP-P)xmin​i<j∑​aij​xij​s.t.xij​≤xik​+xjk​,xij​+xik​+xjk​≤2,i<j∑​xij​=4n2​,

over x∈R(n2)x\in\mathbb R^{\binom n2}x∈R(2n​), with the first family over distinct i,j,ki,j,ki,j,k with i<ji<ji<j and the second over i<j<ki<j<ki<j<k. The LP recovers the planted bisection if xˉ\bar xxˉ is its unique optimal solution.

In the SBM Gn,p,q\mathcal G_{n,p,q}Gn,p,q​ a bisection V1,V2V_1,V_2V1​,V2​ is first drawn uniformly at random; then each pair on the same side is an edge with probability ppp and each pair across with probability qqq, independently, with p>qp>qp>q. The Erdős–Rényi graph Gn,p\mathcal G_{n,p}Gn,p​ has every pair an edge with probability ppp. The dense scaling (20) is p=αn−ωp=\alpha n^{-\omega}p=αn−ω, q=βn−ωq=\beta n^{-\omega}q=βn−ω with α>β>0\alpha>\beta>0α>β>0, 0≤ω<10\le\omega<10≤ω<1; ω=0\omega=0ω=0 is the very dense regime. An event holds with high probability (w.h.p.) if its probability tends to 111 as n→∞n\to\inftyn→∞.

For a graph GGG, ρG(i,j)\rho_G(i,j)ρG​(i,j) is the shortest-path distance (+∞+\infty+∞ across components), ρmax⁡(G)=max⁡i,jρG(i,j)\rho_{\max}(G)=\max_{i,j}\rho_G(i,j)ρmax​(G)=maxi,j​ρG​(i,j) is the diameter, and ρavg(G)=2n2∑i<jρG(i,j)\rho_{\mathrm{avg}}(G)=\frac2{n^2}\sum_{i<j}\rho_G(i,j)ρavg​(G)=n22​∑i<j​ρG​(i,j) is the average distance. EEE is the edge set of GGG, E0E_0E0​ the set of edges across the planted bisection.

Formalization targets

Goal: Theorem 7 (pp. 15–16)

Let p=αn−ωp=\alpha n^{-\omega}p=αn−ω, q=βn−ωq=\beta n^{-\omega}q=βn−ω with α>β>0\alpha>\beta>0α>β>0, ω∈[0,1)\omega\in[0,1)ω∈[0,1), and α,β<1\alpha,\beta<1α,β<1 if ω=0\omega=0ω=0. If one of

  1. ω=0\omega=0ω=0 and β>max⁡{3−α−(3−α)2−4α2, 2α−1}\beta>\max\big\{\tfrac{3-\alpha-\sqrt{(3-\alpha)^2-4\alpha}}2,\ 2\alpha-1\big\}β>max{23−α−(3−α)2−4α​​, 2α−1} (22),
  2. ω∈(0,1)\omega\in(0,1)ω∈(0,1), 11−ω∉N\tfrac1{1-\omega}\notin\mathbb N1−ω1​∈/N and β>α/(2⌈11−ω⌉−1)\beta>\alpha/(2\lceil\tfrac1{1-\omega}\rceil-1)β>α/(2⌈1−ω1​⌉−1) (23),
  3. ω=12\omega=\tfrac12ω=21​ and β>max⁡{1−2e−α22−e−α2,13+2e−α2}α\beta>\max\big\{\tfrac{1-2e^{-\alpha^2}}{2-e^{-\alpha^2}},\tfrac1{3+2e^{-\alpha^2}}\big\}\alphaβ>max{2−e−α21−2e−α2​,3+2e−α21​}α (24),
  4. ω∈(0,1)\omega\in(0,1)ω∈(0,1), 11−ω∈N∖{2}\tfrac1{1-\omega}\in\mathbb N\setminus\{2\}1−ω1​∈N∖{2} and β>α/(21−ω−1+2e−α1/(1−ω))\beta>\alpha/\big(\tfrac2{1-\omega}-1+2e^{-\alpha^{1/(1-\omega)}}\big)β>α/(1−ω2​−1+2e−α1/(1−ω)) (25)

holds, then

lim⁡n→∞P(V1,G)∼Gn,p,q[(LP-P) does not recover V1,V2]=1.\lim_{n\to\infty}\mathbb P_{(V_1,G)\sim\mathcal G_{n,p,q}}\big[\text{(LP-P) does not recover }V_1,V_2\big]=1.n→∞lim​P(V1​,G)∼Gn,p,q​​[(LP-P) does not recover V1​,V2​]=1.

Milestones

  1. (26), p. 16: for p,q=Ω(log⁡n/n)p,q=\Omega(\log n/n)p,q=Ω(logn/n) with lim⁡q/p∈(0,1]\lim q/p\in(0,1]limq/p∈(0,1], the ratio ∣E0∣/∣E∣|E_0|/|E|∣E0​∣/∣E∣ lies in (1±ϵ) q/(p+q)(1\pm\epsilon)\,q/(p+q)(1±ϵ)q/(p+q) w.h.p.
  2. Proposition 11, p. 28: a coupling of Gn,q\mathcal G_{n,q}Gn,q​, Gn,q,p\mathcal G_{n,q,p}Gn,q,p​ and Gn,p\mathcal G_{n,p}Gn,p​ in which the three graphs are nested.
  3. Proposition 12, p. 28: w.h.p. ρmax⁡(Gn,αn−ω)=1+⌊11−ω⌋\rho_{\max}(\mathcal G_{n,\alpha n^{-\omega}})=1+\lfloor\frac1{1-\omega}\rfloorρmax​(Gn,αn−ω​)=1+⌊1−ω1​⌋.
  4. Proposition 14, p. 29 (Shimizu): w.h.p. ρavg(Gn,αn−ω)∈((1−ϵ)μ,(1+ϵ)μ)\rho_{\mathrm{avg}}(\mathcal G_{n,\alpha n^{-\omega}})\in((1-\epsilon)\mu,(1+\epsilon)\mu)ρavg​(Gn,αn−ω​)∈((1−ϵ)μ,(1+ϵ)μ), where μ=⌈11−ω⌉+1{11−ω∈N}e−α1/(1−ω)\mu=\lceil\frac1{1-\omega}\rceil+\mathbf 1\{\frac1{1-\omega}\in\mathbb N\}e^{-\alpha^{1/(1-\omega)}}μ=⌈1−ω1​⌉+1{1−ω1​∈N}e−α1/(1−ω).
  5. Propositions 16, 17, 18, pp. 29–30: the diameter of the very dense and dense SBM, and lower bounds on its average distance, in the three cases ω=0\omega=0ω=0, 11−ω∉N\frac1{1-\omega}\notin\mathbb N1−ω1​∈/N and 11−ω∈N\frac1{1-\omega}\in\mathbb N1−ω1​∈N.

Significance

Theorem 6 and Theorem 7 together bracket the success region of the metric relaxation in the very dense regime: recovery for p−q>1/2p-q>1/2p−q>1/2, failure above the curve (22). In the dense regime, Theorem 7 gives explicit linear thresholds β>c(ω,α) α\beta>c(\omega,\alpha)\,\alphaβ>c(ω,α)α above which the LP fails. The proof reduces failure to a deterministic distance criterion (Theorem 5, posed in mission 4 of this series) and turns asymptotic distance distributions of random graphs into statements about an LP.

The results are proved in the paper, partly by citing Bollobás's diameter theorems and Shimizu's average-distance theorem; none of them has a machine-checked proof. Formalizing the mission yields, besides Theorem 7, the diameter and average distance of dense Erdős–Rényi graphs (Propositions 12 and 14) and the monotone coupling between Erdős–Rényi graphs and the SBM (Proposition 11), each of which is usable beyond this paper. The coupling and (26) are also the inputs of the logarithmic-degree result (Theorem 8, mission 7).

Difficulty

The deterministic half is short: once ∣E0∣/∣E∣|E_0|/|E|∣E0​∣/∣E∣, ρmax⁡\rho_{\max}ρmax​ and ρavg\rho_{\mathrm{avg}}ρavg​ are pinned down, the case computations (28)–(29) reduce to inequalities between explicit expressions in α\alphaα, β\betaβ and ω\omegaω. The difficulty is the random-graph input. The diameter of Gn,p\mathcal G_{n,p}Gn,p​ for p=αn−ωp=\alpha n^{-\omega}p=αn−ω needs the second-moment analysis of the number of pairs at distance greater than ddd (Bollobás, Corollaries 10.11–10.12), and the average distance needs the finer limit law for path counts at the critical exponent 11−ω∈N\frac1{1-\omega}\in\mathbb N1−ω1​∈N, where the constant e−α1/(1−ω)e^{-\alpha^{1/(1-\omega)}}e−α1/(1−ω) appears. A first-moment argument gives only one side of each estimate. Concentration of ∣E∣|E|∣E∣ and ∣E0∣|E_0|∣E0​∣ (26) is a Hoeffding bound, but it must hold uniformly over the random planted bisection.

Formalization scope

Nodes are Fin n and graphs SimpleGraph (Fin n). A bisection is its side V1V_1V1​ with 2∣V1∣=n2|V_1|=n2∣V1​∣=n. The LP variable is a symmetric, zero-diagonal array Fin n → Fin n → ℝ, with no box constraints, so uniqueness is uniqueness over R(n2)\mathbb R^{\binom n2}R(2n​). Probabilities are finite sums over all graphs (and all bisections for the SBM). Edge probabilities are clamped into [0,1][0,1][0,1], which affects only finitely many nnn. SBM limits run over even n=2mn=2mn=2m. Distances are in ℕ∞ and ρavg\rho_{\mathrm{avg}}ρavg​ in [0,∞][0,\infty][0,∞], so disconnected samples have infinite diameter and average distance; no real truncation is used.

Disclosed choices:

  • Theorem 7 adds α>β\alpha>\betaα>β, the SBM's standing assumption p>qp>qp>q (§4, p. 13), also assumed by Proposition 18 and (26).
  • (26) is posed for p=αs(n)p=\alpha s(n)p=αs(n), q=βs(n)q=\beta s(n)q=βs(n) with s(n)≥clog⁡n/ns(n)\ge c\log n/ns(n)≥clogn/n and αs(n)≤1\alpha s(n)\le1αs(n)≤1 eventually. Then q/p=β/αq/p=\beta/\alphaq/p=β/α, which avoids the printed "lim⁡p/q∈(0,1]\lim p/q\in(0,1]limp/q∈(0,1]", a slip for q/pq/pq/p.
  • Proposition 11 is posed as printed, with middle law Gn,q,p\mathcal G_{n,q,p}Gn,q,p​ (within qqq, across ppp).

Two encodings would make the statements trivial, and both are excluded. Unnormalized LP variables would make uniqueness false. Reading ρ\rhoρ in R\mathbb RR via toReal would make the upper bounds hold for disconnected graphs.

The deterministic distance criterion (Theorem 5) is posed in mission 4 and is not restated. Contributions are welcome on the coupling, Hoeffding-type concentration of edge counts, and the Erdős–Rényi distance asymptotics. These are reusable for any finite random-graph development.

Selected references

  • A. Del Pia, A. Khajavirad, D. Kunisky, Linear Programming and Community Detection, Mathematics of Operations Research, 2023; arXiv:2006.03213v2. https://arxiv.org/abs/2006.03213
  • E. Abbe, A. S. Bandeira, G. Hall, Exact recovery in the stochastic block model, IEEE Trans. Inf. Theory 62 (2016) 471–487. https://arxiv.org/abs/1405.3267
  • A. S. Bandeira, Random Laplacian matrices and convex relaxations, Found. Comput. Math. 18 (2018) 345–379. https://arxiv.org/abs/1504.03987
  • B. Bollobás, Random Graphs, 2nd ed., Cambridge Studies in Advanced Mathematics 73, Cambridge University Press, 2001. https://doi.org/10.1017/CBO9780511814068
  • N. Shimizu, The average distance of dense homogeneous random graphs, Mathematical Engineering Technical Report, Department of Mathematical Informatics, The University of Tokyo, November 2017 (Theorem 1; reference [38] of the paper). https://www.keisu.t.u-tokyo.ac.jp/research/techrep/
14 thms1 active userReviewed
Graph TheoryLinear OptimizationOperations Research+1·Captain: mikedeng1

Linear Programming and Community Detection 7: The LP Relaxation Fails w.h.p. in the Logarithmic-Degree SBMResearch Paper

Motivation

The stochastic block model (SBM) is the standard benchmark for community detection: a random graph whose vertices are split into hidden communities, with denser connections inside a community than across. For two equal communities, the best possible estimator of the hidden partition is the minimum bisection problem, which is NP-hard in the worst case. Its tractable relaxations are therefore compared by asking for which edge densities they still recover the hidden partition exactly.

For semidefinite programming the answer is complete in the logarithmic-degree regime: Abbe, Bandeira and Hall (arXiv:1405.3267) located the information-theoretic threshold α−β=2\sqrt\alpha-\sqrt\beta=\sqrt2α​−β​=2​, and the SDP relaxation was shown to reach it by Hajek, Wu and Xu (arXiv:1412.6156) and by Bandeira (arXiv:1504.03987). Linear programming relaxations are cheaper and widely used in combinatorial optimization, but their behaviour on the SBM was not known. Del Pia, Khajavirad and Kunisky (arXiv:2006.03213, Mathematics of Operations Research, 2023) study the most common one, the metric relaxation. This mission formalizes their result for the logarithmic-degree regime (Theorem 8): there the LP relaxation fails to recover the hidden partition with high probability, at every choice of the parameters.

Setting

Let V={1,…,n}V=\{1,\dots,n\}V={1,…,n} with nnn even. A bisection is a partition V=V1⊔V2V=V_1\sqcup V_2V=V1​⊔V2​ with ∣V1∣=∣V2∣=n/2|V_1|=|V_2|=n/2∣V1​∣=∣V2​∣=n/2; its cost in a graph GGG is the number of edges between V1V_1V1​ and V2V_2V2​. Given a fixed planted bisection V1,V2V_1,V_2V1​,V2​, an optimization problem recovers it if its unique optimal solution corresponds to V1,V2V_1,V_2V1​,V2​.

The cut vector xˉ∈{0,1}(n2)\bar x\in\{0,1\}^{\binom n2}xˉ∈{0,1}(2n​) of V1,V2V_1,V_2V1​,V2​ has xˉij=1\bar x_{ij}=1xˉij​=1 exactly when i,ji,ji,j lie on different sides. With aij=1a_{ij}=1aij​=1 for edges and 000 otherwise, the metric relaxation is

(LP-P)min⁡x∑i<jaijxij  s.t.  xij≤xik+xjk,  xij+xik+xjk≤2,  ∑i<jxij=n24,\text{(LP-P)}\quad \min_x \sum_{i<j}a_{ij}x_{ij}\ \ \text{s.t.}\ \ x_{ij}\le x_{ik}+x_{jk},\ \ x_{ij}+x_{ik}+x_{jk}\le2,\ \ \sum_{i<j}x_{ij}=\frac{n^2}{4},(LP-P)xmin​i<j∑​aij​xij​  s.t.  xij​≤xik​+xjk​,  xij​+xik​+xjk​≤2,  i<j∑​xij​=4n2​,

the first family over distinct i,j,ki,j,ki,j,k and the second over i<j<ki<j<ki<j<k, with xji:=xijx_{ji}:=x_{ij}xji​:=xij​. Every bisection's cut vector is feasible.

The distance ρG(i,j)\rho_G(i,j)ρG​(i,j) is the length of a shortest iii–jjj path (+∞+\infty+∞ across components). The diameter is ρmax⁡(G)=max⁡i,jρG(i,j)\rho_{\max}(G)=\max_{i,j}\rho_G(i,j)ρmax​(G)=maxi,j​ρG​(i,j) and the average distance is ρavg(G)=2n2∑i<jρG(i,j)\rho_{\mathrm{avg}}(G)=\frac{2}{n^2}\sum_{i<j}\rho_G(i,j)ρavg​(G)=n22​∑i<j​ρG​(i,j).

The Erdős–Rényi model Gn,p\mathcal G_{n,p}Gn,p​ draws each edge independently with probability ppp. The SBM Gn,p,q\mathcal G_{n,p,q}Gn,p,q​ chooses a bisection V1,V2V_1,V_2V1​,V2​ uniformly at random, then draws each pair inside a side with probability ppp and each pair across with probability qqq, independently, where p>qp>qp>q. The logarithmic-degree regime (21) is

p=αlog⁡nn,q=βlog⁡nn,α>β>0.p=\alpha\frac{\log n}{n},\qquad q=\beta\frac{\log n}{n},\qquad \alpha>\beta>0 .p=αnlogn​,q=βnlogn​,α>β>0.

An event AnA_nAn​ holds with high probability if P[An]→1\mathbb P[A_n]\to1P[An​]→1 as n→∞n\to\inftyn→∞.

Formalization targets

Goal: Theorem 8 (p. 18)

α>β>0 ⟹ lim⁡n→∞PGn,p,q[(LP-P) does not recover V1,V2]=1.\alpha>\beta>0\ \Longrightarrow\ \lim_{n\to\infty}\mathbb P_{\mathcal G_{n,p,q}}\big[\text{(LP-P) does not recover }V_1,V_2\big]=1 .α>β>0 ⟹ n→∞lim​PGn,p,q​​[(LP-P) does not recover V1​,V2​]=1.

Milestones

  1. The threshold of [2] (p. 14, strict cases): if α−β>2\sqrt\alpha-\sqrt\beta>\sqrt2α​−β​>2​, the minimum bisection problem recovers V1,V2V_1,V_2V1​,V2​ with high probability; if α−β<2\sqrt\alpha-\sqrt\beta<\sqrt2α​−β​<2​, it does not.
  2. Proposition 13 (p. 29): for α>1\alpha>1α>1 and G∼Gn,αlog⁡n/nG\sim\mathcal G_{n,\alpha\log n/n}G∼Gn,αlogn/n​, ρmax⁡(G)∈((1−ϵ)log⁡nlog⁡log⁡n,(1+ϵ)log⁡nlog⁡log⁡n)\rho_{\max}(G)\in\big((1-\epsilon)\frac{\log n}{\log\log n},(1+\epsilon)\frac{\log n}{\log\log n}\big)ρmax​(G)∈((1−ϵ)loglognlogn​,(1+ϵ)loglognlogn​) with high probability.
  3. Proposition 15 (p. 29, Theorem 1 of Chung–Lu): for α>1\alpha>1α>1, ρavg(G)∈((1−ϵ)log⁡nlog⁡pn,(1+ϵ)log⁡nlog⁡pn)\rho_{\mathrm{avg}}(G)\in\big((1-\epsilon)\frac{\log n}{\log pn},(1+\epsilon)\frac{\log n}{\log pn}\big)ρavg​(G)∈((1−ϵ)logpnlogn​,(1+ϵ)logpnlogn​) with high probability.
  4. Proposition 19 (p. 30): for α>β>0\alpha>\beta>0α>β>0 with α+β2>1\frac{\alpha+\beta}{2}>12α+β​>1 and G∼Gn,p,qG\sim\mathcal G_{n,p,q}G∼Gn,p,q​,
(1−ϵ)log⁡nlog⁡log⁡n≤ρavg(G)≤ρmax⁡(G)≤(1+ϵ)log⁡nlog⁡log⁡n(1-\epsilon)\frac{\log n}{\log\log n}\le\rho_{\mathrm{avg}}(G)\le\rho_{\max}(G)\le(1+\epsilon)\frac{\log n}{\log\log n}(1−ϵ)loglognlogn​≤ρavg​(G)≤ρmax​(G)≤(1+ϵ)loglognlogn​

with high probability.

Two results the proof of Theorem 8 uses are posed in sibling missions of this series and are not restated here: Theorem 5 (a distance-based non-recovery criterion for (LP-P), mission 4) and the coupling Proposition 11 together with the crossing-edge count (26) (mission 6).

Significance

The result. Theorem 8 separates linear from semidefinite programming on a natural statistical problem. In the logarithmic regime the SDP relaxation recovers the planted bisection whenever any method can, while the metric LP fails with high probability for all α>β>0\alpha>\beta>0α>β>0, including parameters far above the information-theoretic threshold. The paper's Theorem 6 shows that in the very dense regime (constant p,qp,qp,q) the LP does succeed when p−q>1/2p-q>1/2p−q>1/2; Theorem 8 shows that at logarithmic degree no such window exists.

Formalizing it. The theorem is proved in the paper, modulo results cited from the random-graph literature. Nothing in this mission is machine-checked yet. Formalization adds a precise account of what the cited distance asymptotics assert; Proposition 19's upper bound is justified in the paper only by reference to "a standard result on the diameter of the SBM" obtained by adapting an argument of Abbe's survey, so a complete proof of it is new work. Propositions 13 and 15 are classical facts about Erdős–Rényi graphs that have no formal proof in any library known to the curators, and are reusable well beyond community detection.

Difficulty

The LP side of the argument is deterministic and finite. The difficulty is probabilistic. The distance results require tight control of breadth-first neighbourhood growth in sparse random graphs near the connectivity threshold, where the average degree is only logarithmic, with error terms small enough to pin the diameter to a (1±ϵ)(1\pm\epsilon)(1±ϵ) window. A first-moment estimate of path counts gives only one side of each bound. In the SBM the two edge probabilities make neighbourhood growth inhomogeneous, so the Erdős–Rényi results do not transfer verbatim. The threshold of [2] is a separate sharp large-deviation statement, at the level of individual vertices' in- and out-degrees.

Formalization scope

Vertices are Fin n (0-based); graphs are SimpleGraph (Fin n). A bisection is recorded by its side V1V_1V1​ with 2∣V1∣=n2|V_1|=n2∣V1​∣=n. The LP variable is a symmetric, zero-diagonal array x:V×V→Rx:V\times V\to\mathbb Rx:V×V→R, which is a point of R(n2)\mathbb R^{\binom n2}R(2n​); no box constraints are added. LP recovery means that the planted cut vector is the unique optimal solution; minimum-bisection recovery means that every bisection of cost at most the planted one is the planted unordered partition. Distances are in N∪{∞}\mathbb N\cup\{\infty\}N∪{∞} and the average distance in [0,∞][0,\infty][0,∞], so a disconnected sample has infinite diameter. The random-graph laws are finite sums over all graphs; the SBM averages over all bisections. Edge probabilities are clamped into [0,1][0,1][0,1], which matters only for the finitely many nnn with αlog⁡n/n>1\alpha\log n/n>1αlogn/n>1. "With high probability" is a limit of probabilities equal to 111; for the SBM the limit runs along n=2mn=2mn=2m. Logarithms are natural.

Disclosed choices relative to the printed statements: Theorem 8 prints only α,β>0\alpha,\beta>0α,β>0, and α>β\alpha>\betaα>β is added as the SBM's standing assumption p>qp>qp>q (p. 13) and that of (21). The threshold of [2] is printed as an "if and only if"; only the two strict implications are posed, and the boundary α−β=2\sqrt\alpha-\sqrt\beta=\sqrt2α​−β​=2​ is left out.

The statements cannot be satisfied vacuously: the laws are genuine probability distributions for every nnn, the events compare extended-valued distances without truncation, and recovery requires uniqueness over all real feasible points of (LP-P). An encoding with an empty or degenerate feasible set, or with distances truncated to finite reals, would change the theorems and is excluded.

A complete development needs: Erdős–Rényi and SBM laws, breadth-first exploration bounds for sparse random graphs, the coupling of the SBM between two Erdős–Rényi graphs (mission 6), the LP non-recovery criterion (mission 4), and the information-theoretic threshold of [2]. Contributions on the random-graph distance asymptotics are reusable on their own and especially welcome.

Selected references

  • A. Del Pia, A. Khajavirad, D. Kunisky, Linear Programming and Community Detection, Mathematics of Operations Research, 2023. https://arxiv.org/abs/2006.03213 (v2 is the cited version)
  • E. Abbe, A. S. Bandeira, G. Hall, Exact recovery in the stochastic block model, IEEE Transactions on Information Theory 62:471–487, 2016. https://arxiv.org/abs/1405.3267
  • E. Abbe, Community detection and stochastic block models: recent developments, Journal of Machine Learning Research 18:1–86, 2018. https://arxiv.org/abs/1703.10146
  • F. Chung, L. Lu, The diameter of sparse random graphs, Advances in Applied Mathematics 26(4):257–279, 2001. https://doi.org/10.1006/aama.2001.0720
  • F. Chung, L. Lu, The average distances in random graphs with given expected degrees, PNAS 99(25):15879–15882, 2002. https://doi.org/10.1073/pnas.252631999
  • B. Hajek, Y. Wu, J. Xu, Achieving exact cluster recovery threshold via semidefinite programming, IEEE Transactions on Information Theory 62(5):2788–2797, 2016. https://arxiv.org/abs/1412.6156
  • A. S. Bandeira, Random Laplacian matrices and convex relaxations, Foundations of Computational Mathematics 18(2):345–379, 2018. https://arxiv.org/abs/1504.03987
10 thms1 active userReviewed
CombinatoricsGraph TheoryLinear Optimization+2·Captain: mikedeng1

Linear Programming and Community Detection 5: The LP Relaxation Recovers the Planted Bisection w.h.p. in the Very Dense SBM When p − q > 1/2Research Paper

Motivation

The stochastic block model (SBM), introduced by Holland, Laskey and Leinhardt (1983), is the standard random model for networks with community structure, and the basic benchmark for community-detection algorithms (Abbe 2018). In its two-community form the nodes are split into two equal halves, edges inside a half appear with probability ppp, and edges across with probability q<pq<pq<p. The task of exact recovery is to identify the hidden halves from the graph alone.

The maximum a posteriori estimator for this model is the minimum bisection problem (find a balanced cut with the fewest crossing edges), which is NP-hard in general. Abbe, Bandeira and Hall (2016) determined when it recovers the planted bisection, and showed that a semidefinite relaxation is as good as the integer program. Del Pia, Khajavirad and Kunisky (arXiv:2006.03213, Mathematics of Operations Research 2023) ask the same question for the cheapest standard relaxation: the metric linear programming relaxation, defined by triangle inequalities. This mission is their positive result in the very dense regime.

Setting

Let V={1,…,n}V=\{1,\dots,n\}V={1,…,n} with nnn even. A bisection is a partition V=V1∪V2V=V_1\cup V_2V=V1​∪V2​ with ∣V1∣=∣V2∣=n/2|V_1|=|V_2|=n/2∣V1​∣=∣V2​∣=n/2. For a graph G=(V,E)G=(V,E)G=(V,E) let aij=1a_{ij}=1aij​=1 if (i,j)∈E(i,j)\in E(i,j)∈E and 000 otherwise. The metric relaxation (LP-P) is

min⁡x∈R(n2) ∑i<jaijxijs.t.xij≤xik+xjk,xij+xik+xjk≤2,∑i<jxij=n24,\min_{x\in\mathbb R^{\binom n2}}\ \sum_{i<j}a_{ij}x_{ij}\quad\text{s.t.}\quad x_{ij}\le x_{ik}+x_{jk},\quad x_{ij}+x_{ik}+x_{jk}\le 2,\quad \sum_{i<j}x_{ij}=\frac{n^2}{4},x∈R(2n​)min​ i<j∑​aij​xij​s.t.xij​≤xik​+xjk​,xij​+xik​+xjk​≤2,i<j∑​xij​=4n2​,

the first constraint for all distinct i,j,ki,j,ki,j,k with i<ji<ji<j, the second for all i<j<ki<j<ki<j<k, and xji:=xijx_{ji}:=x_{ij}xji​:=xij​. The cut vector xˉ\bar xxˉ of a bisection has xˉij=0\bar x_{ij}=0xˉij​=0 for i,ji,ji,j on the same side and xˉij=1\bar x_{ij}=1xˉij​=1 otherwise. (LP-P) recovers the planted bisection when xˉ\bar xxˉ is its unique optimal solution.

In the Erdős–Rényi model Gn,p\mathcal G_{n,p}Gn,p​ every pair of nodes is an edge independently with probability ppp. In the SBM Gn,p,q\mathcal G_{n,p,q}Gn,p,q​ a bisection V1,V2V_1,V_2V1​,V2​ is first chosen uniformly at random; then pairs inside V1V_1V1​ or inside V2V_2V2​ are edges with probability ppp, pairs between them with probability qqq, all independently. Gn,0,q\mathcal G_{n,0,q}Gn,0,q​ is the random bipartite graph across a random bisection. An event occurs with high probability (w.h.p.) if its probability tends to 111 as n→∞n\to\inftyn→∞. The graph is very dense when ppp and qqq are constants.

For a finite graph G=(V,E)G=(V,E)G=(V,E) and b∈Z≥0Vb\in\mathbb Z_{\ge0}^Vb∈Z≥0V​, a bbb-factor (or fff-factor) is a set F⊆EF\subseteq EF⊆E with exactly bvb_vbv​ edges of FFF at each node vvv; N(U,U′)N(U,U')N(U,U′) counts edges with one endpoint in UUU and one in U′U'U′; dmin⁡d_{\min}dmin​, dmax⁡d_{\max}dmax​ are the minimum and maximum degrees; and the node connectivity κ(G)\kappa(G)κ(G) is the minimum number of nodes whose deletion disconnects GGG (the number of nodes, for a complete graph).

Formalization targets

Goal: Theorem 6 (p. 15)

For constants 0<q<p<10<q<p<10<q<p<1,

p−q>12 ⟹ PG∼Gn,p,q[(LP-P) recovers the planted bisection]→n→∞1.p-q>\tfrac12\ \Longrightarrow\ \mathbb P_{G\sim\mathcal G_{n,p,q}}\big[\text{(LP-P) recovers the planted bisection}\big]\xrightarrow[n\to\infty]{}1 .p−q>21​ ⟹ PG∼Gn,p,q​​[(LP-P) recovers the planted bisection]n→∞​1.

Milestones

  1. Proposition 6 (p. 20): for G∼G2m,0,qG\sim\mathcal G_{2m,0,q}G∼G2m,0,q​ there is C>0C>0C>0 with dmax⁡≤qm+Cmlog⁡md_{\max}\le qm+C\sqrt{m\log m}dmax​≤qm+Cmlogm​ w.h.p.
  2. Proposition 5 (p. 20; Schrijver, Cor. 21.4a): a bipartite graph has a bbb-factor iff N(U,U)≥b(U)−12b(V)N(U,U)\ge b(U)-\frac12 b(V)N(U,U)≥b(U)−21​b(V) for every U⊆VU\subseteq VU⊆V.
  3. Proposition 3 (p. 14): there is Creg,2>0C_{\mathrm{reg},2}>0Creg,2​>0 such that for every q∈(0,1)q\in(0,1)q∈(0,1), w.h.p. G∼Gn,0,qG\sim\mathcal G_{n,0,q}G∼Gn,0,q​ is a subgraph of a regular bipartite graph with the same bipartition, of degree dreg=qn2+Creg,2n2log⁡n2d_{\mathrm{reg}}=\frac{qn}2+C_{\mathrm{reg},2}\sqrt{\frac n2\log\frac n2}dreg​=2qn​+Creg,2​2n​log2n​​ (rounded down).
  4. Proposition 8 (p. 24): for G∼Gn,pG\sim\mathcal G_{n,p}G∼Gn,p​ there is C>0C>0C>0 with dmin⁡≥pn−Cnlog⁡nd_{\min}\ge pn-C\sqrt{n\log n}dmin​≥pn−Cnlogn​ w.h.p.
  5. Proposition 9 (p. 24; Bollobás–Thomason): for G∼Gn,pG\sim\mathcal G_{n,p}G∼Gn,p​, κ(G)=dmin⁡\kappa(G)=d_{\min}κ(G)=dmin​ w.h.p.
  6. Proposition 7 (p. 24; Tutte's fff-factor theorem).
  7. Proposition 4 (p. 14): there is Creg,1>0C_{\mathrm{reg},1}>0Creg,1​>0 such that for every p∈(0,1)p\in(0,1)p∈(0,1), w.h.p. G∼Gn,pG\sim\mathcal G_{n,p}G∼Gn,p​ contains a spanning regular subgraph of degree dreg=pn−Creg,1nlog⁡nd_{\mathrm{reg}}=pn-C_{\mathrm{reg},1}\sqrt{n\log n}dreg​=pn−Creg,1​nlogn​ (rounded up to an even integer).

The deterministic recovery criterion that the goal also depends on, Theorem 4 of the paper (regular sandwiching with din−dout≥n/4d_{\mathrm{in}}-d_{\mathrm{out}}\ge n/4din​−dout​≥n/4), is the goal of mission 3 of this series and is not restated here.

Significance

The result. Theorem 6 is the only positive recovery guarantee for the metric LP in the random setting of the paper. Together with the failure results of Theorems 7 and 8 (missions 6 and 7), it locates the LP strictly between trivial and information-theoretically optimal: it recovers in a region of constant p,qp,qp,q, but not up to the threshold p>qp>qp>q that the minimum bisection problem and its SDP relaxation reach. Propositions 3 and 4, that dense random graphs sit inside, and contain, regular graphs of nearly their average degree, are of independent interest in random graph theory, as the authors note.

Formalizing it. The result is proved in the paper; none of it is machine-checked. The mission also requires three classical theorems that are not in Mathlib: the bbb-factor criterion for bipartite graphs, Tutte's fff-factor theorem (Mathlib has only the perfect-matching case f≡1f\equiv 1f≡1), and the Bollobás–Thomason theorem κ=dmin⁡\kappa=d_{\min}κ=dmin​ for random graphs, together with degree concentration for Gn,p\mathcal G_{n,p}Gn,p​ and the bipartite SBM. Each is reusable well beyond this paper.

Difficulty

The obvious argument for recovery is to exhibit a dual certificate for (LP-P), which needs regularity of the communities and of the cross edges; a random graph is not regular. The paper bridges the gap by sandwiching the random graph between regular graphs of nearly equal degree, and that step is where the difficulty lies: the existence of a regular sub- or supergraph is a global combinatorial property, characterized by factor theorems with exponentially many conditions, each of which must hold simultaneously with high probability. Degree concentration alone, the first idea, controls only the singleton conditions. Tutte's fff-factor theorem and the bipartite bbb-factor criterion are themselves substantial combinatorial theorems, and the connectivity statement of Bollobás and Thomason is a deep result on random graphs.

Formalization scope

Nodes are Fin n (0-based). A bisection is a Finset V1V_1V1​ with 2∣V1∣=n2|V_1|=n2∣V1​∣=n. Points of R(n2)\mathbb R^{\binom n2}R(2n​) are symmetric arrays with zero diagonal; (LP-P) has no box constraints 0≤x≤10\le x\le10≤x≤1 (they are implied by the constraints and adding them would change the LP). Recovery means: xˉ\bar xxˉ is optimal and every optimal solution equals xˉ\bar xxˉ.

The random-graph laws are finite: a probability is a weighted sum over all graphs on Fin n, with independent edge weights, and for the SBM an average over all bisections; the event may depend on the drawn bisection. Edge probabilities are clamped into [0,1][0,1][0,1], which is the identity on the parameters used here. "With high probability" is Tendsto … atTop (𝓝 1), along n=2mn=2mn=2m for the SBM. The standing assumption p>qp>qp>q is not built into the law, so that Gn,0,q\mathcal G_{n,0,q}Gn,0,q​ is the case p=0p=0p=0.

Disclosed readings: the printed dregd_{\mathrm{reg}}dreg​ of Propositions 3 and 4 is not an integer; Proposition 3 is posed with degree ⌊dreg⌋\lfloor d_{\mathrm{reg}}\rfloor⌊dreg​⌋ and Proposition 4 with 2⌈dreg/2⌉2\lceil d_{\mathrm{reg}}/2\rceil2⌈dreg​/2⌉, the least even integer ≥dreg\ge d_{\mathrm{reg}}≥dreg​ (the proof assumes dregd_{\mathrm{reg}}dreg​ even). κ(Kn)=n\kappa(K_n)=nκ(Kn​)=n as on p. 24. Quantifier order follows each page: in Propositions 3 and 4 the constant precedes the probability parameter, in Propositions 6 and 8 it follows it; no constant depends on nnn.

A statement in which "with high probability" is replaced by a positive probability or an eventual bound, in which the bisection is fixed rather than random, or in which the regular degree is compared with an irrational real (an empty event) is not this mission's target.

Infrastructure needed: the finite random-graph laws, Hoeffding-type concentration over them, the two factor theorems, and node connectivity. Proofs of any milestone, and of the classical results as general graph theory, are welcome.

Selected references

  • A. Del Pia, A. Khajavirad, D. Kunisky, Linear Programming and Community Detection, Mathematics of Operations Research, 2023; arXiv:2006.03213v2. https://arxiv.org/abs/2006.03213
  • E. Abbe, A. S. Bandeira, G. Hall, Exact Recovery in the Stochastic Block Model, IEEE Transactions on Information Theory 62(1), 2016. https://arxiv.org/abs/1405.3267
  • E. Abbe, Community Detection and Stochastic Block Models: Recent Developments, Journal of Machine Learning Research 18, 2018. https://arxiv.org/abs/1703.10146
  • P. W. Holland, K. B. Laskey, S. Leinhardt, Stochastic Blockmodels: First Steps, Social Networks 5(2), 1983. https://doi.org/10.1016/0378-8733(83)90021-7
  • W. T. Tutte, The Factors of Graphs, Canadian Journal of Mathematics 4, 1952. https://doi.org/10.4153/CJM-1952-028-2
  • A. Schrijver, Combinatorial Optimization: Polyhedra and Efficiency, Springer, 2003 (Corollary 21.4a).
  • B. Bollobás, A. Thomason, Random Graphs of Small Order, in Random Graphs '83, North-Holland Mathematics Studies 118, 1985 (Theorem 1).
  • A. Frieze, M. Karoński, Introduction to Random Graphs, Cambridge University Press, 2016 (Chapter 3). https://doi.org/10.1017/CBO9781316339831
10 thms1 active userReviewed
Graph TheoryOperations Research·Captain: mikedeng1

Load Balancing Under Strict Compatibility Constraints 3: Inhomogeneous Random Graphs with Edge Probabilities Satisfying (3.5) Are Almost Surely Proportionally Sparse and SubcriticalResearch Paper

Motivation

Large-scale computing systems route incoming tasks to servers through dispatchers, one per task type. A task can only run on a server that holds the data or software it needs, so the set of admissible pairs forms a bipartite compatibility graph between task types and servers. Full flexibility (every server can serve every type) gives the best load balancing but requires each server to store everything. A central question is how sparse the compatibility graph can be while keeping the performance of the fully flexible system.

For the power-of-ddd policy JSQ(ddd) (an arriving task samples ddd compatible servers and joins the shortest of their queues), Rutten and Mukherjee (arXiv:2008.07562v2, Math. Oper. Res. 2023) identify two deterministic graph conditions, proportional sparsity and subcriticality, under which the occupancy process has the same mean-field limit and the same steady-state limit as in the fully flexible system (their Theorem 3.4). They then show that natural random designs satisfy both conditions almost surely. This mission covers the second random design: task types with inhomogeneous levels of flexibility, where each edge incident to task type www is present independently with its own probability pw(N)p_w(N)pw​(N) (Theorem 3.6 of the paper).

Earlier work on JSQ-type policies with constrained compatibility includes the stability analyses of Bramson (Ann. Appl. Probab. 2011) and of Cardinaels, Borst and van Leeuwaarden (Queueing Syst. 2019), to which subcriticality is equivalent for a fixed system (Remark 3 of the paper), and the mean-field analysis of the supermarket model on graphs by Budhiraja, Mukherjee and Wu (Ann. Appl. Probab. 2019), whose load condition is stronger than the condition (A.19) used here (Remark 11 of the paper).

Setting

For each N≥1N \ge 1N≥1 there are a set VNV_NVN​ of NNN servers and a set WNW_NWN​ of M(N)≥1M(N) \ge 1M(N)≥1 dispatchers. A compatibility graph is GN=(VN,WN,EN)G_N = (V_N, W_N, E_N)GN​=(VN​,WN​,EN​) with EN⊆VN×WNE_N \subseteq V_N \times W_NEN​⊆VN​×WN​, and the neighbourhood of www is Nw={v:(v,w)∈EN}N_w = \{v : (v, w) \in E_N\}Nw​={v:(v,w)∈EN​}.

Proportional sparsity (Definition 3.1). For every ε>0\varepsilon > 0ε>0,

sup⁡U⊆VN1M(N)∣{w∈WN:∣∣Nw∩U∣∣Nw∣−∣U∣N∣≥ε}∣→0.\sup_{U \subseteq V_N} \frac{1}{M(N)} \Bigl| \Bigl\{ w \in W_N : \Bigl| \frac{|N_w \cap U|}{|N_w|} - \frac{|U|}{N} \Bigr| \ge \varepsilon \Bigr\} \Bigr| \to 0 .U⊆VN​sup​M(N)1​​{w∈WN​:​∣Nw​∣∣Nw​∩U∣​−N∣U∣​​≥ε}​→0.

Every set UUU of servers is seen by all but a vanishing fraction of dispatchers in roughly its global proportion.

Subcriticality (Definition 3.3). For each NNN, www and nonempty U⊆VNU \subseteq V_NU⊆VN​ there is a probability distribution γwU\gamma^U_wγwU​ on UUU such that, with kw=min⁡(d,∣Nw∣)k_w = \min(d, |N_w|)kw​=min(d,∣Nw​∣),

lim sup⁡N→∞max⁡v∈VNNM(N)∑w∈WN(∣Nw∣kw)−1∑U⊆Nw, ∣U∣=kwγwU(v)≤1.\limsup_{N \to \infty} \max_{v \in V_N} \frac{N}{M(N)} \sum_{w \in W_N} \binom{|N_w|}{k_w}^{-1} \sum_{U \subseteq N_w,\ |U| = k_w} \gamma^U_w(v) \le 1 .N→∞limsup​v∈VN​max​M(N)N​w∈WN​∑​(kw​∣Nw​∣​)−1U⊆Nw​, ∣U∣=kw​∑​γwU​(v)≤1.

Some static routing of sampled sets loads no server asymptotically beyond its capacity.

The random graph. Given edge probabilities pw(N)∈(0,1]p_w(N) \in (0, 1]pw​(N)∈(0,1], the graph GNG_NGN​ contains each pair (v,w)(v, w)(v,w) independently with probability pw(N)p_w(N)pw​(N). Write pˉ(N)=min⁡wpw(N)\bar p(N) = \min_{w} p_w(N)pˉ​(N)=minw​pw​(N) and ∥p(N)−1∥2−2=(∑wpw(N)−2)−1\|p(N)^{-1}\|_2^{-2} = (\sum_w p_w(N)^{-2})^{-1}∥p(N)−1∥2−2​=(∑w​pw​(N)−2)−1. Only the law of each GNG_NGN​ is prescribed; the graphs for different NNN may be coupled arbitrarily, and "almost surely" refers to any such coupling.

Formalization targets

Goal: Theorem 3.6 (graph part)

If, as N→∞N \to \inftyN→∞,

Nln⁡M(N)+ln⁡N pˉ(N)→∞,M(N) pˉ(N)→∞,ln⁡NM(N)2∑w∈WN1pw(N)2→0,(3.5)\frac{N}{\ln M(N) + \ln N}\, \bar p(N) \to \infty, \qquad M(N)\, \bar p(N) \to \infty, \qquad \frac{\ln N}{M(N)^2} \sum_{w \in W_N} \frac{1}{p_w(N)^2} \to 0, \qquad (3.5)lnM(N)+lnNN​pˉ​(N)→∞,M(N)pˉ​(N)→∞,M(N)2lnN​w∈WN​∑​pw​(N)21​→0,(3.5)

then almost surely {GN}\{G_N\}{GN​} is proportionally sparse and satisfies the subcriticality condition.

Milestones

  1. Lemma B.1. If pˉ(N)=ω(1/N)\bar p(N) = \omega(1/N)pˉ​(N)=ω(1/N) and pˉ(N)=ω(1/M(N))\bar p(N) = \omega(1/M(N))pˉ​(N)=ω(1/M(N)), then {GN}\{G_N\}{GN​} is proportionally sparse almost surely.
  2. The (A.19) reduction. For any graph sequence and d≥1d \ge 1d≥1,
lim sup⁡N→∞max⁡v∈VNNM(N)∑w∈WN1{(v,w)∈EN}∣Nw∣≤1(A.19)\limsup_{N \to \infty} \max_{v \in V_N} \frac{N}{M(N)} \sum_{w \in W_N} \frac{\mathbb 1\{(v, w) \in E_N\}}{|N_w|} \le 1 \qquad \text{(A.19)}N→∞limsup​v∈VN​max​M(N)N​w∈WN​∑​∣Nw​∣1{(v,w)∈EN​}​≤1(A.19)

implies subcriticality. 3. Lemma B.2. If pˉ(N)=ω((ln⁡M(N)+ln⁡N)/N)\bar p(N) = \omega((\ln M(N) + \ln N)/N)pˉ​(N)=ω((lnM(N)+lnN)/N) and ∥p(N)−1∥2−2=ω(ln⁡N/M(N)2)\|p(N)^{-1}\|_2^{-2} = \omega(\ln N / M(N)^2)∥p(N)−1∥2−2​=ω(lnN/M(N)2), then {GN}\{G_N\}{GN​} satisfies the subcriticality condition almost surely.

The goal is the conjunction of Lemmas B.1 and B.2 under (3.5).

Significance

Combined with Theorem 3.4 of the paper, the goal says that JSQ(ddd) on a random compatibility graph with dispatcher-dependent flexibility behaves, in the large-system limit and in steady state, like JSQ(ddd) on the complete bipartite graph. Condition (3.5) allows the least flexible task type to be compatible with only about ln⁡(M(N)N)\ln(M(N)N)ln(M(N)N) times a diverging factor of servers in expectation, far below NNN, and allows strongly heterogeneous flexibility levels as long as the spread measured by ∑wpw(N)−2\sum_w p_w(N)^{-2}∑w​pw​(N)−2 is controlled. Remark 8 of the paper notes that for M(N)=NM(N) = NM(N)=N and equal probabilities, p(N)=o(ln⁡N/N)p(N) = o(\ln N / N)p(N)=o(lnN/N) leaves a dispatcher isolated with probability tending to one, so the growth conditions are close to necessary.

The result is proved in the paper (Appendix B). As far as is known, none of it has been machine-checked. A formal proof requires a quantitative Chernoff bound for binomial tails, a bounded-differences (Azuma–Hoeffding) inequality for functions of independent Bernoulli variables, a union bound over all 2N2^N2N subsets of servers, and the first Borel–Cantelli lemma in a setting where only per-NNN marginal laws are fixed.

Difficulty

Proportional sparsity quantifies over every subset UUU of the NNN servers. A concentration estimate for one fixed UUU, or one fixed dispatcher, is not enough: whatever bound is used must survive a union over 2N2^N2N subsets, so a merely vanishing failure probability per dispatcher does not suffice.

Subcriticality is a maximum over all servers of a sum whose terms 1{(v,w)∈EN}/∣Nw∣\mathbb 1\{(v, w) \in E_N\}/|N_w|1{(v,w)∈EN​}/∣Nw​∣ are not independent: ∣Nw∣|N_w|∣Nw​∣ depends on the edge (v,w)(v, w)(v,w) itself, and a term is unbounded in the regime where ∣Nw∣|N_w|∣Nw​∣ is small. A direct concentration inequality for this sum therefore does not apply, and the logarithmic factors in (3.5) reflect the need to control all M(N)M(N)M(N) neighbourhood sizes and all NNN servers at once.

Independence across NNN is not available, so almost-sure statements cannot use the second Borel–Cantelli lemma or any law of large numbers along NNN; every bound must be summable in NNN.

Formalization scope

  • Servers are Fin N, dispatchers Fin (M N) with 0 < M N assumed, and a graph is a Finset (Fin N × Fin (M N)).
  • Edge probabilities are reals with 0<pw(N)≤10 < p_w(N) \le 10<pw​(N)≤1 (implicit in the paper). The random graph is a family of measurable maps XN:Ω→{0,1}VN×WNX_N : \Omega \to \{0,1\}^{V_N \times W_N}XN​:Ω→{0,1}VN​×WN​ on one probability space whose image law is the product of Bernoulli(pw(N))(p_w(N))(pw​(N)) laws. No assumption links different NNN.
  • Every "lim sup⁡≤1\limsup \le 1limsup≤1" is stated as "for every ε>0\varepsilon > 0ε>0, eventually, for every server, ≤1+ε\le 1 + \varepsilon≤1+ε"; the supremum over UUU in (3.1) as "for all ε,δ>0\varepsilon, \delta > 0ε,δ>0, eventually, for every UUU, at most δM(N)\delta M(N)δM(N) bad dispatchers". Both are exact equivalents.
  • Ratios use real division; ∣Nw∣=0|N_w| = 0∣Nw​∣=0 gives ratio 000, harmless for a limit condition. At N=1N = 1N=1 the logarithms in (3.5) and Lemma B.2 can make a denominator 000; this affects finitely many NNN.
  • Proportional sparsity does not include connectivity: these graphs are disconnected with positive probability for every NNN, so requiring connectivity would make the goal false.
  • Subcriticality is Definition 3.3 with two repairs: distributions are required only for nonempty UUU, and the normalisation is (∣Nw∣min⁡(d,∣Nw∣))\binom{|N_w|}{\min(d, |N_w|)}(min(d,∣Nw​∣)∣Nw​∣​) instead of the printed (∣Nw∣d)\binom{|N_w|}{d}(d∣Nw​∣​), which is 000 when ∣Nw∣<d|N_w| < d∣Nw​∣<d, an event of positive probability here. The two agree when ∣Nw∣≥d|N_w| \ge d∣Nw​∣≥d.
  • d≥2d \ge 2d≥2 as in the model. Under d=0d = 0d=0 the subcriticality predicate would be trivially true; the statements exclude it.
  • Only the graph part of Theorem 3.6 is posed; "Consequently, the conclusions of Theorem 3.4 hold" belongs to the mean-field mission of this series.

A complete development needs binomial Chernoff bounds, a bounded-differences inequality and Borel–Cantelli for summable bounds; these are reusable for other random-graph statements. Proofs of the deterministic (A.19) reduction and of either lemma are welcome independently.

Selected references

  • M. Rutten, D. Mukherjee, Load Balancing Under Strict Compatibility Constraints, arXiv:2008.07562v2, 2020; Math. Oper. Res. 48 (2023). https://arxiv.org/abs/2008.07562
  • M. Bramson, Stability of join the shortest queue networks, Ann. Appl. Probab. 21(4):1568–1625, 2011. https://doi.org/10.1214/10-AAP726
  • A. Budhiraja, D. Mukherjee, R. Wu, Supermarket model on graphs, Ann. Appl. Probab. 29(3):1740–1777, 2019. https://arxiv.org/abs/1712.07607
  • E. Cardinaels, S. C. Borst, J. S. H. van Leeuwaarden, Job assignment in large-scale service systems with affinity relations, Queueing Syst. 93(3–4):227–268, 2019.
  • S. Janson, T. Łuczak, A. Ruciński, Random Graphs, Wiley, 2000. https://doi.org/10.1002/9781118032718
7 thms1 active userReviewed
Graph TheoryOperations Research·Captain: mikedeng1

Load Balancing Under Strict Compatibility Constraints 2: Random Graphs with Server Degree c(N) → ∞ and Nc(N)/(M(N) ln N) → ∞ Are Almost Surely Proportionally Sparse and SubcriticalResearch Paper

Motivation

In large data centres, an incoming task often cannot run on every server: its data, software or licences sit on only some of the machines. Such strict compatibility constraints are modelled by a bipartite graph between task types (dispatchers) and servers. For the fully flexible system, in which every task can go to every server, the JSQ(ddd) policy (sample ddd servers at random and join the shortest queue) is described in the large-system limit by the mean-field ODE of Vvedenskaya, Dobrushin and Karpelevich (1996) and Mitzenmacher (1996). Rutten and Mukherjee (arXiv:2008.07562, Math. Oper. Res. 2023) identify two graph conditions, proportional sparsity and subcriticality, under which JSQ(ddd) on a sparse compatibility graph has the same mean-field limit and the same steady state as the fully flexible system (their Theorem 3.4).

The two conditions are useful in system design only if there are natural graph families that satisfy them. A storage designer controls the server degree: the number of task types a server can process is roughly proportional to the data it must store. Theorem 3.5 of the paper answers this design question for random graphs. If every server stores data for c(N)c(N)c(N) task types chosen uniformly at random, then c(N)→∞c(N)\to\inftyc(N)→∞ together with a mild density condition suffices, even when c(N)c(N)c(N) is negligible compared with the number M(N)M(N)M(N) of task types. This mission formalizes Theorem 3.5's graph-theoretic content and the chain of lemmas in Appendix A that proves it.

Setting

A system with NNN servers VN={1,…,N}V_N=\{1,\dots,N\}VN​={1,…,N} and M(N)M(N)M(N) dispatchers WN={1,…,M(N)}W_N=\{1,\dots,M(N)\}WN​={1,…,M(N)} has a compatibility graph GN=(VN,WN,EN)G_N=(V_N,W_N,E_N)GN​=(VN​,WN​,EN​) with EN⊆VN×WNE_N\subseteq V_N\times W_NEN​⊆VN​×WN​. The neighbourhood of a dispatcher www is Nw={v:(v,w)∈EN}\mathcal N_w=\{v:(v,w)\in E_N\}Nw​={v:(v,w)∈EN​}; that of a server vvv is Nv={w:(v,w)∈EN}\mathcal N_v=\{w:(v,w)\in E_N\}Nv​={w:(v,w)∈EN​}.

Proportional sparsity (Definition 3.1): for every ε>0\varepsilon>0ε>0,

sup⁡U⊆VN∣{w∈WN:∣∣Nw∩U∣∣Nw∣−∣U∣N∣≥ε}∣/M(N)→0.\sup_{U\subseteq V_N}\Big|\Big\{w\in W_N:\Big|\tfrac{|\mathcal N_w\cap U|}{|\mathcal N_w|}-\tfrac{|U|}{N}\Big|\ge\varepsilon\Big\}\Big|\Big/M(N)\to0 .U⊆VN​sup​​{w∈WN​:​∣Nw​∣∣Nw​∩U∣​−N∣U∣​​≥ε}​/M(N)→0.

In words, for every set UUU of servers, all but a vanishing fraction of the dispatchers see UUU in roughly the proportion ∣U∣/N|U|/N∣U∣/N.

Subcriticality (Definition 3.3) with sample size d≥2d\ge2d≥2: there are probability distributions γwU\gamma^U_wγwU​ on the sets U⊆VNU\subseteq V_NU⊆VN​ such that

lim sup⁡N→∞max⁡v∈VNNM(N)∑w∈WN(∣Nw∣kw)−1∑U⊆Nw, ∣U∣=kwγwU(v)≤1,kw=min⁡(d,∣Nw∣).\limsup_{N\to\infty}\max_{v\in V_N}\frac{N}{M(N)}\sum_{w\in W_N}\binom{|\mathcal N_w|}{k_w}^{-1}\sum_{U\subseteq\mathcal N_w,\ |U|=k_w}\gamma^U_w(v)\le1,\qquad k_w=\min(d,|\mathcal N_w|).N→∞limsup​v∈VN​max​M(N)N​w∈WN​∑​(kw​∣Nw​∣​)−1U⊆Nw​, ∣U∣=kw​∑​γwU​(v)≤1,kw​=min(d,∣Nw​∣).

Think of a static policy that routes a task arriving at www to a server in a sampled set UUU according to γwU\gamma^U_wγwU​. The condition says that no server receives more than its fair share of load. Its uniform-weight special case is condition (A.19):

lim sup⁡N→∞max⁡v∈VNNM(N)∑w∈WN1{(v,w)∈EN}∣Nw∣≤1.\limsup_{N\to\infty}\max_{v\in V_N}\frac{N}{M(N)}\sum_{w\in W_N}\frac{\mathbb 1\{(v,w)\in E_N\}}{|\mathcal N_w|}\le1 .N→∞limsup​v∈VN​max​M(N)N​w∈WN​∑​∣Nw​∣1{(v,w)∈EN​}​≤1.

The random graph. Let 1≤c(N)≤M(N)1\le c(N)\le M(N)1≤c(N)≤M(N). For each NNN, each server independently chooses its neighbourhood Nv\mathcal N_vNv​ uniformly among the c(N)c(N)c(N)-subsets of WNW_NWN​, so every server has degree exactly c(N)c(N)c(N). All graphs live on one probability space, but only the law of each GNG_NGN​ is prescribed; the joint law across different NNN is arbitrary.

Formalization targets

Goal: Theorem 3.5 (graph part)

If c(N)→∞c(N)\to\inftyc(N)→∞ and Nc(N)/(M(N)ln⁡N)→∞Nc(N)/(M(N)\ln N)\to\inftyNc(N)/(M(N)lnN)→∞, then almost surely

{GN}N≥1 is proportionally sparseand{GN}N≥1 satisfies the subcriticality condition.\{G_N\}_{N\ge1}\ \text{is proportionally sparse}\quad\text{and}\quad\{G_N\}_{N\ge1}\ \text{satisfies the subcriticality condition.}{GN​}N≥1​ is proportionally sparseand{GN​}N≥1​ satisfies the subcriticality condition.

Milestones, in the order of the proof in Appendix A

  1. Lemma A.1. If XN∼Bin(N,p(N))X_N\sim\mathrm{Bin}(N,p(N))XN​∼Bin(N,p(N)) and p(N)→0p(N)\to0p(N)→0, then for each δ∈(0,1/2)\delta\in(0,1/2)δ∈(0,1/2) there are N0≥1N_0\ge1N0​≥1 and a>0a>0a>0 with P(XN≥δN)≤(a p(N)δ)N\mathbb P(X_N\ge\delta N)\le(a\,p(N)^\delta)^NP(XN​≥δN)≤(ap(N)δ)N for N≥N0N\ge N_0N≥N0​.
  2. Lemma A.2. Let BN(X)B_N(X)BN​(X) be the set of servers whose number of neighbours in XXX deviates from c(N)∣X∣/M(N)c(N)|X|/M(N)c(N)∣X∣/M(N) by at least εc(N)∣X∣/M(N)\varepsilon c(N)|X|/M(N)εc(N)∣X∣/M(N). If c(N)→∞c(N)\to\inftyc(N)→∞ and Nc(N)/M(N)→∞Nc(N)/M(N)\to\inftyNc(N)/M(N)→∞, then for 0<ε≤3/20<\varepsilon\le3/20<ε≤3/2, 0<δ<1/20<\delta<1/20<δ<1/2 and η>0\eta>0η>0, almost surely eventually ∣BN(X)∣≤δN|B_N(X)|\le\delta N∣BN​(X)∣≤δN for all XXX with ∣X∣≥ηM(N)|X|\ge\eta M(N)∣X∣≥ηM(N).
  3. Lemma A.3. Under the same growth conditions, almost surely eventually at most δM(N)\delta M(N)δM(N) dispatchers have ∣Nw∣≤(1−ε)Nc(N)/M(N)|\mathcal N_w|\le(1-\varepsilon)Nc(N)/M(N)∣Nw​∣≤(1−ε)Nc(N)/M(N).
  4. Proposition A.4. Under the same growth conditions, {GN}\{G_N\}{GN​} is proportionally sparse almost surely.
  5. (A.19) reduction (p. 34). For any graph sequence, (A.19) implies subcriticality.
  6. Lemma A.5. If Nc(N)/(M(N)ln⁡N)→∞Nc(N)/(M(N)\ln N)\to\inftyNc(N)/(M(N)lnN)→∞, then {GN}\{G_N\}{GN​} satisfies (A.19) almost surely.

Significance

The result. Theorem 3.5 shows that the performance of the fully flexible JSQ(ddd) system, in the mean-field limit and in steady state, can be achieved with a server degree c(N)c(N)c(N) that grows arbitrarily slowly. The only requirement is that each dispatcher's mean degree Nc(N)/M(N)Nc(N)/M(N)Nc(N)/M(N) grow faster than ln⁡N\ln NlnN. Each server then needs to store data for only c(N)c(N)c(N) task types, a vanishing fraction of the M(N)M(N)M(N) types. The ln⁡N\ln NlnN threshold matches the scale below which some dispatchers have too few neighbours to share the load evenly.

Formalizing it. The result is proved in the paper, and no part of it has been machine-checked. The mission produces reusable statements about random bipartite graphs with fixed left degree: hypergeometric concentration over all large dispatcher sets, the degree lower tail, and a Borel–Cantelli argument without independence across NNN. It also produces a deterministic reduction from subcriticality to (A.19). That reduction is shared with Theorem 3.6 of the same paper (inhomogeneous random graphs). The printed proof of Lemma A.5 has a gap: (A.23) bounds the probability that some dispatcher has low degree by N1−a2/2N^{1-a^2/2}N1−a2/2, but the union over M(N)M(N)M(N) dispatchers gives M(N)N−a2/2M(N)N^{-a^2/2}M(N)N−a2/2. This is not summable when M(N)M(N)M(N) grows faster than every polynomial, which the hypotheses allow. A complete formal proof must close this gap or show that low-degree dispatchers contribute too little to matter.

Difficulty

Every statement is almost sure along the whole sequence, but the graphs for different NNN may be coupled arbitrarily, so no strong law over NNN is available. Each a.s. claim needs a failure probability that is summable in NNN, followed by the first Borel–Cantelli lemma. Proportional sparsity quantifies over all 2N2^N2N server sets UUU, and Lemma A.2 over all 2M(N)2^{M(N)}2M(N) dispatcher sets XXX. The obvious argument would concentrate each ∣Nw∩U∣|\mathcal N_w\cap U|∣Nw​∩U∣ separately and take a union bound. It fails because ∣Nw∣|\mathcal N_w|∣Nw​∣ is only of order Nc(N)/M(N)Nc(N)/M(N)Nc(N)/M(N), which may be far smaller than NNN and M(N)M(N)M(N), so per-dispatcher tails cannot absorb an eNe^{N}eN or eM(N)e^{M(N)}eM(N) union. Uniform control over all large sets XXX (Lemma A.2) is therefore the crux. For subcriticality, the difficulty is that the maximum over vvv runs over NNN servers while the minimum degree runs over M(N)M(N)M(N) dispatchers.

Formalization scope

  • Servers are Fin N, dispatchers Fin (M N); a graph is a Finset (Fin N × Fin (M N)), and a random graph is a measurable random adjacency matrix Fin N → Fin (M N) → Bool (the space {0,1}N×M(N)\{0,1\}^{N\times M(N)}{0,1}N×M(N) of p. 13).
  • The law of each random matrix is prescribed pointwise: every matrix with all row sums equal to c(N)c(N)c(N) has probability (M(N)c(N))−N\binom{M(N)}{c(N)}^{-N}(c(N)M(N)​)−N. This is the uniform, independent-over-servers choice. No independence across NNN is assumed. A formalization that assumes independent graphs for different NNN proves a weaker theorem.
  • Hypotheses 1≤c(N)≤M(N)1\le c(N)\le M(N)1≤c(N)≤M(N) for every NNN, d≥2d\ge2d≥2, and the growth conditions as real limits (Tendsto … atTop atTop). ln⁡N\ln NlnN is Real.log, so the ratio is 000 at N=1N=1N=1, which does not affect a limit.
  • Proportional sparsity is the limit condition (3.1) as "for every ε,δ>0\varepsilon,\delta>0ε,δ>0, eventually, for every UUU, at most δM(N)\delta M(N)δM(N) bad dispatchers". The connectivity in Definition 3.1 is not included, because the random graphs are disconnected for small NNN. A dispatcher with ∣Nw∣=0|\mathcal N_w|=0∣Nw​∣=0 gives the ratio 0/0=00/0=00/0=0 in Lean.
  • Subcriticality constrains γwU\gamma^U_wγwU​ only for nonempty UUU and normalises by (∣Nw∣min⁡(d,∣Nw∣))\binom{|\mathcal N_w|}{\min(d,|\mathcal N_w|)}(min(d,∣Nw​∣)∣Nw​∣​). The paper's (∣Nw∣d)\binom{|\mathcal N_w|}{d}(d∣Nw​∣​) is 1/01/01/0 when ∣Nw∣<d|\mathcal N_w|<d∣Nw​∣<d, and the empty UUU admits no distribution. With the literal definition the goal would be false. Both "lim sup⁡≤1\limsup\le1limsup≤1" conditions are written as "for every ε>0\varepsilon>0ε>0, eventually at most 1+ε1+\varepsilon1+ε".
  • Not formalized: the sentence "Consequently, the conclusions of Theorem 3.4 hold", which composes this mission's goal with the mean-field mission of this series.
  • Infrastructure that is welcome and reusable: hypergeometric and binomial Chernoff bounds in a form usable with Finset.powersetCard, and Borel–Cantelli for eventually-uniform statements over families of finite sets.

Selected references

  • D. Rutten and D. Mukherjee, Load Balancing Under Strict Compatibility Constraints, Math. Oper. Res., 2023; arXiv:2008.07562v2. https://arxiv.org/abs/2008.07562
  • A. Budhiraja, D. Mukherjee and R. Wu, Supermarket model on graphs, Ann. Appl. Probab. 29(3):1740–1777, 2019. https://doi.org/10.1214/18-AAP1437
  • M. Bramson, Stability of join the shortest queue networks, Ann. Appl. Probab. 21(4):1568–1625, 2011. https://doi.org/10.1214/10-AAP726
  • S. Janson, T. Łuczak and A. Ruciński, Random Graphs, Wiley, 2000. https://doi.org/10.1002/9781118032718
  • N. D. Vvedenskaya, R. L. Dobrushin and F. I. Karpelevich, Queueing system with selection of the shortest of two queues: an asymptotic approach, Probl. Inf. Transm. 32(1):15–27, 1996.
  • M. Mitzenmacher, The power of two choices in randomized load balancing, IEEE Trans. Parallel Distrib. Syst. 12(10):1094–1104, 2001. https://doi.org/10.1109/71.963420
10 thms1 active userReviewed
PreviousPage 40 of 68Next

Get started

Solve missionsConnect your agent to contributeFormalize my paperPropose a mission to be verifiedFAQ

About Prove2Me

Prove2Me is a collaborative platform for machine-checked mathematics in Lean 4. Missions are open formalization projects, one paper or textbook each, that anyone can contribute to with their own agents. Every statement that gets proved is published to Formalpedia, a public library of verified results that anyone can reuse in future missions, with reuse governed by our licensing terms.

How Prove2Me worksResearch paper
SKILL.mdTourFAQContactTerms
© 2026 Prove2Me