Proportional Volume Sampling and Approximation Algorithms for A-Optimal Design: Sampling with an α-Approximate (d−1, d)-Wise Independent Measure Gives an α-ApproximationResearch Paper
Motivation
In linear regression with a budget, an experimenter holds candidate experiments, each described by a vector , and may run only of them. If the experiments in are run and the unknown parameter is estimated by least squares, the covariance of the estimate is proportional to . A-optimal design chooses to minimise the trace of that matrix, which is the mean squared error of the estimator. The criterion is one of the classical optimality criteria of experimental design (Pukelsheim, Optimal Design of Experiments, SIAM 2006) and appears in feature selection, sensor placement and matrix sparsification.
Selecting the best -set exactly is NP-hard (the paper proves this in its Theorem 1.7), so the question is approximation. Avron and Boutsidis (SIAM J. Matrix Anal. Appl. 2013) connected A-optimal design to volume sampling (Deshpande–Rademacher, FOCS 2010), which samples a set with probability proportional to , and obtained the approximation ratio . Nikolov, Singh and Tantipongpipat (arXiv:1802.08318; Mathematics of Operations Research, 2022) reweight volume sampling by a second measure that can depend on a solution of the natural convex relaxation. Their first result, Theorem 1.3, shows that the approximation ratio of this scheme is controlled by a single combinatorial property of , approximate -wise independence. Their later theorems obtain ratios (when ) and (when is large) by exhibiting such measures.
Setting
Let be a real matrix and . For , is the submatrix of the columns indexed by , so . The A-optimal design problem is
with the value for sets whose Gram matrix is singular.
The convex relaxation (1)–(3) replaces the indicator of by :
Write for the matrix with columns , , so that , and .
Let () be the subsets of of size exactly (at most) , and a probability measure on them. Proportional volume sampling with measure (Definition 1.1) is the distribution with
The measure is -approximate -wise independent with respect to a feasible (Definition 1.2) if for all with and ,
For a square matrix with eigenvalues , is the -th elementary symmetric polynomial ; .
Algorithm 1 (p. 8) solves the relaxation for , samples , and pads with arbitrary indices to size if it is smaller.
Formalization targets
Goal: Theorem 1.3
If is an optimal solution of (1)–(3) and is -approximate -wise independent with respect to , then the output of Algorithm 1 satisfies, for every -set ,
so the algorithm is an -approximation in expectation, and some -set has objective at most (the integrality gap is at most ). The constant is a parameter: the goal fixes no value of it, so later constructions of better measures plug into it unchanged.
Milestones
- Lemma 2.1: for .
- Identity (4): .
- Identity (6): for a matrix of rank .
- The expected objective under is at most (p. 9).
- (p. 9).
- Inequality (5): .
- (p. 7).
- Padding a set with further vectors does not increase (p. 8).
Significance
Theorem 1.3 turns an approximation question about a continuous, spectral objective into a question about the marginals of a distribution on sets: any measure with approximately independent - and -marginals relative to the relaxation's solution yields an approximation algorithm with the same ratio, and simultaneously bounds the integrality gap of the relaxation. The paper's Theorems 1.4 and 1.5 are proved by applying it to hard-core distributions, giving a -approximation for and a -approximation for .
The result is proved in the paper; nothing here is open. To our knowledge none of it has a machine-checked proof. The formalization produces, beyond the theorem itself, a reusable linear-algebra layer that the pinned Mathlib lacks: the Cauchy–Binet formula for Gram matrices, its extension (4) to all elementary symmetric functions of the spectrum, and the identity . These are standard tools for determinantal point processes and volume sampling, and a later formalization of the hard-core constructions (Theorems 1.4, 1.5) would build directly on this mission's goal.
Difficulty
The reduction itself is a short calculation once both sides of (5) are written as ratios of sums over - and -subsets; the difficulty is in the linear algebra that produces those ratios. Identity (4) needs Cauchy–Binet for non-square matrices together with the relation between the coefficients of the characteristic polynomial, its roots and sums of principal minors; none of this is in Mathlib in the required form, and the eigenvalue side requires that the characteristic polynomial of a real symmetric matrix splits over . Applying (4) to every separately and exchanging the order of summation is bookkeeping between index sets of different sizes, which is where informal proofs are quietest. The expectation also has to be handled honestly at sets with singular : they are never sampled, but the paper's displayed equality on p. 9 counts them, so the first display holds only as an inequality.
Formalization scope
Everything is over , with index sets Finset (Fin n) and a Matrix (Fin d) (Fin n) ℝ. A probability measure on is a nonnegative function on subsets of with total mass vanishing on sets larger than ; this covers as a special case. is the elementary symmetric function of the real roots of the characteristic polynomial, applied only to symmetric matrices. Lean's matrix inverse is on singular matrices, so every comparison of objectives assumes positive definiteness of the inverted matrix, and OPT is expressed by quantifying over -sets with positive definite Gram matrix rather than by an infimum.
The statements add the following, each disclosed in the item's Formalization Note: (for the natural-number subtraction ); a positive normaliser , which Definition 1.1 presupposes; positive definiteness of , which is what a finite CP means; Definition 1.2 in cross-multiplied form; and "" in place of the printed "" in the first display of p. 9. Theorem 1.3 assumes a suitable only for the optimal , as the remark after the theorem says; this implies the printed form. The padding of Algorithm 1 is an arbitrary map with and on the support of .
The goal is not inequality (5) alone and is not defined as a sum of minors, and the expectation is not defined as the ratio of p. 9; any of those would make milestones true by definition. A sorry-free sanity file shows that all hypotheses of the goal are simultaneously satisfiable (, , , the point mass at , ).
Contributions are welcome at every milestone; the Cauchy–Binet layer (milestones 2 and 3) is reusable well beyond this mission and is the natural place to start.
Selected references
- A. Nikolov, M. Singh, U. Tantipongpipat, Proportional Volume Sampling and Approximation Algorithms for A-Optimal Design, arXiv:1802.08318v5, 2018; Mathematics of Operations Research, 2022. https://arxiv.org/abs/1802.08318
- H. Avron, C. Boutsidis, Faster Subset Selection for Matrices and Applications, SIAM Journal on Matrix Analysis and Applications 34(4):1464–1499, 2013. https://arxiv.org/abs/1201.0127
- A. Deshpande, L. Rademacher, Efficient Volume Sampling for Row/Column Subset Selection, FOCS 2010, 329–338. https://doi.org/10.1109/FOCS.2010.38
- Z. Mariet, S. Sra, Elementary Symmetric Polynomials for Optimal Experimental Design, NeurIPS 2017. https://arxiv.org/abs/1705.09677
- F. Pukelsheim, Optimal Design of Experiments, SIAM Classics in Applied Mathematics, 2006. https://doi.org/10.1137/1.9780898719109