Secretaries with Advice 3: A Closed Form for the Backward-Induction Dual Solution of the Secretary-with-Samples LPResearch Paper
Motivation
In the secretary problem candidates arrive one at a time in uniformly random order; at each arrival the decision maker sees only how the current candidate compares with the earlier ones and must accept or reject irrevocably, aiming to accept the best candidate overall. Dütting, Lattanzi, Paes Leme and Vassilvitskii, Secretaries with Advice (arXiv:2011.06726), study a family of variants in which each candidate also carries a signal about its quality, and show that for many of them the optimal stopping rule is the solution of a finite factor-revealing linear program whose dual can be solved greedily by backward induction.
One of their main examples is the secretary problem with samples: before the candidates arrive, further elements of the same random order are revealed as a sample, and a candidate's signal is the number of sample elements smaller than it. This model interpolates between the classical problem (, optimal success probability tending to ) and, as , the full-information problem of Gilbert and Mosteller (1966). The paper's Theorem 17 gives the optimal dual solution of this LP in closed form.
Setting
Fix natural numbers (secretaries) and (samples). Rounds are ; signals are . The dual LP of §5 has the coefficients
with for , and reads
The greedy solution (equation (8) of the paper) sets and, going backwards,
The un-clipped recursion is the same backward induction with the removed. In Lean, the generic recursion (8) is greedyDual (and unclippedDual) over an arbitrary finite signal type, and the samples instance is samplesGreedy n k i s (resp. samplesUnclipped) with s : Fin (k + 1). Products with equal .
Formalization targets
Goal: Theorem 17
For every and : , and for
Milestones (the steps of the paper's proof)
- Reduction (p. 17): if has the closed form for all , then , with an explicit double sum.
- Two forms of (pp. 17–18): the double sum collapses to .
- Key identity (p. 18): .
- Base case (p. 18) and 5. inductive step (Appendix B, pp. 26–28) of the key identity.
- Un-clipped closed form (pp. 17–19): equals the expression inside the max, for all .
Significance
Theorem 16 of the paper shows that the greedy dual solution is optimal and that the optimal policy for secretaries with samples is a threshold policy in the signal: accept a best-so-far candidate in round iff its signal is large enough. Theorem 17 makes this solution explicit, so the optimal thresholds can be read off from the sign of a closed-form expression rather than computed by running the backward recursion, and it is the discrete precursor of the paper's Corollary 18, which recovers the Gilbert–Mosteller full-information thresholds in the limit .
The identities are proved in the paper by explicit manipulation of finite sums and products of binomials. No machine-checked version exists. A formal proof checks the long index computations of the proof and of Appendix B, and closes a gap: the paper proves the identity for the un-clipped recursion only ("for the proof we can ignore the ") and does not argue why the clipped greedy solution has the same form. The goal as stated here, with the maximum, is the statement the paper asserts.
Difficulty
The identities are rational-function identities in , and the round indices, with sums whose ranges depend on the summation variables and products whose ranges may be empty. The natural first idea, induction on directly on the formula, requires the key identity for , a double sum over and of products of rising factorials; the paper handles it by a second backward induction over , and both the base case and the step re-index sums with shifted bounds and boundary terms such as and .
The passage from the un-clipped to the clipped recursion is a separate difficulty: once is clipped to at a later round, the sums used at earlier rounds differ from those of the un-clipped recursion, and the closed form survives only because the clipped entries do not affect the relevant sums. That argument is not in the paper.
Formalization scope
- Rounds are natural numbers (1-based, as printed); signals are
Fin (k + 1)in the recursion and naturals in the formulas. All arithmetic is in ; naturals are cast before , and the natural differences , , appear only where the ranges keep them non-truncated. - Products and sums are over
Finset.Icc a b, empty when , matching the paper's convention. - The recursion is defined by well-founded recursion on and sums over all later rounds and all signals; for encodes the paper's .
- The goal keeps the . Stating instead the identity for the un-clipped recursion would be a different (weaker-content) theorem; it is milestone 6.
- No probability model appears: the coefficients are taken as given by their formulas on p. 15. Their probabilistic derivation, the optimality of the greedy solution (Theorem 16), the running time of the LP algorithm, the "natural interpretation" paragraph, Corollary 18 and the plots are out of scope.
- The generic recursion
greedyDualis reusable for any instance of the paper's factor-revealing LP (the binary-classifier model of §6 included). General lemmas about sums of rising factorials (ascPochhammer) and binomial re-indexing developed for the milestones are welcome contributions.
Selected references
- P. Dütting, S. Lattanzi, R. Paes Leme, S. Vassilvitskii, Secretaries with Advice, arXiv:2011.06726v1, 2020; Mathematics of Operations Research, 2024. https://arxiv.org/abs/2011.06726
- J. P. Gilbert, F. Mosteller, Recognizing the Maximum of a Sequence, Journal of the American Statistical Association 61(313), 1966. https://doi.org/10.1080/01621459.1966.10502008
- T. S. Ferguson, Who Solved the Secretary Problem?, Statistical Science 4(3), 1989. https://doi.org/10.1214/ss/1177012493