Optimal No-regret Learning in Repeated First-price Auctions 2: The ML-IS-UCB Bidding Policy Achieves Expected Regret at Most C√T log³ T for Every Oblivious Private-Value SequenceResearch Paper
Learning to bid in first-price auctions
Display advertising has been moving from second-price to first-price auctions: several exchanges began rolling them out in 2017, and Google Ad Manager completed its move at the end of 2019 (Davies 2019). In a second-price auction bidding one's value is a dominant strategy; in a first-price auction the optimal bid depends on the distribution of the competing bids, which a bidder does not know and must learn from repeated play. Most of the learning literature on auctions takes the seller's side of the second-price auction (reserve prices, e.g. Cesa-Bianchi, Gentile, Mansour 2015); on the bidder's side of the first-price auction, the main prior approach is the cross-learning framework of Balseiro et al. 2019.
Han, Zhou and Weissman (arXiv:2003.09795) study the bidder's problem with the feedback that ad exchanges actually provide, where only the winning price is announced, and with no assumption on the distribution of the competing bids. Their two upper bounds are regret: Theorem 1 for iid private values and Theorem 2, the subject of this mission, for values chosen by an oblivious adversary. A matching lower bound holds even under full information (Theorem 9).
Setting
A single bidder plays rounds. In round she observes a private value , bids , and wins if , where is the highest other bid (HOB). Her payoff is (1)
The HOBs are iid with an unknown cumulative distribution function on ; may be discrete, continuous or mixed. Feedback is censored: after round only is revealed, so is observed only when the bidder loses. The expected reward is , and the regret of a policy against the value sequence is (3)
the shortfall against an oracle that knows . A policy bids in round as a (possibly randomized) function of and of the feedback of rounds . An oblivious adversary fixes the whole value sequence in advance, independently of the bids and the HOBs.
The policy analysed is ML-IS-UCB (Algorithm 4). It restricts bids to the grid , , with ; it splits time into levels after an initialization phase of zero bids; and at each level it computes a width , an interval-splitting reward estimate and a candidate set by the displays (12)–(16).
Formalization targets
Goal: Theorem 2
There is an absolute constant such that for every some policy satisfies, for every value sequence and every on ,
The constant is unspecified, so the goal survives any improvement of the constants below.
The explicit bound: Theorem 7
For every , the ML-IS-UCB policy satisfies
Milestones
In the order of the proof (Section 4.3 and Appendix C):
- Lemma 7: with probability at least , for all and , and is bounded by (18) in terms of the true interval probabilities .
- Lemma 8: on that event the best grid bid is never eliminated, and a round assigned to level has regret at most against the grid.
- Display (19): the grid regret is at most .
- Lemma 16: a deterministic counting bound for the sums .
- Lemma 9: .
- Theorem 7.
A supporting item, not a milestone, states the quantization step for the grid of Algorithm 4.
Significance
The result shows that adversarial values cost at most a logarithmic factor relative to iid values in repeated first-price auctions with censored feedback, and that no regularity of is needed. ML-IS-UCB is also an algorithmic template: its widths are computed before the data of the current level is used, which makes the estimates at each level independent of the bids that select them, and its interval-splitting estimator uses the variance of each interval rather than a uniform bound. Remark 3 notes that replacing the interval probabilities by gives only .
All results of the paper are proved in the preprint; none of them is machine-checked. A complete development formalizes a multi-level elimination algorithm with data-dependent level assignment, a Bernstein-type concentration argument for estimates indexed by random sets, and the counting lemma behind the logarithmic factors. Each of these is reusable for other elimination-based bandit algorithms.
Difficulty
The obvious route is a single UCB index per grid bid, (11) on p. 27, built from all past observations through the estimates . It fails because is not a normalized binomial variable: each bid depends on the earlier HOBs, so conditioned on the bids the HOBs are no longer independent and is not even unbiased. Self-normalized martingale bounds control each separately but not the sum that the reward estimate needs. Algorithm 4 restores conditional independence through its level structure, and Lemma 7 has to make that precise: the sets are random, and the bound must hold uniformly over all with a variance term estimated from a held-out sample. Lemma 9 then needs the variance-sensitive width (18); a crude width loses a factor (Remark 3).
Formalization scope
- Model. Rounds are
Fin T; Lean round is paper time . The HOB law is a probability measureμGonℝwithμG [0,1]ᶜ = 0, and is(μG (Set.Iic x)).toReal; atoms are allowed. The maximum in (3) is a supremum over . - Policies. A policy has internal randomness and, for each , a bidding rule whose round- bid receives only the values of rounds and the feedback of rounds , enforced by the argument types; bids lie in and are jointly measurable. The expectation is a Bochner integral over and , and Theorems 2 and 7 also assert integrability, so the integral cannot be a junk value.
- Algorithm 4. Grid indices are 0-based (, ); inside the algorithm times are the paper's . Every quantity is defined by its formula for every . The estimators (14) and are computed from the feedback, not from , and the algorithm recomputes its own past decisions, so it is a censored-feedback rule. "An arbitrary " is a selection rule depending on the time and the level only. If no level breaks (impossible for , ) the round goes to level with bid .
- Hypotheses added to the printed statements, each disclosed in its item: in Theorem 2 (the printed bound is at ); Lemmas 8, 9 and (19) are stated pathwise under the inequalities of Lemma 7 that their proofs use; Lemma 16 assumes that the set contains initialization rounds with bid , which its proof uses; the quantization display is adapted from the grid of Lemma 4 to that of Algorithm 4.
- Ruled out. A policy that reads , or future values, a constant chosen after , or , a policy chosen after or , and an expectation of a non-integrable function would each make the goal trivial; the statements exclude all four.
- Needed and welcome. Bernstein and multiplicative Chernoff bounds for binomial variables (Lemmas 14, 15 of the paper are cited from the literature), measurability of the run of the algorithm, and the harmonic-sum bound . Contributions toward any milestone, or alternative proofs of the goal, are welcome.
Selected references
- Y. Han, Z. Zhou, T. Weissman, Optimal No-regret Learning in Repeated First-price Auctions, Operations Research, 2025; preprint arXiv:2003.09795v7, 2024. https://arxiv.org/abs/2003.09795
- N. Cesa-Bianchi, C. Gentile, Y. Mansour, Regret Minimization for Reserve Prices in Second-Price Auctions, IEEE Transactions on Information Theory 61(1), 2015. https://doi.org/10.1109/TIT.2014.2365772
- P. Auer, Using Confidence Bounds for Exploitation-Exploration Trade-offs, Journal of Machine Learning Research 3, 2002. https://www.jmlr.org/papers/v3/auer02a.html
- S. Balseiro, N. Golrezaei, M. Mahdian, V. Mirrokni, J. Schneider, Contextual Bandits with Cross-Learning, NeurIPS 2019. https://arxiv.org/abs/1809.09582
- J. Davies, What to know about Google's implementation of first-price ad auctions, Digiday, 2019. https://digiday.com/media/buyers-welcome-auction-standardization-as-google-finally-goes-all-in-on-first-price/