Motivation
A bandit sampling process models a population from which values are drawn one at a time: the values are independent with a distribution f(⋅∣θ) whose parameter θ is unknown, the decision maker holds a prior on θ and updates it by Bayes' theorem, and each draw earns a reward. When several such populations compete for the same sampling effort, the Gittins index theorem reduces the allocation problem to computing one index per population (Gittins, J. R. Stat. Soc. B 1979). The obstacle is computation: the state is a posterior distribution, so the index is a function of every parameter of the posterior.
Chapter 7 of Gittins, Glazebrook and Weber, Multi-armed Bandit Allocation Indices (2nd ed., Wiley 2011, doi:10.1002/9780470980033), shows that location and scale structure removes most of that dependence. For a normal population with unknown mean and variance, Theorem 7.15 says that the index in the state (xˉ,s,n) is xˉ+sν(0,1,n), so a single function of n suffices; the tables of Chapter 8 rest on this. The same reduction holds for target processes, where the aim is to observe a value above a target T as soon as possible (Theorems 7.19–7.23).
This mission continues the public series Multi-armed Bandit Allocation Indices I–VI in the namespace AllocationIndices. Mission VI formalized the location and scale halves of the theory (Theorems 7.9 and 7.11) and the location invariance of target processes (Theorem 7.17), all now proved; this mission states the joint location–scale results on the same published definitions.
Setting
A sampling model SamplingModel Θ Q (published, AllocationIndices_Sampling) consists of a likelihood f(⋅∣θ), a family of priors π(⋅∣p) on θ∈Θ indexed by the parameters p∈Q of a conjugate family, and the update p↦px of those parameters when x is the next value sampled; the family is conjugate on a set of parameters if the update computes the Bayesian posterior there. The reward process has state p, draws x from the predictive distribution f(⋅∣p)=∫f(⋅∣θ)π(dθ∣p), moves to px, and earns the predictive mean r(p). Rewards are discounted by at, 0<a<1; for a stopping time τ, Rτ(p) is the expected discounted reward before τ, and ν(p) is the Gittins index (2.6). The target process with target T stops as soon as a sampled value is ≥T.
Two calibration quantities compare a process with a known one. The calibration payoff against a standard bandit process paying λ per period is
R(λ,p)=τsup{Rτ(p)+λ(1−a)−1E(aτ∣p)},
the supremum over all stopping times, τ being the time at which the standard process is first selected (calibrationPayoff). The target cost against a standard target process with known success probability φ is
M(φ,p,T)=τinf{Cτ(p,T)+φ−1Qτ(p,T)},
where Cτ is the expected number of samples drawn before τ or before the target is reached, and Qτ the probability that τ comes first (targetCost, undiscounted, valued in [0,∞]).
Parameters μ and σ>0 are joint location and scale parameters if f(⋅∣bμ+c,bσ) is the image of f(⋅∣μ,σ) under x↦bx+c for b>0, c∈R. With σ unknown the conjugate prior on (μ,σ) has parameters (xˉ,s,n), which change by the rules of the sample mean, the sample standard deviation and the number of observations, and xˉ,s are joint location and scale parameters of the prior family: the prior with parameters (bxˉ+c,bs,n) is the image of that with parameters (xˉ,s,n) under (μ,σ)↦(bμ+c,bσ). With σ known the parameters are (xˉ,n,σ) and only xˉ,n change.
Formalization targets
Goal: Theorem 7.15 (p. 193)
For the reward process with σ unknown, in a state with s>0, n>1, for all λ, b>0 and c,
R(bλ+c, bxˉ+c, bs, n)=c(1−a)−1+bR(λ,xˉ,s,n),ν(xˉ,s,n)=xˉ+sν(0,1,n).
Both conclusions are part of the goal (AllocationIndices.Targets.location_scale_invariance).
Milestones
- Theorem 7.13 (p. 192), σ known: R(bλ+c,bxˉ+c,n,bσ)=c(1−a)−1+bR(λ,xˉ,n,σ) and ν(xˉ,n,σ)=xˉ+σν(0,n,1).
- Theorem 7.19 (p. 194), target process with a scale parameter: M(φ,xˉ,n,T)=M(φ,bxˉ,n,bT) and, for T>0, ν(xˉ,n,T)=ν(xˉ/T,n,1).
- Theorem 7.21 (p. 194), target process, σ known, as printed: M(φ,xˉ,n,σ,T)=M(φ,xˉ+c,n,bσ,bT+c) and ν(xˉ,n,σ,T)=ν((xˉ−T)/σ,n,1,0).
- Theorem 7.23 (p. 194), target process, σ unknown: M(φ,xˉ,s,n,T)=M(φ,bxˉ+c,bs,n,bT+c) and ν(xˉ,s,n,T)=ν((xˉ−T)/s,1,n,0).
Significance
The result reduces the index of a two-parameter family of posteriors to a function of one variable and makes the normal reward process with unknown variance computable at all; the Chapter 8 tables of ν(0,1,n) and the corresponding target-process tables are only meaningful because of it. The identities for R and M are what the calibration method of computing those tables uses, so they are stated too.
The theorems are proved in the book only in outline: the proofs of Theorems 7.13 and 7.15 are "left as exercises", and those of 7.19–7.23 are said to be similar to their reward-process counterparts. A formal proof supplies the arguments the book omits. Mission VI proved the one-parameter cases on the platform; the joint cases, the calibration payoff R and the target cost M have no machine-checked version.
Difficulty
The book reasons with densities and improper priors. In the formal model the state is a parameter vector, the posterior is Mathlib's Bayesian inverse, defined only almost everywhere with respect to the predictive distribution, and stopping times are measurable functions of an entire trajectory of states. Relating the processes started at (xˉ,s,n) and at (bxˉ+c,bs,n) requires relating two Markov chains on an uncountable state space, together with their stopping times, rather than two densities. The two index identities are also not formal consequences of the identities for R and M in the published model, whose index is defined as a supremum of reward rates rather than by calibration.
Formalization scope
Discrete time throughout; R uses the factor (1−a)−1 exactly as printed in (7.15), so no unit conversion is involved; the index is gittinsIndex (2.6) of the reward or target chain with 0<a<1, and M is undiscounted as in (7.16). The book's improper priors (π0(μ) uniform, π0(μ,σ)∝σ−d) are used only through the proper posteriors they generate; as in Corollaries 7.14, 7.16 and mission VI, the state is the parameter vector of a proper conjugate family, required to be conjugate on n>0 (σ known) or on s>0, n>1 (σ unknown), the domain on which the sample-standard-deviation update is defined and keeps s>0. "σ known" means the prior is concentrated on the scale coordinate of the state. The prior family's location–scale property is an image of measures; the printed density form (7.25) has a Jacobian b where b2 is needed and lacks the shift +c in the conditioning argument, so as printed no probability density satisfies it. The reward theorems assume integrable observations and finite expected discounted absolute rewards from every state, which makes R a finite supremum over a nonempty set; this rules out the trivializing reading in which a real supremum of an unbounded set defaults to 0, and the goal states both the identity for R and the index identity, not the reward shift r(bxˉ+c,bs,n)=br(xˉ,s,n)+c alone. In Theorem 7.19 the index identity carries T>0, which the book leaves implicit; in Theorem 7.23 the stray argument σ of ν(xˉ,s,n,σ,T) is dropped.
Beyond the paper. The first identity of Theorem 7.21 scales σ and T by b but shifts xˉ without scaling it; with normal observations, φ=1/2, n=1, T=0, xˉ=3, b=2, the two sides are at most 1.034 and at least 1.144. The faithful item states it as printed; a corrected variant with bxˉ+c is posed beside it.
Definitions are in AllocationIndices.Targets.Model; the model, conjugacy, reward and target chains, HasScaleParameter, PriorHasScaleParameter and meanUpdate are the published AllocationIndices_Sampling. Lemmas on transporting a Markov chain's path measure and its stopping times along a measurable bijection of the state space are reusable beyond this mission; proofs of single milestones are welcome.
Selected references