Lemma 7, p. 19 — the conservative policy loses at most ξ/(1−γ) in its own MDP: V̂^π − V̂^{π̂c} ≤ ξ/(1−γ)

Let $\widehat{\mathcal M} = (\mathcal S, \mathcal A, \widehat P, r, \gamma)$ be a finite discounted MDP (kernel $\widehat P$, rewards in $[0,1]$, $\gamma\in(0,1)$) with a linearly ordered action set, and let $\widehat Q^\star$, $\widehat V^\star$ be its optimal Q- and value functions. Let $\varsigma \le \xi$ and let $\widehat\pi_{\mathrm c}$ follow the conservative rule $\widehat\pi_{\mathrm c}(s) = \min\{a : \widehat Q^\star(s,a) > \widehat V^\star(s) - \varsigma\}$. Then for every deterministic policy $\pi$ and every state $s$,

In the paper $\pi = \pi^\star$, the optimal policy of the true MDP, and $\widehat P$ is the empirical kernel; the bound controls the middle term of the decomposition (51) of $V^\star - V^{\widehat\pi_{\mathrm c}}$.

Formalization Note The statement holds for any kernel and for any policy $\pi$ in place of $\pi^\star$, which is stronger than the page (the paper's proof ends with $\widehat V^{\pi^\star} \le \widehat V^\star$, valid for every $\pi$). Only $\varsigma \le \xi$ is assumed; the rule cannot hold for $\varsigma \le 0$, so $\varsigma \in [0,\xi]$ of the page is implied.