§2 (p. 444), §3 (p. 446) — a deterministic discounted process has a stationary optimal policy

Let $0<\beta<1$, and consider a finite deterministic stationary Markov decision process with initial state $s_0$. Then some stationary policy $\delta$ (one with $\delta(s,t)=\delta(s,0)$ for all $s,t$) is optimal among all policies $\delta'(s,t)$:

where $(s_t)$ and $(s'_t)$ are the state sequences generated from $s_0$ by $\delta$ and $\delta'$.

This is the deterministic special case of the classical fact the paper invokes ("It is well known that in the two latter cases ... there exist stationary optimal policies"); it is the step that reduces the optimization over infinite policies to a finite one over sigmas.

Formalization Note The paper cites this fact for general finite stochastic processes; its general form is on the platform as BertsekasDP.discounted_main_theorem (part 3). This item states the deterministic case directly in this mission's model, so that it can be used without translating between the two models. The comparison is between the discounted costs of every policy, with no reference to an infimum.