§3, discounted case (p. 446) — the path of a stationary policy is a sigma, and every sigma is such a path

Consider a finite deterministic stationary Markov decision process with initial state $s_0$.

1. For every stationary policy $\delta$ there is a sigma $P$ from $s_0$ whose nodes $w_0,\dots,w_N$ are the first $N+1$ states $s_0,\dots,s_N$ generated by $\delta$, and whose discounted cost equals that of $\delta$ for every $0<\beta<1$: $J_\beta(\delta)=c_\beta(P)$.
2. Conversely, for every sigma $P$ from $s_0$ there is a stationary policy $\delta$ whose first $N+1$ states are the nodes $w_0,\dots,w_N$ of $P$ and with $J_\beta(\delta)=c_\beta(P)$ for every $0<\beta<1$.

Here $J_\beta(\delta)=\sum_{t\ge0}\beta^tc(s_t,\delta(s_t,t))$ and $c_\beta(P)$ is the discounted cost of the infinite walk that follows $P$ and repeats its cycle forever. Together with the existence of a stationary optimal policy, this identifies the optimal discounted cost with the least discounted cost of a sigma.

Formalization Note The cost identities are stated for the paper's discount range $0<\beta<1$, where both infinite sums converge.