Motivation
A partially observable Markov decision process (POMDP) models a controller that cannot see the state of the system it steers, only noisy observations of it. Inventory control with unrecorded demand, machine maintenance with imperfect inspection, and target tracking from sensor readings all have this form. The standard way to solve a POMDP is to replace it by its belief MDP, also called the completely observable MDP (COMDP): the state becomes the posterior distribution of the hidden state given everything observed so far. The belief MDP is fully observed, so the theory of Markov decision processes applies to it, but only if the belief MDP satisfies that theory's hypotheses. For MDPs with Borel state and action spaces, unbounded costs and possibly noncompact action sets, the hypothesis that yields optimality equations, convergence of value iterations and stationary optimal policies is Assumption (W*) of Feinberg, Kasyanov and Zadoianchuk (Math. Oper. Res. 2012): the cost is K-inf-compact and the transition probability is weakly continuous.
Before Feinberg, Kasyanov and Zgurovsky (arXiv:1401.2168v2, published in Math. Oper. Res. 41(2), 2016), sufficient conditions for the belief MDP to be weakly continuous required a weakly continuous filter (Hernández-Lerma, 1989, as recalled on p. 11 of arXiv:1401.2168v2), a condition that fails in many natural models. The paper replaces it by a condition on the primitive data of the POMDP.
Setting
Let X (states), Y (observations) and A (actions) be Borel subsets of Polish spaces. Let P(dx′∣x,a) be the transition kernel on X given X×A, Q(dy∣a,x) the observation kernel on Y given A×X, and c:X×A→R∪{+∞} a Borel one-step cost bounded below. P(X) is the set of probability measures on X with the topology of weak convergence (∫fdμn→∫fdμ for bounded continuous f). A sequence μn converges setwise if μn(C)→μ(C) for every Borel C, and in the total variation if supf∣∫fdμn−∫fdμ∣→0 over Borel f with values in [−1,1]. A kernel is weakly (setwise, total-variation) continuous if xn→x implies the corresponding convergence of the measures.
For a belief z∈P(X) and an action a, let R(B×C∣z,a)=∫X∫BQ(C∣a,x′)P(dx′∣x,a)z(dx) be the joint law of the next state and observation (3.1), and R′(C∣z,a)=R(X×C∣z,a) its observation marginal (3.2). A filter is a kernel H(dx∣z,a,y) with R(B×C∣z,a)=∫CH(B∣z,a,y)R′(dy∣z,a) (3.3): H(z,a,y) is the posterior after observing y. The belief MDP has state space P(X), action set A, cost cˉ(z,a)=∫c(x,a)z(dx) (3.8), and transition q(⋅∣z,a), the law of H(z,a,y) when y∼R′(⋅∣z,a) (3.5). A cost is K-inf-compact on S1×S2 if its restriction to K×S2 has compact level sets for every compact K⊆S1.
Formalization targets
Goal: Theorem 3.6
If either Assumption (D) (0≤α<1) or Assumption (P) (c≥0 and 0≤α≤1) holds, c is bounded below and K-inf-compact on X×A, P is weakly continuous and Q is continuous in the total variation, then the belief MDP satisfies Assumption (W*):
cˉ is bounded below and K-inf-compact on P(X)×A,(zn,an)→(z,a)⟹q(⋅∣zn,an)→q(⋅∣z,a) weakly.
The two halves
- Cost half (Theorem 3.4, through Lemma 6.1 and Lemma 5.1(ii)): cˉ is bounded below by the same constant as c and K-inf-compact.
- Transition half (Theorem 3.7): under the hypotheses on P and Q, R′ is setwise continuous and Assumption (H) holds: whenever zn→z and an→a, along a subsequence H(znk,ank,y)→H(z,a,y) weakly for R′(⋅∣z,a)-almost every y. Theorem 3.5 then gives weak continuity of q. The route runs through Theorem 5.2, Lemma 5.3, Corollary 5.4, Theorem 5.5 and Lemma 5.6.
Significance
Theorem 3.6 supplies the hypothesis under which the cited general MDP theory gives stationary optimal policies, optimality equations and convergence of value iterations for total-cost POMDPs in terms of P, Q and c alone, without compactness of the action set, boundedness of the cost or continuity of the filter. It covers, for example, observation kernels with densities continuous in (a,x), and the paper applies it to inventory control with incomplete records and to the Kalman filter. Example 4.1 of the same paper shows that total-variation continuity of Q cannot be weakened to setwise continuity.
The result is proved in the paper; this proposal poses its formal statement and the supporting results as proof targets. Completing them would provide a checked bridge from POMDPs to the general MDP theory under weak continuity, together with reusable facts about weak, setwise and total-variation convergence: equicontinuity of integrals against weakly continuous kernels (Theorem 5.2), the convergence-in-probability criterion (Theorem 5.5), and the generalized Fatou lemmas.
Difficulty
The posterior H(z,a,y) is defined only for R′(⋅∣z,a)-almost every y and has no continuous version in general, so weak continuity of q cannot be read off from continuity of H. The tempting argument, that continuous P and Q make the Bayes posterior move continuously, fails: Y need not be countable and Q need not have densities. What has to be controlled is the joint law R on sets (O1∖O2)×C uniformly in the observation set C; convergence for each fixed C is not enough. Turning this uniform control into almost-sure convergence of posteriors along a subsequence requires a diagonal argument over a countable base of X. On the cost side, the difficulty is that c may take the value +∞ and is not continuous; K-inf-compactness has to be transported through integration against a weakly converging sequence of measures.
Formalization scope
All definitions live in FeinbergPOMDP.WStar.Model. X,Y,A are separable metrizable spaces with their Borel σ-algebras (PolishSpace is not assumed, since Borel subsets of Polish spaces need not be Polish); X is additionally nonempty and standard Borel in Theorems 3.6 and 3.7, where a filter must exist. P(X) is Mathlib's ProbabilityMeasure with the weak topology. All continuity notions are the paper's sequential ones; total variation is the supremum over Borel functions with values in [−1,1], written in ε–N form, while the set-indexed suprema of (5.15) and (5.17) keep their set form. The cost is written c=ℓ+f with ℓ∈R and f taking values in [0,∞], so cˉ=ℓ+∫fdz, exact even when c=+∞ on a set of positive measure. K-inf-compactness is the published FeinbergLiang.ACOE.KInfCompact. A filter is a map H:P(X)→A→Y→P(X) that satisfies (3.3) and is jointly Borel measurable in (z,a,y), as the paper's stochastic kernel is; q is the image of R′ under H(z,a,⋅) for the Borel σ-algebra of the weak topology on P(X).
Since q does not depend on the choice of filter, every statement about q is made for every filter satisfying (3.3). A fixed filter that ignores the observation, such as H(z,a,y)=z, would make q a point mass and the weak continuity trivial; it satisfies (3.3) only when observations carry no information, and is excluded by quantifying over all filters.
The goal states the Assumption (W*) conclusion of Theorem 3.6. Its tail, "statements (i)–(vi) of Theorem 3.1 hold", invokes Theorem 2.1 of Feinberg et al. (2012) applied to the belief MDP; it is cited rather than proved in this paper. Generalized Fatou's lemmas 5.1(i) and 5.1(ii) are referenced from the published Schäl (1993) items. A complete development needs weak and setwise convergence on ProbabilityMeasure, portmanteau-type arguments, disintegration (Measure.condKernel) and the identification of the Borel and Giry σ-algebras on P(X); the last is missing from Mathlib and is a welcome contribution in its own right. Proofs of any milestone, and alternative arguments for Theorem 3.7, are welcome.
Selected references
- E. A. Feinberg, P. O. Kasyanov, M. Z. Zgurovsky, Partially Observable Total-Cost Markov Decision Processes with Weakly Continuous Transition Probabilities, arXiv:1401.2168v2 (2014); Mathematics of Operations Research 41(2), 2016. https://arxiv.org/abs/1401.2168v2
- E. A. Feinberg, P. O. Kasyanov, N. V. Zadoianchuk, Average-cost Markov decision processes with weakly continuous transition probabilities, Mathematics of Operations Research 37(4), 2012. https://doi.org/10.1287/moor.1120.0555
- D. P. Bertsekas, S. E. Shreve, Stochastic Optimal Control: The Discrete-Time Case, Academic Press, 1978. https://web.mit.edu/dimitrib/www/soc.html
- M. Schäl, Average optimality in dynamic programming with general state space, Mathematics of Operations Research 18(1), 1993. https://doi.org/10.1287/moor.18.1.163