Motivation
In many sequential decision problems the state of a system is not observed for free. A controller monitoring a machine, a patient or an inventory must pay to inspect it, and between inspections it acts on a belief rather than on the state. The observation cost model (OCM) makes this precise for a finite Markov chain: at each time the controller either pays a fixed cost cobs>0 to observe the state and choose a new action, or keeps the previous action without observing. Reisinger and Tam (arXiv:2201.07908v3, published in Mathematics of Operations Research, 2025) show that, after augmenting the state with the time elapsed since the last observation, the dynamic programming equation of the discounted OCM is a discrete quasi-variational inequality (QVI): an obstacle problem whose obstacle depends on the unknown itself.
Earlier work on the OCM (Anderson and Friedman, 1977; Hajek, Mitzel and Yang, 2008) treated fixed dynamics of the underlying chain. Discrete QVIs with interconnected obstacles and their penalty approximation were analysed by Reisinger and Zhang (2019) for switching problems, with the identity matrix in place of the transition operator Qn. The paper recasts the OCM QVI in an abstract form and proves it is well-posed.
Setting
Let X be a finite state space and A a finite nonempty action set. The unknown is a function u:N×X→RA with components ua(n,x)=ua,xn. A driver is a family Fa:N×X×RA×RN×X→R, a∈A; its third argument is the vector u(n,x) and its fourth the whole function ua. Given a sequence {Qn} of substochastic matrices on X (nonnegative entries, row sums at most 1) and a vector c∈RX×A, the inspection operator is
Mu(n,x)=(Qnu1−c)x,(u1−c)y=a∈Amax(ua,y1−ca,y).
It reads u only at time 1 and couples all actions. The QVI (3.1) is
min{Fa(n,x,u(n,x),ua),ua(n,x)−Mu(n,x)}=0,n∈N, x∈X, a∈A.
Assumption 3.2 asks for a constant β>0 with Fa(n,x,r,ua+θ)−Fa(n,x,s,va)≥βθ whenever u≤v are bounded and θ=ra−sa=maxj(rj−sj)≥0, together with boundedness in x and uniform continuity in r. Assumption 3.12 asks that Fa(n,x,r,ϕ) depend on ϕ only through finitely many values. For the OCM, Fa(n,x,r,ϕ)=ra−γϕ(n+1,x)−(Panra)x.
For the computation the paper truncates time to {1,…,N}, giving a finite QVI (3.7) for a map F~:RN×L×d→RN×L with the monotonicity property (3.5), and replaces it by the penalised problem (3.8), F~a(u)−ρπ(Mu−ua)=0, with a continuous non-decreasing penalty π vanishing exactly on (−∞,0].
Formalization targets
Goal: well-posedness (Theorem 3.13)
Under Assumptions 3.2 and 3.12, a uniform bound supa,n,x∣Fa(n,x,0,0)∣<∞, and ca,x>0:
∃!u:N×X→RA bounded with min{Fa(n,x,u(n,x),ua),ua−Mu}=0 everywhere.
Milestones
- Lemma 3.5: the restriction of F to {1,…,N} satisfies (3.5) with the same β.
- Proposition 3.10: for every ρ≥0, a subsolution of (3.8) lies below a supersolution.
- Lemma 3.11: for c≥0, every solution of (3.8) satisfies ∥uρ∥∞≤∥F~(0)∥∞/β.
- Theorem 3.13, truncated part: (3.8) has a unique solution uρ,N, which increases to a limit uN as ρ→∞; for c>0, uN solves (3.7).
- Proposition 3.8: comparison for the truncated QVI (3.7) when c>0.
- Proposition 3.3: comparison for bounded sub- and supersolutions of (3.1) when c>0.
- Corollary 3.4: bounded solutions of the OCM QVI (2.11) are unique.
Significance
The well-posedness theorem identifies the value function of the discounted observation cost model as the unique bounded solution of a QVI, which is what justifies computing it: truncating time and penalising the obstacle gives a sequence of finite nonlinear systems whose solutions converge to it, and these systems are solved in practice by semismooth Newton iterations. The abstract framework covers other models with a non-local, action-coupling obstacle, such as discretised switching problems.
The results are proved in the paper, partly by reference: existence of the penalised solutions and the convergence uρ,N→uN are delegated to Reisinger and Zhang (2019, Theorems 2.5 and 2.6), and Lemma 3.11 is stated as "a direct adaptation" of a lemma there. None of the results is formalized. A formal development makes the delegated steps explicit and fixes the hypotheses under which the statements hold (see Formalization scope).
Difficulty
The comparison principle (Proposition 3.3) is a maximum principle on an infinite index set: the supremum of u−v need not be attained, so the argument works along maximising sequences and needs the uniform continuity of the driver in r. In the obstacle branch, the inspection operator transports the maximum to time 1, and excluding that case uses c>0 through a chain argument over the finite set A×X; with c=0 comparison fails.
Existence is harder. A direct fixed-point argument fails because the driver is only monotone, not a contraction, and the obstacle depends on the unknown. The paper's route goes through two limits, ρ→∞ and N→∞, each requiring bounds uniform in the other parameter and a passage to the limit in a nonlinear, non-local equation; the N→∞ limit uses Assumption 3.12 to localise the dependence on ϕ.
Formalization scope
Everything is finite-dimensional or a function on the discrete set N×X; no measure theory is involved. Unknowns are curried as u n x a =ua(n,x), costs as c a x. Time in (3.1) ranges over all of N, including 0, and u1 is u 1. The truncated time set {1,…,N} is Fin N with the index k standing for time k+1, so u1 is u 0. Substochastic means nonnegative entries with row sums at most 1; c>0 and c≥0 are componentwise; norms are sup norms, written out entrywise. Bounded sets of r in Assumption 3.2.2 are cubes.
Two additions to the paper's hypotheses are made, each because the statement is false without it. The goal assumes supa,n,x∣Fa(n,x,0,0)∣<∞: Assumption 3.2.2 only bounds F for each fixed n, and the driver ra−γϕ(n+1,x)−n satisfies both assumptions yet has no bounded solution. Corollary 3.4 is stated for bounded solutions of (2.11). The truncated QVI is stated without Problem 3.6's time boundary condition ua(N,⋅)=Mu(N,⋅), which (3.8) does not encode. The continuity of the restricted driver claimed in Lemma 3.5 is not part of that milestone; the truncated part of Theorem 3.13 takes continuity of an abstract F~ as a hypothesis.
The N→∞ clause of Theorem 3.13 ("u=limN→∞uN, monotone from below") is not stated: the paper does not define the truncated operator beyond time N, and with the zero extension the monotonicity in N fails. The goal keeps the clause's well-defined content, existence and uniqueness of the bounded solution. A goal asserting only uniqueness (Proposition 3.3), or existence with the uniform bound hidden inside a definition, would be a different theorem; the bound is an explicit hypothesis.
A complete development needs solvability of finite-dimensional monotone nonlinear systems (the existence result cited from Reisinger and Zhang), monotone convergence in finite dimensions, and maximum-principle arguments on N×X. The comparison principles and the a-priori bound are reusable for other monotone schemes. Proofs of any milestone, and alternative existence arguments for the goal, are welcome.
Selected references
- C. Reisinger and J. Tam, Markov decision processes with observation costs: framework and computation with a penalty scheme, arXiv:2201.07908v3, 2023; Mathematics of Operations Research, 2025. https://arxiv.org/abs/2201.07908v3
- C. Reisinger and Y. Zhang, A penalty scheme for monotone systems with interconnected obstacles: convergence and error estimates, SIAM Journal on Numerical Analysis 57(4):1625–1648, 2019 (reference [28] of the paper).
- R. F. Anderson and A. Friedman, Optimal inspections in a stochastic control problem with costly observations, Mathematics of Operations Research 2(2):155–190, 1977. https://doi.org/10.1287/moor.2.2.155
- B. Hajek, K. Mitzel and S. Yang, Paging and registration in cellular networks: jointly optimal policies and an iterative algorithm, IEEE Transactions on Information Theory 54:608–622, 2008.