Motivation
Machines in production lines deteriorate in ways that are only partly visible. A machine may run in a degraded state that raises costs without announcing itself, and eventually fail in a way that cannot be missed. The decision maker must choose between continuing production, paying for an inspection that reveals the hidden state, and replacing the machine preventively. Models of this kind are partially observable Markov decision processes (POMDPs), and for POMDPs closed-form optimal policies are rare: the value function lives on the space of beliefs and is usually only computed numerically.
H. Zhang and W. Zhang, Analytical Solution to a Partially Observable Machine Maintenance Problem with Obvious Failures (Management Science, 2022; read here as SSRN 2945049, version of 2022-08-21), solve the classical two-state version of this problem completely. Working in the dual "efficient frontier" framework of Zhang (2010), they show that the optimal solution has exactly six possible structures and give each in closed form, with explicit or implicitly characterized thresholds on the replacement and inspection costs.
Setting
A machine is in a hidden state x∈{0,1} (good or bad) or in the observable failure state F. Each period, if the machine has failed it is replaced correctively at cost cF; otherwise one of three actions is chosen:
- production P costs 0 in state 0 and cP in state 1;
- inspection I costs cI and reveals the current state;
- preventive replacement R costs cR and resets the machine to state 0.
Production always takes place. From state 0 the machine moves to state 1 with probability γ0; from state 1 it fails with probability γ1. Write γˉx=1−γx, p0=(γˉ0,γ0)′, p1=(0,γˉ1)′, gP(uF)=(0,cP+βγ1uF)′, gI(uF)=(cI,cI+βγ1uF)′ and PPu=(p0′u,p1′u)′. Costs are discounted by β over an infinite horizon.
A continuation-cost vector u=(u0,u1)′ records the cost of a continuation policy from each hidden state, and uF the cost after a failure. The expected optimal cost at an (unnormalized) belief π=(π0,π1)′≥0 is V(π)=minu∈U∗π′u, where U∗ is the set of extreme points of the efficient frontier. The pair (U∗,uF∗) is the fixed point of the dual value iteration (12)–(16). In support-function form, a pair (W,wF) is mapped to
W′(π)=min{π′gP(wF)+βW((PP)′π), (π0+π1)(cR+βW(p0)), π′gI(wF)+β(π0W(p0)+π1W(p1))},
wF′=cF+βW(p0),
the three terms being P, R and I. A finite set U with a number uF is the optimal solution for an action set if value iteration from zero terminal costs converges to minu∈Uπ′u for every π≥0 and to uF after a failure.
The structures are built from two point sequences. The production sequence is ui+1=gP(uF)+βPPui. Started on the diagonal at (uR,uR)′ with uF=uR+cF−cR, the self-consistency equation uR=cR+βp0′um has a unique solution uR=f(m) (31). Started instead at the inspection point (35), it gives g(m) (40).
Formalization targets
Goal: Theorem 2 (pp. 27–28, Table 1)
Let θ=1−β+β(γ0+γ1)cP+βγ1cF and c^=(1+βγ0)cP−βγˉ1(1−β+βγ0+βγ1)cR+β2γ1γˉ1cF. Under {P,I,R}:
cR≥θ ⇒ [P];γ1γ0θ≤cR<θ ⇒ R[P], IR[P] or [PIR] by explicit thresholds on cI;
cR<min{γ1γ0θ,θ} ⇒ ∃0<c^1≤c^2<cR: [PR], I[PR] or [PIR] as cI≥c^2, c^1≤cI<c^2, cI<c^1,
with every structure an explicit pair (U∗,uF∗) from Table 1, and the thresholds characterized implicitly (uI† on the frontier of U† at cI=c^2; minf=ming at cI=c^1).
Milestones
Proposition 2 (the fixed-point equations), Proposition 3 (the cost of producing forever), Lemma 1 and Lemma 6 (the two sequences in closed form), Lemma 3 and Lemma 7 (the anchors f and g), Proposition 4 (unimodality of f), Theorem 1 (the two-action solution: [P], R[P], [PR], with extreme-point exactness) and Lemma 5 (when inspection enters).
Significance
The result. The theorem turns a POMDP into a finite list of explicit policies. Each structure is a periodic plan that needs no belief updating: produce for m† periods and replace; or inspect every m∗ periods and replace only if the machine is found bad. The thresholds tell when inspection is worth its cost, and the comparative statics of m† and m∗ follow from the closed forms. The authors use the solution to justify common maintenance practices.
Formalizing it. The result is proved in the paper (main text §4, proofs in its e-companion, App. A); no part is machine-checked. The proofs of Theorems 1 and 2 argue geometrically from figures of convex curves in the plane ("the curve bends back", "all the points are still extreme points"). A formal development has to turn these into inequalities between piecewise-linear concave functions, and has to prove convergence of value iteration to the stated sets. The implicit thresholds of part (3) were not checked numerically when this mission was planned. Parts (1) and (2) and Theorem 1 were.
Difficulty
Verifying that a candidate set U is a fixed point is not enough: the goal asserts that value iteration from zero converges to it, so uniqueness of the bounded fixed point must be proved (a contraction argument in the supremum norm over the simplex). More importantly, finding the frontier requires knowing which points of the infinite production sequence are extreme, which depends on the location of the anchor relative to the curve {ui:i≥0} (Fig. 5 of the paper, five subcases). The natural induction on m fails because f is not monotone: its least minimizer m† must be located through the unimodality of the real extension (Proposition 4). In part (3) the thresholds c^1,c^2 are defined implicitly, and showing that the three regimes are ordered in cI requires monotonicity of the whole frontier in cI.
Formalization scope
Everything is defined in the namespace MaintPOMDP.Frontier. Vectors and beliefs are Fin 2 → ℝ (index 0 = good state); beliefs are unnormalized, π≥0. The optimal solution is the limit of the support-function value iteration started from W≡0, wF=0; "the structure is X" means that this limit equals min over the points of row X of Table 1, with the failure cost of that row.
Standing assumptions: 0<β<1, 0<cP<cI<cR<cF (p. 14), and 0<γ0<1, 0<γ1<1, which the paper never states and which are added (the thresholds divide by γ1). The hypothesis γ0=γ1 is added only to Lemmas 1, 3, 6, 7 and Proposition 4, whose printed formulas divide by γ1−γ0; f and g themselves are defined as solutions of the affine anchor equations, so Theorems 1 and 2 cover γ0=γ1. Proposition 4 also carries the range of cR that its proof uses. "m†=argminf" is the least minimizer, and its existence is part of the conclusion. In Theorem 2, the indices nR,nI of Table 1 are existential and extreme-point exactness is not asserted for the I[PR] and [PIR] rows (Table 1's I[PR] row conflicts with the paper's Proposition 8). It is asserted in Theorem 1. In part (3) the thresholds are chosen before cI and the claims hold for every cI∈(cP,cR).
Trivializing readings are ruled out: the value function is the limit of the iterates, not an arbitrary fixed point or a junk limit; identities hold for every belief π≥0, not only at vertices; every frontier set is given by formulas, so no extra points can be slipped in; minimizers are asserted, not assumed; and thresholds involving γ0/γ1 are evaluated with γ1>0.
A complete development needs convergence of value iteration for a monotone contraction on bounded functions over the cone π≥0, explicit solutions of two-dimensional linear recurrences, and lemmas on lower envelopes of finitely many linear functions in two variables. These are reusable beyond this mission, for other two-state POMDPs. Contributions of any milestone, of the γ0 = γ1 closed forms (App. B), or of a proof of Theorem 1 alone are welcome.
Selected references
- H. Zhang, W. Zhang, Analytical Solution to a Partially Observable Machine Maintenance Problem with Obvious Failures, Management Science, 2022. https://doi.org/10.1287/mnsc.2022.4547 (read as SSRN 2945049, https://ssrn.com/abstract=2945049)
- H. Zhang, Partially Observable Markov Decision Processes: A Geometric Technique and Analysis, Operations Research 58(1), 2010. https://doi.org/10.1287/opre.1090.0697
- R. D. Smallwood, E. J. Sondik, The Optimal Control of Partially Observable Markov Processes over a Finite Horizon, Operations Research 21(5), 1973. https://doi.org/10.1287/opre.21.5.1071
- S. M. Ross, Quality Control under Markovian Deterioration, Management Science 17(9), 1971. https://doi.org/10.1287/mnsc.17.9.587