Motivation
Periodic-review inventory models with a fixed ordering cost are the setting in which the (s,S) policy is optimal: when the stock falls below a reorder point s, order up to a level S, otherwise order nothing. Scarf (1960) proved this for independent demands over a finite horizon by introducing K-convexity, and Iglehart (1963) extended it to the discounted infinite horizon. In practice the demand distribution changes with an exogenous environment: the economy, the season, the stage of a product's life cycle. Song and Zipkin (Oper. Res. 41(2), 1993) modelled this environment as a Markov chain of demand states in continuous time with Poisson demand.
Sethi and Cheng (Oper. Res. 45(6), 1997) treat the discrete-time version with general demand densities, nonstationary costs and convex surplus costs of linear growth. The companion mission (I) covers their finite-horizon result, Theorem 4.1. This mission covers the nonstationary discounted infinite-horizon result, Theorem 6.2, together with the existence theory (Theorem 6.1, proved in the paper's Appendix) on which it rests.
Setting
Demand states form the finite set I={1,…,L} and follow a Markov chain with transition matrix P=(pij). In period k=0,1,2,… the demand ξk≥0 has the conditional density φi,k given ik=i, with mean at most M. Orders are placed at the start of a period and delivered at once, and unmet demand is backlogged, so the surplus evolves by xk+1=xk+uk−ξk (2.4). Ordering u≥0 costs ck(i,u)=Kkiδ(u)+ckiu with Kki,cki≥0 (2.1), where δ(z)=1 for z>0 and 0 otherwise. The surplus cost fk(i,⋅) is convex and nonnegative, vanishes at 0, and satisfies fk(i,x)≤C(1+∣x∣) (2.2).
An admissible policy U chooses each order uk≥0 as a measurable function of the past demand states and demands. With a discount factor α, the objective from period n, state i and surplus x is
Jn(i,x;U)=k=n∑∞αk−nE[ck(ik,uk)+fk(ik,xk)],(6.1)
and the value function is vn(i,x)=infUJn(i,x;U). With Fn+1(b)(i,y)=∑jpij∫0∞b(j,y−z)φi,n(z)dz (3.1), the dynamic programming equations are
vn(i,x)=fn(i,x)+u≥0inf{cn(i,u)+αFn+1(vn+1)(i,x+u)},n=0,1,2,…(6.2)
The truncations Jn,k keep the first k periods of (6.1) (6.3), their values are vn,k (6.4), and wn=Jn(⋅;0) is the cost of never ordering (6.6). The function classes are B0 (nonnegative pointwise limits of continuous functions), B1 (those of linear growth) and C1 (those also uniformly continuous in x). A function g is K-convex if K+g(z+y)≥g(y)+z(g(y)−g(y−b))/b for all z≥0, b>0 (4.3). Assumption (4.1) asks Kni≥∑jpijKn+1j≥0, and (4.2) asks cnix+Fn+1(fn+1)(i,x)→+∞ as x→∞.
Formalization targets
Goal: Theorem 6.2
Under (2.1), (2.2), (4.1), (4.2) and 0<α<1, there are sni≤Sni in [−∞,∞) such that
u^n(i,x)=(Sni−x)δ(sni−x)
attains the infimum in (6.2) at every (n,i,x), and its feedback policy, started at any period n and surplus x, has cost Jn(i,x;U^)=vn(i,x): it is optimal among all history-dependent policies.
Milestones
- The order-nothing cost wn is finite, lies in C1, and is the unique C1 solution of wn=fn+αFn+1(wn+1) (6.7).
- The truncated values solve the dynamic programming equations (6.5), vn,0=0, vn,k∈C1, and under (4.2) the infimum in (6.4) is attained.
- Theorem 6.1, (6.8): 0=vn,0≤vn,1≤⋯≤vn,k≤wn.
- Appendix: ∣vn,k(i,x′)−vn,k(i,x)∣≤C∣x′−x∣/(1−α), and the same for vn=limkvn,k.
- Theorem 6.1, (6.9): vn,k↑vn, a solution of (6.2) in B1.
- Theorem 6.1, (6.10): vn∈C1, a B0 minimizer U^ of (6.2) exists, and every such U^ is optimal: vn(i,x)=minUJn(i,x;U)=Jn(i,x;U^).
- Proof of Theorem 6.2: vn,k(i,⋅) is Kni-convex.
- Proof of Theorem 6.2: vn(i,⋅) is Kni-convex.
The structural lemma behind the step from K-convexity to the (s,S) form, Proposition 4.2 of the paper (an extension of Bertsekas' lemma to lower semicontinuous functions coercive only at +∞, with possibly infinite s, S), is posed in the companion finite-horizon mission and is not repeated here.
Significance
Theorem 6.2 says that the classical (s,S) structure survives two generalizations at once: demand driven by a Markov environment, and nonstationary data over an infinite horizon. The two levels may depend on the period and on the current demand state, so only 2L numbers per period describe an optimal policy, and with cyclic demand only L pairs are needed (Corollary 7.1 of the paper). Theorem 6.1 is a self-contained existence result for a positive-cost, unbounded-state dynamic program: value iteration from zero converges to the value function, which solves the optimality equation, and any measurable minimizer is optimal among history-dependent policies.
The results are proved in the paper, with the finite-horizon existence theory delegated to Beyer, Sethi and Taksar (1998). No machine-checked proof of any of them is known. The platform has K-convexity (BertsekasKConvex) and the (s,S) structure lemma of Bertsekas for continuous functions coercive at both ends; it has no discounted infinite-horizon (s,S) theorem with Markov-modulated demand. The related posed items SongZipkinFluct.FixedCost.theorem5e_theorem6 and theorem5ab concern the Song–Zipkin model (continuous time, Poisson demand, integer levels, stationary data), a different model.
Difficulty
The obvious argument is "the finite-horizon value functions are K-convex, so their limit is, and a minimizer of a K-convex optimality equation is (s,S)". Three steps of it fail as stated.
First, the limit of value iteration need not be the value function, nor solve (6.2), in a positive-cost problem with unbounded actions: exchanging limk with infu requires the minimizers of the truncated problems to stay in a compact set uniformly in k, and that bound is where the coercivity assumption (4.2) enters. Second, optimality against history-dependent policies, not only Markov ones, requires a verification argument along histories of states and demands. Third, the functions zn(i,y)=cniy+αFn+1(vn+1)(i,y) may be unbounded below, in which case the reorder level is s=−∞ and the order-up-to level does not exist; the goal allows this case.
Formalization scope
The Lean development (namespace SethiChengSS.Infinite) commits to the following.
- Demand states are
Fin L; periods are natural numbers with the paper's base 0.
- Expectations are sums over state paths of Lebesgue integrals against the product weight ∏tpitit+1φit,t(ξt), valued in [0,∞]. This builds in the conditional independence of ξk and ik+1 given ik that (3.1) uses.
- A policy is a function of the state history and the past demands; admissible means nonnegative and measurable in the demands. The comparison class is never restricted to Markov policies.
- Jn (6.1) is the supremum over k of the truncations Jn,k (6.3), which equals the series because all costs are nonnegative.
- Fn+1 uses the Bochner integral, which is 0 on non-integrable integrands; the C1 conclusions are what make it genuine.
- The levels s, S are extended reals with s≤S, S=+∞, and S real whenever s is (Remark 4.4).
Pins. The paper prints 0<α≤1; the mission assumes 0<α<1, since wn is infinite at α=1 and the Appendix divides by 1−α. Theorem 6.2 is printed without (4.1), which its proof uses through the induction of Section 4; (4.1) is assumed for every period. Theorem 6.1 is printed under (2.1)–(2.2) only, but attainment in (6.2) can fail without coercivity and the Appendix bound (A.3) divides by cni; (4.2) is added to (6.9), (6.10) and to the attainment clause of (6.5). "In class C1" for the sequence wn is read with one growth constant for all periods. The boundary condition "vn+k,0=0" of (6.5) is read as vn,0=0, as the next sentence of the paper states.
The goal cannot be met by a degenerate encoding: the value function is the infimum over all admissible history-dependent policies (not the solution of (6.2) by definition), the demand data must be genuine probability densities with bounded mean, and the levels may not be S=+∞.
A complete development needs: monotone convergence for the path-measure costs, measurable selection of minimizers, the K-convexity closure properties (Proposition 4.1 of the paper; several are proved on the platform as BertsekasDP.kconvex_*), and Proposition 4.2. The model file and the infinite-horizon verification argument are reusable for other Markov-modulated inventory models. Contributions toward any milestone, or toward general lemmas on positive-cost dynamic programs, are welcome.
Selected references
- S. P. Sethi and F. Cheng, Optimality of (s, S) policies in inventory models with Markovian demand, Operations Research 45(6), 931–939, 1997. https://doi.org/10.1287/opre.45.6.931
- J.-S. Song and P. Zipkin, Inventory control in a fluctuating demand environment, Operations Research 41(2), 351–370, 1993. https://doi.org/10.1287/opre.41.2.351
- H. Scarf, The optimality of (S, s) policies in the dynamic inventory problem, in Mathematical Methods in the Social Sciences, Stanford University Press, 1960.
- D. L. Iglehart, Optimality of (s, S) policies in the infinite horizon dynamic inventory problem, Management Science 9(2), 259–267, 1963. https://doi.org/10.1287/mnsc.9.2.259
- D. P. Bertsekas, Dynamic Programming and Stochastic Control, Academic Press, 1976.
- A. Bensoussan, M. Crouhy and J.-M. Proth, Mathematical Theory of Production Planning, North-Holland, 1983.
- D. Beyer, S. P. Sethi and M. Taksar, Inventory models with Markovian demands and cost functions of polynomial growth, Journal of Optimization Theory and Applications 98, 1998 (cited by Sethi and Cheng as forthcoming).