Strong and Weak Equilibria for Time-Inconsistent Stochastic Control in Continuous Time 3: Discrete-Time Equilibria Exist on an Infinite Horizon for Closed Convex Admissible Transition RowsResearch Paper
Motivation
In many dynamic decision problems the criterion an agent uses today is not the one it will use tomorrow. The standard example is non-exponential discounting: if a payoff received periods from now is weighted by and is not geometric, then the plan that is optimal from today's point of view will be abandoned by the agent's future selves. Such problems are called time-inconsistent, and since Strotz (1955) the usual response is to look not for an optimal plan but for an equilibrium: a plan that no future self wants to deviate from, given that all later selves follow it.
Huang and Zhou (arXiv:1809.09243v3, Math. Oper. Res. 2021) study this for finite-state Markov chains, mainly in continuous time. Section 5 of their paper turns to discrete time. On a finite horizon a discrete-time equilibrium is built by backward induction; on an infinite horizon there is no last period to start from, and, as the authors note, "little is known about the existence and characterization of equilibria, even in discrete time" (p. 15). Their Theorem 5.1 gives a general existence result for this case. This mission formalizes it.
Setting
Let and . A time-homogeneous Markov chain on is controlled by choosing its transition matrix ; the -th row is the distribution of given . Rows live in the probability simplex
For each state a set of admissible rows is given, and .
A payoff is continuous, and is the reward periods from now when the chain is at and uses row . The dependence on , the time difference from the present, is what makes the problem time-inconsistent. It is assumed that
The value of from state is , and is the same with the clock shifted by one.
For the concatenation uses for the step at time and for every later step. A matrix is an equilibrium (Definition 5.1) if
the self at time cannot gain by a one-step deviation, given that all later selves use .
Formalization targets
Goal: Theorem 5.1 (p. 16)
If every is nonempty, convex and closed and every is concave on , then
Milestones (proof of Theorem 5.1, Appendix B.1)
- (5.4), p. 15: the one-step decomposition
- Continuity of , p. 27: for each , is continuous on .
- Best response, p. 27: the correspondence has nonempty, closed, convex values and is upper semicontinuous on .
- Kakutani's fixed-point theorem (already proved on the platform, referenced).
Significance
The result. Theorem 5.1 says that infinite-horizon, discrete-time, finite-state time-inconsistent control always admits an equilibrium under convexity and concavity alone: no discounting structure, no stationarity of in , and no compactness beyond what provides. Remark 5.2 of the paper shows that concavity of cannot simply be dropped: with strictly convex time- payoffs, a two-state example has no equilibrium. The discrete-time model is also the bridge to the continuous-time theory: Theorem 5.2 of the same paper (a companion mission) shows that limits of equilibria of time-discretizations are weak equilibria in continuous time.
Formalizing it. The result is proved on paper in one page. The formalization produces a machine-checked existence theorem whose definitions make the probabilistic content explicit (the value is written through the marginals of the chain), together with a reusable decomposition (5.4) and continuity of the shifted value in the transition matrix. None of this is formalized elsewhere on the platform; Kakutani's theorem, the one external ingredient, is.
Difficulty
Backward induction, the standard construction of discrete-time equilibria, has no starting point on an infinite horizon, so existence cannot be obtained by a recursion. The value is an infinite sum of products of matrix powers with , and its regularity in the transition matrix is not automatic: each term is continuous, but passing limits through the sum, and exchanging it with the finite sum over states in (5.4), needs the uniform tail control of (5.2) combined with the stochasticity of every power . Without concavity of the statement fails (Remark 5.2, p. 16, gives a two-state counterexample), so any argument must use it. Finally, the referenced fixed-point theorem is stated on Euclidean space, while the controls are matrices.
Formalization scope
- States are
Fin N(state of the paper is index ); rows areFin N → ℝ; transition matrices areMatrix (Fin N) (Fin N) ℝwith the product topology. - Expectations are written through the one-dimensional marginals of the chain: , , and . This is Fubini plus the marginal laws of a finite chain; no Markov process is constructed. The concatenated value is defined from the process, not by (5.4), so (5.4) is a genuine statement.
- The infinite sums are
tsums. Atsumof a non-summable family is in Lean; here (5.2) and the stochasticity of make every sum absolutely convergent, and no summability hypothesis appears in any statement. - The standing assumptions of §5 are one structure: ; continuous on for all (continuity on , both first factors being discrete); (5.2) as a summable majorant on .
- Added hypothesis: every is nonempty. The page's "convex and closed" allows , in which case and no equilibrium exists.
- Concavity of is assumed on all of , as on the page; closedness of gives compactness because is compact, and no compactness is assumed.
- Upper semicontinuity of is stated with sequences, as on the page.
- A trivializing formalization would define by formula (5.4), or add summability or equilibrium-shaped hypotheses to the goal; none of these is done.
- Infrastructure a complete development needs: bounds on entries of products of stochastic matrices, dominated (Weierstrass M-test) continuity of
tsums, a linear homeomorphism betweenMatrix (Fin N) (Fin N) ℝandEuclideanSpace ℝ (Fin (N*N))to apply the referenced Kakutani theorem. The first two are reusable for any finite-state Markov reward model. Proofs of the milestones in any order are welcome.
Selected references
- Y.-J. Huang and Z. Zhou, Strong and Weak Equilibria for Time-Inconsistent Stochastic Control in Continuous Time, Math. Oper. Res. 46(2), 2021; read as arXiv:1809.09243v3. https://arxiv.org/abs/1809.09243 , https://doi.org/10.1287/moor.2020.1066
- S. Kakutani, A generalization of Brouwer's fixed point theorem, Duke Math. J. 8(3), 457–459, 1941. https://doi.org/10.1215/S0012-7094-41-00838-4
- R. H. Strotz, Myopia and inconsistency in dynamic utility maximization, Rev. Econ. Stud. 23(3), 165–180, 1955. https://doi.org/10.2307/2295722