Motivation
A decision maker is time-inconsistent when the plan that is optimal today stops being optimal tomorrow. The standard source is non-exponential discounting: a reward t units of time ahead is weighted by δ(t), and unless δ is exponential the relative weight of two future dates changes as time passes. Hyperbolic and pseudo-exponential discount functions, which fit experimental and group-decision data better than exponential ones, all have this property. For such an agent "optimal control" is not a meaningful target, and the literature instead looks for equilibrium policies: plans that no current self wants to deviate from, given that all future selves follow the plan.
In discrete time the definition is unambiguous: a policy is an equilibrium if deviating at the current step alone does not help. In continuous time there is no "current step", and the usual definition, due to Ekeland and Lazrak (2006), compares the payoff of the plan with that of a deviation on a short interval [0,ε] through a first-order condition as ε↓0. Huang and Zhou (arXiv:1809.09243, Math. Oper. Res. 2021) call this a weak equilibrium, introduce a strong equilibrium that requires the deviation to be unprofitable for every small ε, and study both for a controlled finite-state Markov chain.
A natural question is whether continuous-time equilibria are the limits of the discrete-time ones as the time step shrinks. This mission formalizes the paper's answer (§5.1): limits of discrete equilibria are always weak equilibria. The paper's Example 5.1 shows they need not be strong.
Setting
The state space is S={1,…,N}. A continuous-time Markov chain on S is controlled through its generator Q=(qij): in state i the row Qi is chosen from an admissible set Di⊆Ei={q:qj≥0 (j=i), qi=−∑j=iqj}, and the control space is Q={Q:Qi∈Di ∀i}. A payoff function f(t,i,q) gives the reward at time-distance t from now, in state i, under row q. The standing assumptions are continuity of t↦f(t,i,q) on [0,∞) and, for each c>0, integrability over [0,∞) of supi,q∈Di,∥q∥≤c∣f(t,i,q)∣ (condition (2.3)). The expected payoff from state i is
F(i,Q)=Ei,Q[∫0∞f(t,Xt,QXt)dt]=∫0∞j∑(etQ)ijf(t,j,Qj)dt,
and F(Q)=(F(1,Q),…,F(N,Q)). With Q⊗εQ∗ the generator that uses Q on [0,ε] and Q∗ afterwards, Q∗∈Q is a weak equilibrium if liminfε↓0(F(i,Q∗)−F(i,Q⊗εQ∗))/ε≥0 for all i and Q∈Q. The first-order quantity is ΓQ∗(q)=f(0,i,q)+q⋅F(Q∗).
Discretization. Fix a mesh δn>0. A transition matrix u (rows in the probability simplex P) has generator Qu,n=(u−I)/δn, the admissible matrices are An={u:Qu,n∈Q}, and the discrete payoff is κn(k,i,α)=f(kδn,i,(α−ei)/δn)δn. The discrete value is Vn(i,u)=Ei,u[∑k≥0κn(k,Xk,uXk)], and un∈An is a discrete equilibrium if Vn(i,un)≥Vn(i,u⊗1un) for all i and u∈An, where u⊗1un uses u at time 0 and un afterwards. The continuation values are Hin(un)=Ei,un[∑k≥0κn(k+1,Xk,uXkn)]. Theorem 5.2 additionally assumes (5.8): for some T>0, t↦∣f(t,i,q)∣ is nonincreasing on [T,∞).
Formalization targets
Goal: Theorem 5.2
Let δn↓0, let un be an equilibrium of the problem with mesh δn, Qn=Qun,n, and suppose f(⋅,i,⋅) is continuous on [0,∞)×Di. If Qnk→Q∗∈Q along a subsequence, then
ΓQ∗(Qi∗)≥ΓQ∗(Qi)for all (i,Q)∈S×Q,and Q∗ is a weak equilibrium.
Milestones
- (5.12): an equilibrium un satisfies κn(0,i,uin)+Hn(un)⋅uin≥κn(0,i,ui)+Hn(un)⋅ui for all u∈An.
- Generator form (proof of Theorem 5.2): dividing by δn, the same inequality reads f(0,i,Qin)+H⋅Qin≥f(0,i,Qiu,n)+H⋅Qiu,n.
- Continuity of Q↦F(Q) on Q (proof of Theorem 3.3).
- Lemma 5.1: Hink(unk)→F(i,Q∗) along the subsequence.
Significance
The theorem justifies the weak-equilibrium notion from discrete time: whatever a time-inconsistent agent does on a fine time grid, the continuous-time limit satisfies the first-order condition (3.10). Combined with the paper's discrete existence result (Theorem 5.1) and compactness of Q, it yields weak equilibria by approximation (Remark 5.3). It also locates the gap between the two notions: the limit is weak but in general not strong.
The result is proved in the paper; nothing about it is open. No part of it has a machine-checked proof. A formal development adds a checked Euler-scheme comparison between (I+δQ)k and ekδQ for generator matrices, a dominated passage to the limit for Riemann sums of the payoff, and continuity of the payoff in the generator, each reusable for other finite-state continuous-time control problems.
Difficulty
The obvious argument is to divide the discrete one-step inequality (5.12) by δn and let n→∞. Its algebraic part is immediate. The difficulty is the convergence of the discrete continuation values Hn(un), which are infinite sums along a discrete chain with transition matrix un, to the continuous payoff F(Q∗), an integral along the continuous chain with generator Q∗. Three errors must be controlled at once: the discrete chain against the chain sampled from etQn (the difference of k-th powers grows with k≈M/δn), the Riemann sum against the integral (needs uniform continuity of f jointly in t and q), and F(Qn) against F(Q∗). The tail of the infinite sums is not controlled by (2.3) alone: integrability of a majorant does not make its samples on a grid summable, which is why (5.8) is assumed. A second subtlety concerns the last step: the paper passes from (3.10) to the weak-equilibrium property through Theorem 3.1, which assumes a condition (3.3) that Theorem 5.2 does not. The conclusion is still true without (3.3), but no numbered statement of the paper provides that step.
Formalization scope
- States are
Fin N; rows are Fin N → ℝ with the sup norm, generators and transition matrices are Matrix (Fin N) (Fin N) ℝ with the product topology.
- Expectations are written through marginals: F(i,Q)=∫0∞∑j(etQ)ijf(t,j,Qj)dt (Bochner integral) and Hin(u)=∑k∑j(uk)ijκn(k+1,j,uj) (
tsum). No Markov process is constructed.
- The weak-equilibrium liminf is encoded as "for every η>0, eventually as ε↓0 the quotient is ≥−η".
- "f(⋅,i,⋅) is continuous" is joint continuity on [0,∞)×Di; (5.8) is required for q∈Di; δn↓0 is positivity, monotonicity and convergence to 0; the subsequence is a strictly increasing φ:N→N.
- Not assumed: integrability of F, summability of Hn or (5.2) for κn (they follow from (2.3) and (5.8)), compactness of Di, and condition (3.3).
- The equilibria un, the generators Qn=(un−I)/δn and the subsequence are part of the goal's data. A version that only assumes "Qn→Q∗" for arbitrary Qn∈Q would be false and is ruled out.
- The continuity milestone is proved in the paper for compact Di; it is stated here without compactness, since only convergent sequences in Q matter.
- Needed infrastructure: bounds on ∥(I+δQ)k−ekδQ∥ for generator matrices, stochasticity of etQ for generator Q, dominated convergence for the payoff integral and for the discrete series. Contributions of these general lemmas are welcome.
Selected references
- Y.-J. Huang and Z. Zhou, Strong and Weak Equilibria for Time-Inconsistent Stochastic Control in Continuous Time, Math. Oper. Res. 46(2), 2021; read as arXiv:1809.09243v3. https://arxiv.org/abs/1809.09243 , https://doi.org/10.1287/moor.2020.1066
- I. Ekeland and A. Lazrak, Being serious about non-commitment: subgame perfect equilibrium in continuous time, tech. rep., University of British Columbia, 2006. http://arxiv.org/abs/math/0604264
- S. N. Ethier and T. G. Kurtz, Markov Processes: Characterization and Convergence, Wiley, 1986. https://doi.org/10.1002/9780470316658