Throughput-Optimal Scheduling Algorithms for LLM Inference and AI Agents: Above Load b_max/t_{b_max} Every Scheduler Diverges, Below It Every Work-Conserving K-FCFS Scheduler Is Positive RecurrentResearch Paper
Motivation
A large language model (LLM) answers a request in two phases. In the prefill phase it reads the whole prompt; in the decode phase it generates the answer one token at a time, because each new token depends on all previous ones. Serving engines such as vLLM, SGLang and Sarathi-Serve run the GPU in iterations: each iteration processes one batch that may mix prefill tokens of some requests with single decode tokens of others, up to a token budget . Which tokens enter each batch is decided by a batch scheduling algorithm, and providers want to know how much traffic a given server can sustain and which scheduling rules achieve that.
Dai, Deng, Li and Peng (arXiv:2504.07347v3, 2026) cast this as a queueing problem: a two-class batch-service queue in which one batch couples the two classes through a shared budget, and a request contributes many tokens per batch in prefill but at most one in decode. Their Theorem 2 identifies the maximal throughput of a single server and shows that a broad class of practical schedulers attains it. The proof method is the fluid-limit approach to stability of stochastic processing networks (Dai 1995; Dai and Harrison, Processing Networks, 2020).
Setting
Time is divided into integer slots. A batch with token load (its number of tokens) takes
slots (eq. (4)). Here are integers and is a block size dividing (Assumption 1).
Requests are labelled in order of arrival. Request brings prefill tokens and decode tokens. The arrival counts of the slots are i.i.d. with mean , the sizes are i.i.d. with means , and the two sequences are independent.
The state at the start of a slot is
where is the remaining time of the current batch, the set of requests in the system, and the unprocessed prefill and decode tokens of request . When the algorithm picks subject to the following feasibility rules (eq. (6)):
- prefill before decode;
- ;
- ;
- total load .
The batch then runs for slots. When it finishes, requests with no tokens left depart, and new arrivals join at the end of every slot. The total number of unprocessed tokens is .
An algorithm is work-conserving if it forms a full batch () whenever the supply is at least . It is K-FCFS if, whenever a request receives a decode token, every request still in the system receives at least one token.
Formalization targets
Goal: Theorem 2
- If , then under every scheduling algorithm and from every initial state
- If , then for every work-conserving K-FCFS algorithm there is a finite set of label-free configurations whose hitting time has finite expectation from every initial state whose requests respect the size bounds :
Part 2 is the corrected form of the paper's "the DTMC is positive recurrent". The labelled state cannot recur, because each new request carries a new label.
Milestones
These follow the order of the paper's argument:
- The rate bound: for (proof sketch, p. 16).
- The cumulative processing bound (eq. (43), corrected).
- The strong law , almost surely (eq. (56)).
- Bounded overtaking in departure order under K-FCFS (Lemma 11, corrected to departures).
- Stability of the work-conserving fluid model (Proposition 15, with nonnegative fluid levels).
Significance
Theorem 2 gives a sharp stability threshold for a single LLM server: the arriving token workload per slot, , against the maximal processing rate of full batches. It shows that every work-conserving K-FCFS scheduler, a class containing Sarathi-Serve-style chunked-prefill schedulers, is throughput-optimal. Schedulers that do not mix prefill and decode tokens in one batch can lose throughput. The result is the base case for the paper's multi-server, multi-class and agent-network extensions.
The paper proves part (a) in a sketch and part (b) by the fluid-limit method, with a separate Lyapunov argument in Appendix C. No part of the result has a machine-checked proof. Several printed steps are false as stated:
- the strict monotonicity of the step-endpoint rates;
- the constant in eq. (43);
- the prefill clause of Lemma 11;
- Proposition 15 without nonnegativity;
- the recurrence of the labelled chain.
A formalization fixes the exact statements that hold and supplies a complete argument.
Difficulty
Part (a) reduces to the rate bound, a pathwise count of completed batches, and the strong law. Part (b) is where the difficulty lies. A Lyapunov drift on the total token count fails directly: while a batch is in progress no tokens are completed, so the one-slot drift is positive whenever . A drift argument must account for the batch in progress. The naive bound "every slot completes tokens on average when the system is large" also needs the supply condition (9) to hold. Requests in decode contribute at most one token per batch, so many tokens can sit in few decode-phase requests. The fluid-limit route needs bounded overtaking, and the paper's prefill version of that is false.
Formalization scope
The Lean development (namespace LLMThroughput.TokenBudget) commits to the following:
- Slots and labels are 0-based, and are natural numbers with , , and .
- The state is a list of requests (label, , ) in label order, together with and the next label.
- A policy is any function from states to batch configurations. It must be feasible (and, where assumed, work-conserving and K-FCFS) at every admissible state.
- Feasibility adds .
- The dynamics is a pathwise recursion driven by sequences and .
- The randomness is a probability space with measurable, i.i.d., mutually independent sequences and integrable .
- The expectation of a hitting time is the Lebesgue integral of an -valued time, so a time that is infinite with positive probability has infinite expectation.
- Positive recurrence is stated for the label-free configuration, with the finite set chosen before the initial state. In part 2 the initial requests obey the source's size bounds , ; part 1 allows any admissible initial state.
A finite set is essential. The set of all states with small token count is infinite once labels are included, and hitting it would be a much weaker claim. Quantifying over a single named policy instead of all work-conserving K-FCFS policies would also change the theorem.
A complete development needs:
- sample-path lemmas for the recursion (admissibility is preserved, tokens are conserved);
- the strong law from Mathlib (
ProbabilityTheory.strong_law_ae); - a Foster–Lyapunov criterion for a process that is not a Markov kernel on the policy's input type;
- for the fluid route, Lipschitz calculus for the fluid model.
The pathwise model and the bounded-overtaking lemma can be reused for other batch-service systems. Proofs of any milestone, and alternative proofs of part (b) (a direct drift argument on an augmented workload), are welcome.
Selected references
- J. G. Dai, Y. Deng, Z. Li, Z. Peng, Throughput-Optimal Scheduling Algorithms for LLM Inference and AI Agents, arXiv:2504.07347v3, 2026. https://arxiv.org/abs/2504.07347
- J. G. Dai, On positive Harris recurrence of multiclass queueing networks: a unified approach via fluid limit models, Annals of Applied Probability 5(1), 1995. https://doi.org/10.1214/aoap/1177004828
- J. G. Dai, J. M. Harrison, Processing Networks: Fluid Models and Stability, Cambridge University Press, 2020. https://doi.org/10.1017/9781108772662
- A. Agrawal et al., Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve, OSDI 2024. https://arxiv.org/abs/2403.02310