  High-demand LLM inference services (e.g., ChatGPT and BARD) support a wide
range of requests from short chat conversations to long document reading. To
ensure that all client requests are processed fairly, most major LLM inference
services have request rate limits, to ensure that no client can dominate the
request queue. However, this rudimentary notion of fairness also results in
under-utilization of the resources and poor client experience when there is
spare capacity. While there is a rich literature on fair scheduling, serving
LLMs presents new challenges due to their unpredictable request lengths and
their unique batching characteristics on parallel accelerators. This paper
introduces the definition of LLM serving fairness based on a cost function that
accounts for the number of input and output tokens processed. To achieve
fairness in serving, we propose a novel scheduling algorithm, the Virtual Token
Counter (VTC), a fair scheduler based on the continuous batching mechanism. We
prove a 2x tight upper bound on the service difference between two backlogged
clients, adhering to the requirement of work-conserving. Through extensive
experiments, we demonstrate the superior performance of VTC in ensuring
fairness, especially in contrast to other baseline methods, which exhibit
shortcomings under various conditions.
