Queueing Theory-Based Stability Analysis Framework for LLM Inference: Addressing Dual Constraints of GPU Memory and Computation

This article proposes the first queueing theory framework that simultaneously incorporates both computational resources and KV cache memory constraints, providing theoretical guidance for GPU cluster configuration in LLM inference services and addressing system stability and capacity planning issues. The framework can accurately determine whether the system is stable under load, helping operation and maintenance personnel balance costs and service quality.

LLM inference is constrained by both computational power and KV cache memory; KV cache becomes a bottleneck as sequence length and concurrent requests increase. Traditional methods treat computation and memory independently, lacking a unified framework to guide system design, leading to either over-provisioning (wasting costs) or under-provisioning (reducing service quality). Existing work rarely analyzes from a stability perspective whether the system can sustain the load (whether the queue is bounded).

This study proposes the first queueing theory framework that considers both computational and GPU memory constraints simultaneously. The core innovation is establishing stability conditions that integrate factors such as request arrival rate, service rate, KV cache memory usage, and GPU memory capacity, deriving formulas for the minimum service rate required to maintain stability and cluster size configuration. This framework provides a scientific basis for GPU cluster capacity planning, avoiding empirical trial and error.

Experiments in real GPU environments show that the deviation between theoretical stability conditions and actual observations is ≤10%, verifying the framework's effectiveness. The experiments cover different load scenarios and model configurations; even with large fluctuations in request arrival rates, it can accurately predict the boundaries of system behavior, demonstrating the framework's engineering practicality.

The framework requires accurate estimation of the statistical characteristics of request arrival rates (average and volatility), service time distribution (influenced by model size, sequence length, and hardware), and KV cache dynamic management strategies. Deployment recommendations include calibrating parameters using historical monitoring data, considering load time-variability, and dynamically adjusting cluster size or implementing adaptive scheduling.

This study lays a theoretical foundation for the scientific management of LLM inference infrastructure. The framework is applicable to current Transformer architectures and can be extended to future architectures. Future research can explore multi-tenant resource isolation, heterogeneous GPU scheduling optimization, and integration with auto-scaling. This tool helps cloud service providers and enterprises balance costs and service quality.