DriftSched: An Adaptive QoS-Aware Scheduling Framework for Multi-Tenant GPU Inference

DriftSched is an adaptive QoS-aware scheduling framework for multi-tenant GPU inference, designed to solve load estimation errors caused by runtime Token drift. Its core mechanism is runtime Token drift compensation. Experiments show that the SJF strategy reduces median latency by 42% compared to FIFO. This article is sourced from an arXiv paper (ID: 2606.02982v1, published on June 2, 2026).

Multi-tenant GPU scheduling faces the challenge of accurately estimating the runtime cost of heterogeneous inference requests. The deviation between the actual output length and the estimated value at admission is called Runtime Token Drift, which leads to issues such as workload misclassification, queue imbalance, increased tail latency, and QoS degradation.

DriftSched includes four core components:

1. Workload Classification: Classify based on request features (input length, expected output length, tenant priority, etc.);
2. Token Budget Estimation: Estimate the required Token budget for requests at admission;
3. Tenant-Aware Queue Management: Support multi-tenant queue isolation and resource allocation;
4. Runtime Drift Compensation: Continuously monitor deviations and dynamically adjust budget estimates via adaptive bias correction (core innovation).

DriftSched evaluated five scheduling strategies:

• FIFO: Simple and fair but lacks optimization;
• Priority: Sorted by priority, which may cause starvation;
• Weighted: Allocate resources proportionally;
• SJF: Prioritize requests with the shortest expected runtime;
• Aging Priority: Dynamically adjust priority to prevent starvation.

Experimental results include:

1. Token drift is prevalent across different workload categories;
2. Adaptive bias correction significantly reduces estimation errors: MAE reduced by 38.8%, RMSE reduced by 40.5%;
3. The SJF strategy performs best: median end-to-end latency is 42% lower than FIFO, and P99 latency is reduced by 16% under continuous GPU competition.

DriftSched has three main contributions:

1. First systematic integration of runtime Token drift into multi-tenant LLM inference scheduling frameworks;
2. Proposed a runtime Token drift compensation mechanism to dynamically optimize budget estimates;
3. Provided a reproducible benchmark framework supporting QoS-aware scheduling evaluation.

Recommendations for LLM inference service providers:

1. Emphasize runtime feedback; static strategies struggle to handle dynamic loads;
2. Prioritize the SJF strategy, which performs excellently in LLM inference scenarios;
3. Establish continuous Token drift monitoring and adaptive tuning mechanisms.

DriftSched effectively solves load estimation errors in multi-tenant LLM inference through a runtime Token drift compensation mechanism. Adaptive bias correction reduces estimation errors by approximately 40%, and the SJF strategy reduces median latency by 42% compared to FIFO, providing technical support for building high-quality LLM inference services.