Speculative Decoding Latency Model: A Practical Framework for Understanding LLM Inference Acceleration in Production Environments

This paper proposes an interpretable speculative decoding latency model. Using Little's Law to infer the effective batch size, it decomposes request latency into load-independent and load-dependent components across prefill, draft generation, and verification stages. It explains why the acceleration effect of speculative decoding weakens as server load increases and provides guidance for production environment configuration. This model fills the gap in existing research that ignores the dynamic characteristics of systems, helping engineers scientifically configure parameters to improve LLM inference performance.

Ideal vs. Reality of Speculative Decoding

Speculative decoding achieves acceleration by using small models to generate candidate tokens and large models to verify them. While it shows significant effects in laboratory environments, its performance in production environments falls far short of expectations due to factors like dynamic request loads and batch processing changes.

Limitations of Existing Research

Existing studies focus on algorithm improvements and isolated performance evaluations, assuming fixed batch sizes or ignoring system dynamic characteristics. Their conclusions are difficult to directly apply to production deployments, leaving engineers in a dilemma of either conservative or aggressive parameter configuration.

Effective Batch Size Inference Based on Little's Law

Using Little's Law from queuing theory (average number of requests in a steady-state system = arrival rate × service time), we infer the effective batch size from observed request arrival rates and system latency. This approach is applicable to various service architectures.

Latency Decomposition

We decompose request latency into three stages: prefill, draft generation, and verification. Each stage is further divided into load-independent (basic computing cost) and load-dependent (resource competition, scheduling overhead, rollback cost, etc.) components. This explains why acceleration weakens with load: under high load, load-dependent components dominate, while speculative decoding mainly optimizes load-independent costs.

Experimental Validation

Validated using the vLLM framework, covering dimensions such as model size, sequence length, request rate, draft length, and acceptance probability. Results show that the model's prediction error is within an acceptable range, successfully explaining phenomena like optimal draft length and the nonlinear impact of model size ratios.

Extension to MoE Models

The framework is extended to Mixture-of-Experts (MoE) models, introducing concepts of expert activation probability and effective service cost. Analysis shows that speculative decoding gains are closely related to acceptance rate and the degree of expert load balancing; uneven expert distribution reduces acceleration effects.

Research Significance

Establishes a systematic way of thinking to analyze the behavior of speculative decoding in production environments. By decomposing complex system behavior into interpretable components, it helps engineers understand phenomena and make informed configuration decisions.

Future Outlook

Extend to complex strategies like tree-based speculation and adaptive speculation; consider heterogeneous hardware environments; integrate online learning to achieve automated configuration optimization.

1. Dynamic Draft Length Adjustment: Adjust in real time based on acceptance rate and current load, using the optimal formula provided by the model.
2. Load-Aware Model Selection: Use small drafters under light loads; configure the validator-drafter ratio conservatively under heavy loads.
3. Capacity Planning: Predict system capacity requirements under different loads to assist hardware investment decisions.
4. Performance Monitoring: Include effective batch size and the proportion of latency in each stage in the monitoring system to detect anomalies in a timely manner.