FASER: A Fine-Grained Speculative Decoding Optimization System for Dynamic LLM Inference

FASER is a fine-grained speculative decoding system optimized for dynamic LLM inference. It addresses the issues of insufficient GPU utilization at low loads and computational waste at high loads in traditional speculative decoding through fine-grained phase management and space reuse techniques. It achieves up to 53% throughput improvement and 1.92x latency reduction in vLLM, providing an efficient solution for LLM inference services.

Speculative Decoding (SD) is an important technique to accelerate LLM inference, whose core is to use a small draft model to generate candidate tokens and then verify them in parallel with the main model. However, traditional SD systems use coarse-grained management, fix the speculative token length, and execute draft and verification serially, which cannot adapt to dynamic traffic changes and leads to performance issues under different loads.

In low-load scenarios, the serial execution of traditional SD causes the verification phase to wait for the draft to complete, leaving the GPU idle and latency accumulating; in high-load scenarios, the fixed speculative length cannot be dynamically adjusted, leading to a large number of candidate tokens being rejected, and computational waste exacerbating congestion.

FASER has two major innovations: 1. Dynamic speculative length adjustment (adjusted independently per request based on historical acceptance rate) + early pruning (terminate subsequent verification if rejected during verification); 2. Phase overlap and space reuse (split verification into blocks, execute overlapping with the draft phase, share GPU resources with minimal interference).

A FASER prototype was implemented in the vLLM framework. Evaluation shows: up to 53% throughput improvement (handling more requests with the same hardware), up to 1.92x reduction in end-to-end latency (significant for response-sensitive scenarios), and performance gains come from refined resource management and scheduling.

FASER reveals that coarse-grained optimization is effective in static environments, but dynamic online services require fine-grained management. This concept has guiding significance for LLM service optimization, represents an important progress in the field of inference optimization, and provides a reference solution for engineers and researchers.