SSD: An LLM Inference Acceleration Scheme Based on Speculative Decoding

The SSD project accelerates large language model (LLM) inference by executing speculative decoding in parallel without compromising output quality, addressing the serial bottleneck of autoregressive token generation. This scheme provides an efficient text generation solution for resource-constrained scenarios such as local deployment and edge computing, with core advantages including parallel verification optimization, adaptive speculative length, and improved memory efficiency.

LLM inference has prominent latency issues due to the serial nature of autoregressive token-by-token generation, affecting real-time interaction experiences. Traditional hardware acceleration (e.g., GPU upgrades) is costly and has diminishing marginal returns; speculative decoding at the algorithm level has become a new direction, and the SSD project is a practical exploration in this area.

Speculative decoding uses a smaller and faster draft model to generate candidate tokens in parallel, then verifies them with the target large model:

1. The draft model generates K candidate tokens;
2. The large model verifies the correctness of the candidate tokens in parallel;
3. Accept consecutive correct tokens and repeat the process. SSD optimizations: Parallel verification mechanism reduces overhead, adaptive speculative length dynamically adjusts the K value, and memory management controls resource usage.

SSD can achieve 1.5-3x lossless acceleration (output distribution is consistent with the original model). Applicable scenarios include:

• Local deployment: Consumer-grade GPU/CPU improves response speed;
• Edge devices: Meets real-time requirements in resource-constrained environments;
• High-throughput services: Increases single-card processing capacity and reduces costs;
• Interactive applications: Low-latency requirements such as chatbots and code completion.

SSD provides an executable program for the Windows platform. System requirements: Win10+, 4GB memory, 2GHz processor. Developers can adjust parameters such as speculative length and batch size, support common input formats, and easily integrate into existing inference pipelines.

Speculative decoding requires attention to:

• The draft model needs to balance speed and accuracy;
• Running two models increases memory overhead;
• Better performance for structured outputs (e.g., code), but lower accuracy for open-ended text;
• High implementation complexity, requiring handling of model synchronization and state management.

Comparison with other technologies:

• Quantization: Reduces precision, can be stacked with SSD;
• Pruning: Removes parameters, may affect quality;
• KV cache optimization: Reduces memory access, complementary to SSD;
• Continuous batching: Improves GPU utilization on the server side, different dimension. Future directions: Multi-model speculation, learning-based speculation, hardware co-optimization, built-in speculation features in model architectures.

SSD represents a beneficial attempt to optimize LLM inference at the algorithm level, improving efficiency through "smarter computing" and providing solutions for resource-constrained environments. As the technology matures, speculative decoding is expected to become one of the standard configurations for LLM inference.