SMC-SD: A New Sequence Monte Carlo-Based Speculative Decoding Acceleration Method

This paper proposes the SMC-SD method, which addresses the 'all-or-nothing' bottleneck of traditional speculative decoding by replacing token-level rejection sampling with a Sequence Monte Carlo-based importance-weighted resampling strategy. Experiments show that this method achieves 2.36x acceleration over speculative decoding and 5.2x over autoregressive decoding, with accuracy loss controlled within 3%, providing an efficient and quality-controllable new path for LLM inference acceleration.

With the expansion of LLM application scenarios, the high latency of autoregressive inference has become a core challenge in deployment. Speculative Decoding (SD) accelerates inference through a combination of small and large models, but traditional SD uses strict rejection sampling: once a draft token is rejected by the target model, all subsequent tokens are discarded, leading to severe efficiency loss. Especially when the draft model's accuracy is limited, the acceleration effect is greatly reduced.

The key innovation of SMC-SD is to use a Sequence Monte Carlo-based importance-weighted resampling strategy to process draft tokens. It maintains a particle swarm (candidate token sequences), the target model evaluates particle weights in parallel, and then resamples to retain high-weight particles. This mechanism avoids the 'all-or-nothing' problem, and based on approximate inference theory, it has strict error bounds to ensure controllable output quality.

1. Parallel particle generation and scoring: Leveraging GPU parallel computing capability, the draft model generates multiple particles simultaneously, and the target model scores them in parallel without increasing memory bandwidth pressure; 2. Vectorized fixed-size operations: Convert verification into rollback-free vectorized operations to eliminate control flow divergence overhead; 3. Stateless resampling: Process the particle swarm independently at each step, simplifying implementation and facilitating distributed deployment.

Experiments show that SMC-SD performs excellently in multiple benchmark tests: 1. Acceleration effect: 2.36x over speculative decoding, 5.2x over autoregressive decoding; 2. Accuracy control: Compared with the target model's output, accuracy loss is <3%; 3. Cross-task stability: Maintains stable acceleration effects in tasks such as reasoning, instruction following, and programming.

Technical Advantages: High memory efficiency (uses idle computing units without increasing bandwidth pressure), simple implementation (core logic is clear and easy to deploy), good compatibility (no need to modify model architecture, can be integrated into existing inference frameworks).

Application Scenarios: Real-time interactive systems (low latency improves user experience), high-throughput services (reduces operational costs), edge device deployment (optimizes performance under limited computing power).

Limitations: Approximation errors may accumulate in extremely long sequence generation; the number of particles needs to balance speed and quality.

Future Directions: Explore error control strategies (such as periodic calibration), automated particle count adjustment mechanisms, combination with technologies like quantization/pruning, and deepen theoretical analysis of the statistical properties of the sequence generation process.

By introducing Sequence Monte Carlo methods to improve speculative decoding, SMC-SD achieves significant acceleration while maintaining output quality, with direct engineering value. It also demonstrates the application potential of classical statistical inference in deep learning. As the demand for LLM deployment grows, such efficient inference technologies will play an important role in AI infrastructure.