SpecKV: How Adaptive Speculative Decoding Dynamically Selects Optimal Speculation Length Based on Model Compression Level

SpecKV proposes a lightweight adaptive controller that dynamically selects the optimal speculation length γ based on the confidence and entropy signals of the draft model. It achieves a 56% inference speedup with almost no additional overhead, and is particularly suitable for model compression scenarios.

Inference acceleration for Large Language Models (LLMs) is a core challenge in AI engineering. Speculative decoding reduces the number of calls to large models via draft models, but existing fixed γ strategies (e.g., γ=4) have limitations: they cannot adapt to sensitivity differences across different task types, and struggle to fit when token acceptance patterns change after model quantization and compression.

The SpecKV team found that the confidence and entropy of the draft model are strongly correlated with the token acceptance rate (correlation coefficient ~0.56). Based on this, they designed a lightweight Multi-Layer Perceptron (MLP) controller that can select the optimal γ value in real time. The controller's training data covers 4 task categories, 4 speculation lengths, and 3 compression levels (FP16, INT8, NF4), totaling 5112 step-level records.

The SpecKV controller is lightweight, adding only 0.34 milliseconds of overhead per decision (accounting for less than 0.5% of single-step time). Compared to the baseline method with fixed γ=4, it achieves a 56.0% performance improvement, which is statistically significant (p < 0.001, paired bootstrap test). This strategy is particularly suitable for model compression scenarios, as it can sense the impact of compression levels on acceptance patterns and adjust dynamically.

SpecKV provides a plug-and-play optimization solution for LLM service providers and edge deployment developers, without the need to modify the underlying model architecture or rely on specific hardware. The research team has open-sourced all analysis data, trained models, and experiment notes to facilitate community reproduction. On resource-constrained edge devices, its adaptive capability can optimize based on real-time input features to enhance user experience.

SpecKV's research shows that the optimization space of speculative decoding is not fully explored, and significant performance improvements can be obtained through a simple adaptive control mechanism. This work reveals the value of internal signals from draft models. As LLM deployment scenarios diversify, adaptive technologies like SpecKV will become standard components in the inference stack.