From Imitation to Collaboration: CoSpec Redefines the Speculative Decoding Paradigm

CoSpec proposes a collaborative speculative decoding method that trains an arbitration strategy via reinforcement learning. When the draft model and target model diverge, it intelligently selects tokens that are more likely to lead to the correct answer, maintaining inference acceleration effects while outperforming the performance of a single target model. This method breaks the limitation of traditional speculative decoding which treats the target model as the sole authority, enabling collaboration rather than imitation between models.

Speculative Decoding (SPD) is an important technology for accelerating large model inference. It uses a draft model to quickly generate candidate tokens, and the target model verifies them in parallel. If they agree, the tokens are accepted to reduce latency. However, the traditional paradigm assumes that the target model is always superior, ignoring the fact that the draft model can instead lead to the correct answer in some token divergence scenarios. Blindly rejecting divergent tokens may discard better options.

CoSpec introduces a reinforcement learning-driven arbitration strategy that evaluates the context to decide whether to accept the draft or target token; training data comes from correct choices at divergence points retrieved from the validation set; the arbitration model is a lightweight network that is compatible with existing SPD frameworks without modifying the original models. This method achieves a paradigm shift from imitation to collaboration, integrating the advantages of both models.

Experiments show that CoSpec retains the acceleration advantages of speculative decoding and its output quality surpasses that of a single target model; the arbitrator can prioritize draft tokens in divergence scenarios where the target model is overconfident or has systematic biases, verifying the collaborative effect of "1+1>2".

CoSpec challenges the simple correspondence between the performance of large and small models, revealing the complementarity of model capabilities; it demonstrates the value of dynamic routing in multi-model systems; the arbitration training process provides a more reliable perspective for uncertainty quantification.

Deployment requires balancing arbitration accuracy and computational overhead by using lightweight arbitration models; it is necessary to fine-tune the arbitration strategy for professional domains; in some scenarios, interpretable arbitration mechanisms need to be developed.

CoSpec represents an important progress in the field of speculative decoding, moving from imitation to collaboration; its direction of intelligent arbitration and collaborative integration may become the standard paradigm for future multi-model inference, emphasizing the value of complementary advantages of multiple models.