DistillSpec Research Project: A Systematic Exploration of Speculative Decoding and Knowledge Distillation

Distill-Spec-Research is an experimental machine learning system research project released by Rmuk655 on GitHub on May 26, 2026, focusing on speculative decoding, knowledge distillation, and efficient LLM inference. The project adopts an "intentionally narrow" design philosophy (single research direction, baseline architecture, evaluation plan) to ensure experimental reproducibility. Its core innovation lies in improving the DistillSpec framework to address the alignment bottleneck between draft models and target models, thereby enhancing acceptance rate and speedup ratio.

The project's research is based on two major fields: speculative decoding and knowledge distillation. Speculative decoding was proposed at the end of 2022; it breaks through the autoregressive bottleneck by parallelly verifying candidate tokens generated by draft models, and subsequent tree-based speculative verification further improves the acceptance rate. Knowledge distillation is a classic technique for transferring knowledge from large models to small models. As a landmark work combining the two, DistillSpec proposes online distillation to optimize draft models to adapt to the characteristics of target models.

The project takes draft-target alignment as its primary problem, covering three aspects: 1. Distribution alignment: The standard language modeling objective cannot guarantee consistent output distributions between draft and target models, so specialized alignment functions need to be explored; 2. Block efficiency and acceptance rate: Optimize block-level acceptance rate instead of single-token accuracy; 3. Balance between latency and throughput: Trade off the inference overhead of draft models and the benefits of parallel computing to find the optimal configuration.

The experiments follow strict design principles: The basic framework is HuggingFace Transformers, with the Qwen3 series as the main experimental models; Weights & Biases are used to record hyperparameters, metrics, etc., to ensure reproducibility; The evaluation adopts a benchmark-driven approach, adhering to the principles of "correctness first" and "reproducibility first".

The repository structure is clearly layered: docs (research scope, principles, plans), src (core code), experiments (configurations and outputs), scripts (auxiliary scripts), logs (experimental notes). Core development principles: Correctness first, reproducibility first, focused scope, strict benchmarking, and rejection of uncontrolled exploration.

Theoretically, an in-depth understanding of draft-target alignment helps reveal the essential laws of LLM inference; Practically, efficient speculative decoding can reduce the latency and cost of LLM services in production environments. As the scale of LLMs grows, the problem of inference efficiency becomes increasingly prominent, and the alignment optimization methods explored by this project are expected to become an important part of the next-generation inference systems.