CSD: A New Method for Knowledge Distillation of Large Language Models via Concrete Score Matching

Concrete Score Distillation (CSD), proposed by the KAIST Artificial Intelligence Laboratory, is a research work accepted by ICLR 2026. To address the information loss problem of probability matching in traditional knowledge distillation, it proposes a method that directly performs score matching at the Logit level, achieving better distillation results while maintaining computational efficiency. Through pairwise Logit residual matching, this method retains more information from the teacher model, providing a new path for large language model compression.

Existing knowledge distillation mostly relies on operations in the probability space (e.g., KL divergence), but the additive constant invariance of the Softmax function leads to Logit information loss (different Logit vectors may map to similar probabilities); Direct Logit Distillation (DLD) uses MSE to match Logits, but it over-constrains (requiring absolute equality and ignoring translation invariance), limiting the solution space. These problems have driven the proposal of CSD.

CSD defines the 'Concrete Score' as the Logit residual between tokens (f[x] - f[y_t]), implemented via the pairwise residual matching loss function: $$ \mathcal{L}{\mathrm{CSD}}(\theta) = \frac{1}{2} \sum{y_t \in \mathcal{V}} \sum_{x \in \mathcal{V}} w(y_t, x) \left( f_\theta[x] - f_\theta[y_t] - f_T[x] + f_T[y_t] \right)^2 $$ This method does not require absolute equality of Logits, only matches relative differences, and ensures numerical stability through logarithmic transformation.

1. Logit-level operation: Retains more information from the teacher model and avoids probability conversion loss; 2. Respects translation invariance: The optimal solution set is a superset of DLD, offering higher optimization freedom; 3. Linear complexity: After mathematical transformation, the computational complexity is linearly related to the vocabulary size, making it suitable for large models; 4. Flexible design space: The weight function can adjust the fidelity-diversity trade-off (e.g., pattern finding/coverage).

CSD performs excellently across multiple models (GPT-2, OpenLLaMA, Gemma, etc., up to 7B parameters) and tasks: It achieves the highest ROUGE-L score in task-agnostic instruction following; Integrating with online strategies like ImitKD improves results; It shows strong performance in task-specific distillation (summarization, translation, GSM8K); It is highly competitive in general dialogue evaluations (MT-Bench, AlpacaEval).

The official implementation of CSD provides complete reproducibility scripts: task-agnostic distillation (scripts corresponding to Table1/2, Figure3/5), task-specific distillation (run_kd_train.py + yaml configuration), and general dialogue distillation (run_csd.py + yaml configuration). The README in each subdirectory contains setup instructions and dependency requirements.

Theoretically, it reveals the additional information capacity of the Logit space; Practically, it provides better results, flexible trade-offs, wide compatibility, and scalability; Domain implications: It encourages researchers to re-examine the optimality of probability matching and explore more refined knowledge transfer mechanisms.

Current limitations include the maximum validated scale being only 7B, insufficient theoretical characterization of optimal solutions, room for computational optimization for large vocabularies, and unvalidated multimodal extensions. Future directions can include validating larger models, in-depth theoretical analysis, improving computational efficiency, and adapting to multimodal scenarios.