MTP-D: Self-Distillation Boosts Multi-Token Prediction, Achieving 220% Inference Acceleration

MTP-D uses self-distillation to increase the acceptance rate of multi-token prediction heads by 7.5%, and its looped extension strategy achieves 220.4% inference acceleration compared to single-head MTP, providing new ideas for optimizing LLM inference efficiency.

As the scale of large language models expands, inference efficiency has become a key bottleneck. Multi-token prediction (MTP) accelerates inference by predicting multiple future tokens in parallel, but it faces two major challenges:

1. Limited acceptance rate of MTP heads
2. Difficulty in joint training of multiple MTP heads

Core Innovation: A simple and efficient self-distillation method

• Minimal additional training cost
• 7.5% increase in MTP head acceptance rate
• Maximally preserves main head performance

Introduce the looped extension strategy:

• Economically and efficiently expand MTP heads
• Achieve 220.4% inference acceleration compared to single-head MTP

Systematic exploration on seven benchmark tests:

• Key insights into distillation strategies
• Scalability potential of MTP

This work effectively improves the performance and inference efficiency of MTP heads, promoting the practical application of MTP in LLMs.