Multi-Token Prediction Inference Acceleration: A Cross-Engine and Cross-GPU A/B Testing Benchmark Study

This article introduces a reproducible benchmark framework based on the Modal cloud platform for evaluating the effectiveness of Multi-Token Prediction (MTP) inference acceleration methods on small language models. It supports comparative testing of the transformers and vLLM dual engines across various GPUs such as A10, A100, H100, and B200. The core finding is that MTP performance is highly correlated with GPU type, inference engine, and prompt type—there is no simple "effective" or "ineffective" conclusion; it needs to be judged based on specific scenarios.

Multi-Token Prediction (MTP) is a speculative decoding technique. Its core idea is to predict multiple subsequent tokens when generating each token—if accurate, it reduces decoding steps and improves throughput, but requires additional computation; if accuracy is low, it increases overhead. There are controversies in the industry regarding its effectiveness: one side believes it can significantly accelerate, while the other argues that the benefits are limited or performance may degrade. This project aims to reveal the dependency of MTP performance on various factors through systematic A/B testing.

The project uses the Modal cloud platform to build a reproducible benchmark framework. The test objects are the Google Gemma 4 E2B-it model plus a draft model; it compares the dual engines of transformers (basic inference) and vLLM v0.21.0 (high-throughput optimization); covers GPUs such as A10, A100-80GB, H100, B200 (note: possible typo in original text); designs three types of prompt scenarios: general, code, and structured, to verify the impact of different tasks on MTP benefits.

Core conclusion of the project: MTP performance ratio depends on the combination of engine, GPU, and prompt. Engine differences: vLLM's PagedAttention interacts complexly with MTP's memory access pattern, while the transformers implementation is more straightforward; GPU differences: high-performance GPUs can quickly complete additional computations, leading to more obvious benefits; prompt differences: code/structured scenarios have high prediction accuracy, so MTP benefits are significant, while general scenarios have limited benefits.

The project uses a modular design, with the main module being multi-token-prediction/. Future plans include adding optimizations like dflash/. Usage steps: clone the repository → configure HF_TOKEN and MODEL_API_KEY → sync dependencies with uv → initialize Modal → run A/B tests on specified GPUs. Results are saved in the metrics/runs/ directory, with each test marked by a timestamp and traceable JSON files to ensure reproducibility.

This project is not a general service framework (Gemma model is hard-coded; modifying deploy/modal/*.py is required to test other models); it does not claim that speculative decoding is universally effective—its core conclusion is that effectiveness is highly dependent on specific contexts.

Application recommendations: Code generation and structured output scenarios are suitable for MTP (high prediction accuracy); open-ended text generation has limited benefits; hardware selection can refer to cross-GPU comparison data; engine selection: vLLM has good throughput performance, while transformers are more stable and easier to debug.

This project reveals the real performance characteristics of MTP through rigorous A/B testing. Its core contribution is proving the context dependency of its effectiveness—this nuanced conclusion is more valuable for engineering decisions. In the iteration of AI technology, empirical and reproducible research is particularly precious, reminding us to treat new technologies carefully, verify hypotheses through experiments rather than blindly follow hype.