A New Paradigm for Multimodal Model Inference Acceleration: A Comprehensive Review of Speculative Decoding Technology

This article introduces an open-source resource library that systematically organizes speculative decoding technologies for multimodal models, covering the latest research progress in fields such as vision-language models, large video models, and text-to-image generation, providing a comprehensive technical reference for researchers and practitioners. As an emerging inference acceleration technology, speculative decoding is addressing the inference latency issues of Multimodal Large Language Models (MLLMs) in tasks like visual understanding and video analysis, becoming a core challenge of concern to academia and industry.

As Multimodal Large Language Models (MLLMs) demonstrate strong capabilities in tasks such as visual understanding, video analysis, and cross-modal generation, their inference latency issues are becoming increasingly prominent. How to accelerate model inference while ensuring generation quality has become a core challenge jointly concerned by academia and industry. Speculative Decoding, as an emerging inference acceleration technology, is sparking a research boom in the multimodal field.

The core idea of speculative decoding is to solve the computational waste problem of large language models generating tokens serially, using a two-stage "draft-verify" strategy: first, a lightweight draft model quickly generates multiple candidate tokens or visual primitives, then the target large model verifies these candidates in parallel, thereby confirming multiple tokens in a single forward pass and reducing inference latency. In multimodal scenarios, since inputs and outputs include heterogeneous data such as images and text, efficient speculative decoding requires targeted algorithm design.

The open-source resource library classifies research into seven categories based on application scenarios:

1. Vision-Language Models: The most active field, with representative works including On Speculative Decoding for Multimodal Large Language Models (April 2024), Spec-LLaVA (ICML2025 Workshop), ViSpec (October 2025), HiViS (November 2025). The trend is to let the draft model focus on text generation while retaining visual perception capabilities;
2. Autoregressive Text-to-Image Generation: Exploring acceleration of visual token prediction;
3. Large Video Models: Focusing on candidate generation in the time dimension and content understanding;
4. Vision-Language-Action Models: Serving tasks like robot control, requiring acceleration of perception, understanding, and decision-making;
5. Speech and Audio: Supporting acceleration of speech recognition, synthesis, etc.;
6. Diffusion Models: Transferring speculative ideas to accelerate sampling;
7. Point Cloud Synthesis: Assisting 3D visual applications.

The technology evolution trends include:

• From General to Specialized: Shifting from migration from the text domain to specialized algorithms for specific modal combinations;
• Refinement of Draft Models: Improving multimodal context understanding capabilities while maintaining speed;
• Optimization of Verification Strategies: Exploring complex mechanisms such as tree-based parallel verification and adaptive verification depth;
• End-to-End Optimization: Jointly optimizing the entire "draft-verify" process.

The value of this resource library for practitioners:

• Technology Selection Guide: Compare methodologies and experimental results to choose the appropriate technical route;
• Implementation Reference: Use code links (if available) of included papers as a starting point for development;
• Trend Insight: Grasp the trajectory of technology evolution by sorting papers chronologically;
• Research Inspiration: Discover under-explored research directions.

Open challenges include:

• Modal Alignment: Designing a unified framework to handle differences in representation spaces and generation rhythms of different modalities;
• Quality-Speed Trade-off: Balancing acceleration and generation quality;
• Dynamic Adaptability: Dynamically adjusting the draft-verify strategy based on input complexity;
• Hardware Co-optimization: Combining specialized hardware (TPU, NPU) to unlock potential. In the future, more innovative methods are expected to enable multimodal models to serve practical applications with lower latency.

Multimodal speculative decoding is in a period of rapid development, and this open-source resource library provides a valuable entry point for tracking field progress. For developers aiming to improve the inference efficiency of multimodal models and scholars researching acceleration technologies, it is a reference material worth paying attention to. As multimodal AI applications penetrate, the importance of speculative decoding technology will become increasingly prominent.