Hybrid Verified Decoding: A New Paradigm for Speculative Decoding Acceleration in Agent Workflows

This article introduces Hybrid Verified Decoding (HVD), an optimized speculative decoding method for agent workflow scenarios. By learning to predict the expected acceptance length of cached drafts, it dynamically selects verification strategies (cached drafts or model drafters), solving the problem of uncertain benefits from parameter-free drafts. Experiments show that this method achieves an average speedup of 2.73x compared to EAGLE3 in agent workflow scenarios, providing a new path for optimizing LLM inference latency.

The core bottleneck of LLM inference lies in the serial nature of autoregressive decoding, leading to linear latency growth when generating long texts. Speculative decoding breaks this seriality via the "draft + verification" approach, but existing solutions have limitations: model-driven drafting requires additional training, and parameter-free drafts (e.g., cache matching) have uncertain benefits in agent workflows—cached drafts may not match later, leading to wasted verification overhead.

The core of Hybrid Verified Decoding is the introduction of a benefit predictor to dynamically select verification strategies: when the expected acceptance length of a cached draft is above a threshold, verify the cache; otherwise, switch to the model drafter. The benefit predictor is trained via supervised learning, with input features including cache matching length, contextual semantic features, and historical verification statistics, and its inference overhead is negligible.

In evaluations using 3 mainstream LLMs and 16 datasets, Hybrid Verified Decoding performs exceptionally well in agent workflow scenarios: it achieves an average speedup of 2.73x compared to EAGLE3, outperforming EAGLE3 in all settings with a maximum speedup exceeding 3x; the advantage is consistent across models of different sizes—smaller models have larger benefit spaces, while larger models utilize resources more efficiently.

The analysis reveals: 1. Fixed prompt structures (e.g., instruction templates) in agent workflows create numerous caching opportunities; 2. High-benefit cached drafts are concentrated in specific regions and easily identified by the predictor; 3. Dynamically selecting draft sources is more effective than fixed strategies, as it can adapt to the generated context in real time.

Implications: 1. Runtime draft selection is a new frontier in speculative decoding; 2. Lightweight predictors can significantly improve performance even with moderate accuracy; 3. There is large room for scenario-specific optimization. Deployment considerations: Need to maintain caches and model drafters; predictors need regular retraining to adapt to distribution shifts; pay attention to cumulative overhead under extremely high throughput.

Hybrid Verified Decoding represents an important step in the evolution of speculative decoding from single optimization to intelligent scheduling. It provides a feasible path for optimizing inference latency in agent workflows (the fastest-growing area of LLM applications), and runtime draft selection is worthy of further exploration.