PipeSD: A Speculative Decoding Acceleration Framework for Cloud-Edge Collaborative Inference

PipeSD is a speculative decoding acceleration framework designed for cloud-edge collaborative inference scenarios. Its core uses a pipeline scheduling mechanism and a Bayesian optimization-based validation triggering strategy to address the issues of low resource utilization and improper validation timing in existing cloud-edge collaborative speculative decoding. It achieves up to 2.16x end-to-end speedup and 25.3% energy reduction, suitable for edge computing, privacy-sensitive applications, and other scenarios.

With the popularization of Large Language Model (LLM) applications, inference deployment is evolving toward cloud-edge collaboration, which offers advantages such as reducing cloud load, supporting offline operation, and enhancing data privacy. However, edge resources are limited, making efficient inference a key challenge. Speculative decoding technology improves speed by having edge draft models generate candidate tokens and cloud target models perform parallel validation, but existing frameworks have two major bottlenecks: 1. Serial processing (generation-transmission-validation-return) leads to low resource utilization; 2. Fixed-threshold validation triggering strategies are inflexible—both early and late validation affect efficiency.

PipeSD proposes two innovations to address the bottlenecks:

1. Dynamic Programming-based Token Batch Processing Pipeline Scheduling: The edge streams tokens while the cloud validates as it receives them. A dynamic programming approach optimizes the batch processing strategy (considering edge generation speed, network characteristics, cloud throughput, etc.), enabling overlap between computation and communication to maximize parallelism.
2. Dual-Threshold NAV Trigger + Bayesian Optimization Parameter Tuning: Introduces an upper threshold (to force validation and prevent excessive edge generation) and a lower threshold (to trigger validation when the edge is idle and avoid resource waste); integrates a lightweight Bayesian optimization parameter tuner that automatically adjusts threshold parameters during runtime to adapt to dynamic scenarios.

Validated on a real cloud-edge test platform (built with llama-cpp-python, PyTorch, FastAPI), testing two draft-target model pairs and four scenarios:

• Speedup effect: Achieves 1.16-2.16x end-to-end speedup compared to the optimal baseline;
• Energy optimization: Reduces energy consumption by 14.3%-25.3% (due to improved resource utilization and reduced unnecessary computation);
• Scenario adaptability: Maintains stable advantages under different network conditions, model sizes, and input lengths.

Technical significance and application value of PipeSD:

• Edge computing scenarios: Makes LLM deployment on resource-constrained devices more feasible, reducing cloud dependency;
• Privacy-sensitive applications: Processes sensitive data locally and only transmits intermediate results, balancing privacy and performance;
• Cost optimization: Energy reduction translates to operational cost savings and environmental benefits. Summary: PipeSD effectively solves the problems of cloud-edge collaborative speculative decoding, with significant speedup and energy optimization effects, which will help popularize edge AI applications.

Current limitations: Only two model pairs were evaluated; more model combinations need to be tested; the Bayesian optimization parameter tuner requires more aggressive strategies in extremely dynamic scenarios. Future directions: Extend to multi-edge device collaboration; integrate model compression techniques such as quantization and pruning; optimize for scenarios like real-time dialogue and code generation.