SpecFed: A Federated LLM Inference Acceleration Framework Combining Speculative Decoding and Compressed Transmission

SpecFed is a federated LLM inference acceleration framework that combines speculative decoding and compressed transmission, aiming to solve the communication bottleneck of federated inference in edge computing. Its core innovations include introducing speculative decoding for parallel processing, and adopting Top-K compressed transmission and server-side reconstruction strategies to significantly reduce communication overhead while maintaining high generation fidelity.

Federated inference alleviates the computational pressure on a single device by distributing model inference and aggregating results, but the autoregressive nature of LLMs brings two major challenges:

1. Frequent full forward propagation: Each new token requires worker nodes to perform complete inference, limiting decoding throughput;
2. Communication bottleneck: Each worker node needs to transmit a token probability distribution of tens of thousands of dimensions, which becomes the main source of end-to-end latency.

SpecFed introduces speculative decoding into the federated scenario to achieve parallel processing:

• Principle: A lightweight draft model generates candidate sequences, and a large target model verifies them in parallel;
• Federated adaptation: Each worker node independently generates drafts, and the central server verifies and aggregates the results to improve overall throughput.

To alleviate the communication bottleneck, SpecFed adopts Top-K compressed transmission:

• Compression strategy: Worker nodes only transmit the K tokens with the highest probabilities and their probability values, reducing from tens of thousands of dimensions to K dimensions with a compression ratio exceeding 100x;
• Server reconstruction:

1. Uniform diffusion: The remaining probability is evenly distributed to the unselected tokens;
2. Temperature scaling: Scale the Top-K probabilities based on the temperature parameter to infer the overall distribution.

The robustness of SpecFed is verified through theoretical analysis in three aspects:

1. Local reconstruction error: The error introduced by Top-K compression is bounded under mild assumptions;
2. Aggregation bias: The bias after aggregating compressed probabilities still maintains reasonable statistical properties;
3. Acceptance rate bias: Compressed transmission does not significantly reduce the acceptance rate of speculative decoding, ensuring the acceleration effect.

Experiments evaluate generation fidelity, communication overhead, and end-to-end latency in federated edge scenarios:

• High fidelity: The generation quality has no significant difference from the uncompressed baseline;
• Communication reduction: Overhead is reduced by several orders of magnitude, reducing bandwidth requirements and transmission latency;
• End-to-end acceleration: The combination of speculative decoding and compression improves throughput, making edge deployment more practical.

Current limitations:

1. The K value needs to balance compression ratio and fidelity, depending on specific tasks;
2. Fixed K value is not dynamically adjusted;
3. The unified strategy is not optimal under heterogeneous networks. Future directions:
4. Adaptive K value;
5. Hierarchical compression;
6. Integration with secure aggregation;
7. Collaborative optimization with model parallelism.