Kairu: A High-Performance Speculative Decoding Engine for HuggingFace Models

Kairu is an open-source speculative decoding engine designed specifically for HuggingFace models. It offers features such as EAGLE-style draft generation, dynamic early exit, and token budget control. It significantly improves the inference speed of large language models (LLMs) without sacrificing output quality, is compatible with the existing HuggingFace ecosystem, and supports real-time performance monitoring and cost control.

As the scale of LLMs continues to expand, inference latency has become a key bottleneck in practical deployment. Speculative decoding, as an emerging acceleration technology, achieves inference acceleration without reducing output quality by using a draft model to quickly generate candidate tokens and a target model to perform parallel validation. Kairu is an open-source practice in this field, bringing enterprise-level speculative decoding capabilities to the HuggingFace ecosystem.

1. EAGLE-style draft generation: Reuses intermediate layer features of the target model, eliminating the need to learn semantic representations from scratch, and achieves higher prediction accuracy with fewer parameters;
2. Dynamic early exit: Dynamically stops computation based on prediction confidence, reducing average inference costs when processing simple content;
3. Token budget control: Supports setting a maximum token consumption limit to avoid resource overspending;
4. Real-time performance monitoring: Provides key metrics such as throughput, acceleration ratio, and acceptance rate to help optimize the system.

Kairu adopts a modular design and is compatible with HuggingFace's generation interface, allowing existing projects to migrate at zero cost. The inference process includes:

1. Draft generation: The draft model quickly generates K candidate tokens;
2. Validation: The target model processes the draft sequence in parallel to compute the real probability distribution;
3. Acceptance decision: Determines the accepted tokens and rollback position based on probability ratios;
4. Iterative continuation: Starts the next round of generation from the accepted position. The validation phase minimizes overhead through optimized tensor operations.

Kairu适用于多种场景：

• Real-time dialogue systems: Reduce response latency and improve user experience;
• Batch text processing: Save computing costs;
• Edge device deployment: Reduce the number of forward passes to enable feasible inference;
• API service optimization: Improve concurrency or reduce infrastructure costs.

Kairu follows a permissive license that allows commercial use and can be directly installed via pip. For integration, simply replace HuggingFace's AutoModelForCausalLM with Kairu's wrapped class and configure the parameters. The project welcomes community contributions, including support for new models, optimization of draft training strategies, etc.

Kairu represents an important advancement in the field of LLM inference optimization, driving speculative decoding from academic research to production practice. As model scales grow and applications expand, inference efficiency will become a key competitive dimension, and mastering engineering solutions like Kairu will help enhance system competitiveness.