RTP-LLM: In-depth Analysis of Alibaba's Open-Source High-Performance Large Model Inference Engine

RTP-LLM is a large language model inference acceleration engine developed by Alibaba's Foundation Model Inference Team. As a sub-project of Havenask, it undertakes the mission of large-scale LLM services within the group, and has been widely deployed in core businesses such as Taobao, Tmall, and Cainiao, and is open-sourced for developers. It features technical characteristics like high-performance CUDA optimization, multi-level quantization, and dynamic batching. Verified in production environments, it provides the community with a production-grade inference engine option.

RTP-LLM is an independently developed inference acceleration engine by Alibaba. As a sub-project of Havenask, it supports internal LLM services of the group and has been applied to multiple business units including Taobao, Tmall, Xianyu, and Cainiao. The version 0.2.0 was released in September 2025 with enhanced performance and upgraded features. Its design goal is to support diverse model architectures and deployment scenarios while maintaining high throughput and low latency.

High-Performance CUDA Kernels

Integrates optimizations like PagedAttention (reduces memory fragmentation), FlashAttention (improves attention layer efficiency), and FlashDecoding (lowers decoding latency) to enhance GPU utilization.

Quantization Technology Stack

Supports WeightOnly INT8/INT4 quantization (including GPTQ and AWQ schemes) and adaptive KV Cache quantization, flexibly balancing precision and efficiency.

Dynamic Batching Optimization

Maximizes batch size with low latency through efficient scheduling and memory management.

Hardware Adaptation

Specialized optimization for V100 GPUs, adapted to Yitian ARM CPUs; heterogeneous platforms like AMD ROCm and Intel CPUs are under development.

• Separate Inference Architecture: Decouples Prefill/Decode, optimizing resource allocation for the characteristics of the two stages;
• LoRA Multi-Service Deployment: A single model instance supports multiple LoRA adapters, sharing weights to reduce memory usage;
• Multimodal Input: Natively supports mixed image-text input;
• Distributed Inference: Multi-machine multi-GPU tensor parallelism to break through single-card memory limits;
• Context Caching: Reuses KV Cache to reduce multi-turn dialogue latency;
• Speculative Decoding: Parallel verification of candidate tokens to accelerate generation.

RTP-LLM has been widely verified in Alibaba's core products:

• Taobao Wenwen: AI shopping assistant handling massive queries;
• Aidge: International AI platform serving global merchants;
• OpenSearch LLM Intelligent Q&A Version: Alibaba Cloud search base;
• Taobao Search Long-Tail Query Rewriting: Related technologies have been published in papers. These scenarios ensure its stability, performance, and functional integrity.

Ecosystem Compatibility

Compatible with the HuggingFace ecosystem, supporting weight formats like SafeTensors, PyTorch, and Megatron, and adapted to P-tuning and pruned models.

Developer Resources

Provides installation guides, quick starts, backend tutorials, contribution guidelines, and performance benchmark tools. The documentation site rtp-llm.ai supports both Chinese and English.

Community Sharing

The team shares practical experiences such as distributed inference, heterogeneous design, and Attention optimization through technical blogs.

Version History

• June 2024: Architecture refactoring, C++ core rewrite, and initiation of multi-hardware support;
• January 2025: Released the separate Prefill/Decode architecture, adapted to Yitian ARM CPUs;
• September 2025: Version 0.2.0 with enhanced performance and upgraded features.

Future Directions

Plans to expand heterogeneous hardware support, optimize dynamic batching strategies, reduce streaming generation latency, and improve quantization schemes.