rvLLM: Rewriting vLLM in Rust to Build a High-Performance LLM Inference Engine

rvLLM is a Rust-based rewrite of the popular vLLM inference engine, offering full OpenAI API compatibility. It addresses Python's limitations in vLLM (Global Interpreter Lock, garbage collection pauses, large dependencies) with significant improvements in startup speed, memory usage, and throughput—making it a high-performance alternative for LLM service deployment.

vLLM has been a leading open-source LLM service engine due to its PagedAttention technology. However, Python's inherent issues (GIL, GC pauses, large dependency体积) hinder large-scale deployment. rvLLM emerges as a Rust-based solution to these problems.

rvLLM consists of 23 Rust crates and 15 handwritten CUDA kernels, supporting FlashAttention-2 and CUDA graph features. Key benefits:

• No GIL: Enables parallel execution of scheduling, sampling, and tokenization across CPU cores.
• Zero GC Pauses: Deterministic memory management via Rust's ownership model.
• Minimal Deployment: 16MB static binary with zero runtime dependencies (vs Python vLLM's ~500MB).
• Direct GPU Calls: Uses cudarc to bypass PyTorch, reducing overhead for tensor operations and kernel scheduling.

Throughput: On A100 GPU (FP16, 32 concurrent requests), rvLLM achieves ~3,500 tokens/sec. Comparison with Python vLLM:

Metric	rvLLM	Python vLLM	Improvement
Startup Time	6s	~120s	20x
Binary Size	16MB	~500MB	31x
CPU Memory	348MB	~1GB	3x
CPU Operations: Rust outperforms Python (numpy) in tasks like repetition penalty (11x faster), polynomial sampling (5.5x), and batch sampling (8.5x).

GPU Compatibility: Supports NVIDIA GPUs from V100 (sm_70) to Blackwell series (sm_122). Kernels can be compiled for specific architectures (e.g., CUDA_ARCH=sm_90 bash kernels/build.sh). Installation:

• Cargo: cargo install rvllm
• Pip: pip install rvllm
• Source: Build with cargo build --release --features cuda (GPU) or without (mock-GPU). Docker: Build via make docker; run with docker run --gpus all -p8000:8000 rvllm:latest serve --model ....

rvLLM supports OpenAI-compatible endpoints: /v1/completions, /v1/chat/completions, /v1/models, /health, /metrics. Examples:

• Curl chat: curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{"messages":[{"role":"user","content":"Explain quantum computing"}],"max_tokens":200}'
• Python client: Use OpenAI SDK with base_url="http://localhost:8000/v1" (no API key needed). Integrates with LiteLLM and LangChain.

rvLLM signals a trend of migrating LLM infrastructure to system languages like Rust. Key industry benefits:

• Cost Optimization: Lower memory usage and higher throughput reduce deployment costs.
• Latency Sensitivity: Faster startup and lower P99 latency for real-time apps.
• Edge Deployment: Small binary size适合 edge devices. Conclusion: rvLLM is a low-migration-cost, high-gain alternative to Python vLLM, with potential to become a standard for LLM service deployment as it matures.