Deep Dive into vLLM Inference Optimization: From HuggingFace Baseline to PagedAttention Practice

This study note deeply analyzes the inference optimization principles of vLLM through comparative experiments, covering core aspects such as KV cache issues, the PagedAttention mechanism, and the setup of an OpenAI-compatible API server. Starting from the HuggingFace baseline, it gradually demonstrates how vLLM improves inference performance through memory management optimization, making it suitable for developers who want to understand the underlying principles of LLM inference.

LLM inference faces challenges from concurrent requests and hardware resource constraints. Traditional KV cache uses contiguous memory block allocation, leading to significant waste (e.g., only 20.3% utilization for 5 requests) and limiting concurrency (ideal 97 users vs actual 19 users). The KV cache stores Key/Value vectors in the self-attention mechanism to avoid redundant computations, but the traditional allocation method has become a bottleneck.

The experimental environment is a Ryzen7 7840HS laptop (without discrete GPU), using Docker to pull the vLLM CPU image to ensure consistency. The lightweight SmolLM-135M (135 million parameters) model is chosen for reproducibility. The baseline uses HuggingFace Transformers, with single-request performance measured as: tokens_per_second=49.62, total time=2.0155 seconds (generating 100 tokens).

vLLM's single-request performance is 1.2x higher than the baseline (Tokens/sec:49.8 vs61.3). PagedAttention draws on the idea of pagination management, splitting the KV cache into small pages, achieving a memory utilization rate of 95.4% (saving 4.7x memory) and a 4.8x improvement in concurrency (92 users vs19 users).

Steps to build an OpenAI-compatible API server: 1. Start the server (command includes parameters like model, port, etc.); 2. Client calls using the OpenAI library (just modify base_url). The measured latency is 1.29 seconds (7 prompt tokens, generating 50 tokens), with reasonable HTTP overhead.

Key Insights: 1. Optimization is layered (baseline → vLLM → API); 2. Memory management is the core bottleneck; 3. Batching improves throughput; 4. OpenAI compatibility reduces migration costs. Best Practices: Understanding underlying principles helps with architectural decisions and tuning.

Application Scenarios: Local development and testing (rapid prototyping), edge deployment (CPU optimization available), private data scenarios (compliance requirements). Learning Extensions: Directions like continuous batching, speculative decoding, quantization techniques, tensor parallelism, prefix caching, etc.