mini-infer: A Zero-to-One Implementation of LLM Inference Engine and Complete Tech Stack Analysis

mini-infer is a zero-to-one built LLM inference engine project, with its core positioning as an educational tool and prototype verification platform. It implements key mechanisms of modern inference systems such as PagedAttention, continuous batching, prefix caching, and speculative decoding—each feature comes with independent benchmark data and reproduction methods. Compared to production-grade systems like vLLM, mini-infer provides a clear learning path with minimal code, helping developers deeply understand the principles of LLM inference.

In the LLM inference field, production-grade systems like vLLM have complex code, making it hard for learners to get started. mini-infer's goal is not to compete with production-grade features, but to serve as an educational tool and prototype verification platform:

• Implement key mechanisms such as PagedAttention, continuous batching, chunked prefill, and prefix caching;
• Each implementation prioritizes correctness and comes with detailed performance measurements;
• The core serving path achieves 100% throughput of HF Transformers on Qwen2.5-7B, and supports a --dry-run mode to verify interfaces without model weights.

mini-infer implements multiple core LLM inference technologies:

1. PagedAttention: Uses flash_attn's block_table to manage KV cache and avoid memory fragmentation;
2. Continuous Batching: Based on AsyncEngine with OpenAI-compatible HTTP API, allowing new requests to dynamically join batches;
3. Chunked Prefill: Splits long sequence prefill into small chunks to reduce latency jitter;
4. Prefix Caching: Reuses prefix KV cache based on block-level hashing and LRU eviction strategy;
5. Speculative Decoding: Uses a small draft model to predict the output of the large model for faster inference;
6. CUDA Graph: Static capture of decode_batch to reduce CPU overhead;
7. Flash Decoding: Uses Triton's split-K optimization to improve SM utilization;
8. Tensor Parallelism: Adopts NCCL all-reduce and Megatron-LM sharding strategy;
9. PD Decoupling: Separates prefill and decode phases with two co-located processes.

Benchmark data for each technology verifies its effectiveness:

• PagedAttention (batch=8): Throughput of 406 tokens/s, on par with HF Transformers;
• Continuous Batching: When concurrency increases from 1 to 8, throughput linearly scales from 55.7 tok/s to 219.1 tok/s (3.9x improvement);
• Chunked Prefill: Reduces ITL peak by 57%-67% in mixed scenarios;
• Prefix Caching: Reduces TTFT by 22% in shared prefix scenarios;
• Speculative Decoding: 0.5B draft +7B target model has an acceptance rate of 55.85%;
• CUDA Graph: Reduces decoding latency by 28.9% for 1.5B model with batch=1;
• Flash Decoding: 3.31x speedup at sequence length 4096, SM utilization increases from 9% to 103%;
• Tensor Parallelism (TP=2): Output is completely consistent with single-card (correctness verification);
• PD Decoupling: Prefill 12.3ms, transmission 14.7ms, decoding 519ms.

mini-infer uses a modular code structure:

• core/: Core configurations like EngineConfig, Request, SamplingParams;
• runtime/: Runtime components like LLMEngine, Scheduler, AsyncEngine;
• cache/: KVCacheManager (BlockTable + Prefix Cache);
• modeling/: ModelRunner implementation;
• kernels/: Kernels like PagedAttention, Triton decode;
• parallel/: Tensor parallelism, replication, pipeline parallelism;
• serving/: FastAPI server, OpenAI Schema compatibility layer; In addition, the benchmarks directory contains 21 independent scripts, and the tests directory has 287 test items (most support dry_run).

mini-infer supports pip installation and quick startup:

1. Installation: pip install -e ".[serve,dev]"
2. Dry-run mode (no model needed): mini-infer-serve --dry-run --port 8000
3. Real model startup: mini-infer-serve --model /path/to/Qwen2.5-7B --port 8000 After the service starts, you can call it via the OpenAI-compatible API, which supports streaming output and multi-turn conversations.

Comparison with vLLM

Dimension	mini-infer	vLLM
Goal	Zero-to-one implementation and measurement of key inference mechanisms	Production-grade: high throughput, multi-model, SLO guarantee
PagedAttention	Same approach as vLLM	Same approach, more mature
Model Coverage	Qwen2.5 / DeepSeek-V2	Dozens of architectures, auto-adaptation
Scheduler	Hand-implemented, four queues + chunked prefill	Full SLO, KV sharing awareness
Deployment	Single-machine prototype	K8s, multi-machine RDMA, full monitoring

Engineering Value

mini-infer provides a streamlined entry point for LLM inference learners. Compared to vLLM's tens of thousands of lines of code, it implements core mechanisms with fewer lines and includes benchmark data. It is suitable for:

• Engineers who want to enter LLM system development (learning platform);
• Researchers who want to verify new mechanism prototypes (extensible experimental framework).