Building vLLM Core from Scratch: A Deep Dive into the mini-vllm Project

mini-vllm is an education-oriented LLM inference engine built from scratch. Based on the TinyLlama-1.1B-Chat model, it fully reproduces vLLM's core technical architecture (continuous batching, paged KV cache, etc.) and includes a real-time visualization tool. The project is not only a learning tool but also a production-grade engineering practice—every line of code is written manually, and correctness is ensured through layer-by-layer comparison testing with Hugging Face. Its core goal is to help developers understand the internal working principles of modern large language model inference engines, with all key links clearly visible.

Original Author and Source

• Original Author/Maintainer: Vidhun Vijayakumar Suja (vsvidhun06-blip)
• Source Platform: GitHub
• Project Name: mini-vllm
• Original Link: https://github.com/vsvidhun06-blip/mini-vllm
• Release Time: June 2026

Project Overview

mini-vllm is an education-oriented LLM inference engine built from scratch based on the TinyLlama-1.1B-Chat model. It reproduces vLLM's core technologies, and its code has been verified for correctness through layer-by-layer comparison testing with Hugging Face, aiming to help developers understand the internal principles of LLM inference engines.

1. Continuous Batching

Traditional LLM inference processes requests sequentially, leading to low GPU utilization. mini-vllm implements continuous batching, allowing multiple requests to run in parallel. In benchmark tests, 4 concurrent requests (each generating 32 tokens) achieved a 3.1x speedup compared to sequential processing. The scheduler can mix prefill (new requests) and decode (existing requests) phases.

2. Paged KV Cache (PagedAttention)

It reproduces vLLM's PagedAttention, dividing the KV cache into fixed blocks of 16 tokens, similar to virtual memory management. Advantages: memory efficiency (no need to pre-allocate maximum continuous memory), memory sharing (prefix cache/parallel sampling), dynamic expansion (allocating and releasing blocks on demand). Tests show that correctness is maintained through admission control even when cache blocks are tight (only 6 blocks available).

3. Real-Time Visualization System

A built-in WebSocket real-time visualization tool allows observation of: request state transitions (WAITING→PREFILL→DECODE→DONE), KV cache block usage (color grids), scheduler decisions, event logs, and performance metrics. Each request is marked with a deterministic hash color to help understand complex scheduling algorithms.

Principle of Speculative Decoding

A smaller "draft model" is used to quickly generate candidate tokens, and the main model verifies them at once. If accurate, it reduces the number of forward passes to speed up inference.

Experimental Results

The author tested combinations of different draft token counts (K) and early exit layers (exit_layer):

Configuration	Speedup	Acceptance Rate
K=4, exit=8	0.46x	1.1%
K=2, exit=8	0.65x	2.1%
K=4, exit=18	0.59x	28.3%
K=2, exit=18	0.78x	37.5%

Key Findings

1. K=2 is better than K=4: fewer draft positions lead to less error accumulation and higher acceptance rates;
2. Acceptance rate threshold: must meet α>exit_layer/22 (exit=8 requires 36%, exit=18 requires 82%), and the highest experimental rate of 37.5% did not reach the break-even point;
3. Model limitations: TinyLlama was not trained for early exit, so intermediate layer residual output decoding效果 is poor.

Engineering Insights

Speculative decoding acceleration depends on draft quality; untrained models are not suitable as their own draft models. Special training mechanisms are needed (LayerSkip fine-tuning, EAGLE draft heads, Medusa multi-head decoding). The v0.3 version has ready infrastructure (KV rollback, scheduler integration, etc.).

HF Consistency as Correctness Anchor

Each layer of the engine undergoes consistency testing with the transformers library (tolerance atol=1e-4), including forward propagation, cache generation, paged cache generation, and scheduler testing, ensuring that the code built from scratch behaves consistently with the industry-standard library.

Key Engineering Decisions

1. Layer-wise KV cache position handling: each layer reads its own cache length to avoid off-by-one errors;
2. Mixed prefill/decode batching: sequential prefill (one forward pass per batch of admitted requests) + single batch decoding for all decode requests, with pure decoding in steady state;
3. Paged cache layout: separate K/V pools, layer-first order, block size before KV head count, SDPA receives views instead of step-by-step copies;
4. Synchronous event bus: the scheduler sends events synchronously from worker threads, bridged to asynchronous WebSocket subscribers via loop.call_soon_threadsafe, and the event bus is unaware of asyncio.

Educational Value

Provides developers with: a runnable complete codebase (about 2000 lines of core code), detailed architecture documentation and design decision records, real-time visualization tools, and layer-by-layer test cases.

Engineering Reference

Demonstrates: high-performance inference implementation in resource-constrained environments, paged cache details, scheduler design trade-offs, and speculative decoding limitations and optimization directions.

Research Inspiration

Provides empirical data: impact of different configurations on acceptance rates, quantitative analysis of break-even conditions, and guidance for future improvement directions.

mini-vllm is a rare combination of engineering implementation and learning resources. As a master of software engineering, the author demonstrates a deep understanding of LLM inference technology. The project's highlights lie in showing the process of function implementation: HF consistency testing, real-time visualization, detailed documentation, and honest experimental records (including "failed" attempts at speculative decoding) reflect engineering rigor.

For the Chinese technical community, it is an excellent entry point to understand the core principles of industrial-grade inference engines like vLLM. The v0.3 speculative decoding infrastructure is ready, waiting for trained draft mechanisms to unlock its potential, reminding us that algorithm correctness, data quality, and model training are equally critical in machine learning system engineering.