tiny-vllm Project Introduction

This article introduces the tiny-vllm project, an educational implementation of an LLM inference engine using C++/CUDA. The project provides an in-depth analysis of the Safetensors format, BF16 floating-point principles, the PagedAttention mechanism, and the complete inference workflow, offering systematic learning resources for developers who want to understand the underlying principles of large model inference.

The project is developed by Jędrzej Maczan, open-sourced under the Apache 2.0 license, with concise code that is fully functional and accompanied by detailed educational documentation.

Project Background and Features

The vLLM codebase is large and complex, making it difficult for beginners to understand the underlying principles. tiny-vllm addresses this issue: it is written from scratch using C++/CUDA, with concise code that is fully functional, making it suitable for learning.

Implemented features include: loading real models from Safetensors, complete LLM forward propagation (prefill + decode), pure CUDA kernel computation, KV caching, static/continuous batching, online Softmax, and PagedAttention.

LLM Inference Workflow and Tech Choices

Four-Step Workflow from LLM Design to Service

1. Model Design: Use Python/PyTorch to design the architectural blueprint
2. Model Implementation: Write code to define the specific structure
3. Model Training: Run backpropagation to produce weight files (e.g., Safetensors)
4. Model Serving: The inference engine loads weights and executes (the role of tiny-vllm)

Why Choose C++ and CUDA

• Performance: GPU acceleration for matrix operations is significant
• C++ Advantages: Zero-overhead abstractions, direct memory control, seamless integration with CUDA
• Cost: High development complexity; tiny-vllm shows how to overcome these complexities

Key Technology Analysis: Format and Data Type

Safetensors Format

File structure:

1. Header Size (8 bytes): Size of the JSON header
2. JSON Header: Tensor metadata (dtype, shape, offsets)
3. Tensor Data: Actual weight values

Advantages: Memory-mapping friendly, allowing on-demand loading of multi-gigabyte models

BF16 Floating-Point

• 16-bit structure: 1 sign bit +8 exponent bits +7 mantissa bits
• Same exponent range as FP32, slightly lower precision
• Avoids numerical overflow of FP16, suitable for AI training/inference

Architecture and Core Mechanism

Llama3.2 1B Architecture

• Embedding Layer: Maps tokens to 2048-dimensional vectors
• 16 Transformer Decoder Layers:
  • Attention Sub-layer: Q/K/V projection, GQA, RoPE, attention computation, output projection
  • MLP Sub-layer: Gate/Up projection, SiLU activation, Down projection
• RMS Normalization + Residual Connections: Stabilize deep networks
• Output Head: Linear transformation + Argmax

PagedAttention Mechanism

• Inspired by OS virtual memory management
• Splits KV cache into fixed-size blocks, tracks mappings via a block table
• Advantages: Eliminates fragmentation, on-demand allocation, memory sharing, supports continuous batching

Inference Workflow and Optimization

Two Stages of Inference

1. Prefill Stage: Process input prompts, compute KV for each token in parallel
2. Decode Stage: Generate tokens one by one, append KV cache serially

Optimization Techniques

• Continuous Batching: Add new prefill requests during the decode stage to maintain high GPU utilization
• Online Softmax: Maintain running maximum and correction factors to achieve numerically stable streaming computation

Project Value and Target Audience

Technical Value

• Systematic learning materials: From file parsing to complete inference workflow
• CUDA practice cases: Memory management, thread organization, kernel optimization
• Teaching-friendly: Concise code suitable for classroom use

Target Audience

• Developers who want to deeply understand LLM inference
• Engineers learning CUDA programming
• University teachers (teaching resources)

Minimal Dependencies

Only depends on nlohmann/json, CUDA toolchain, and cuBLAS

Conclusion

Contribution of tiny-vllm: Pursues understandability rather than maximum functionality, helping developers build a solid foundation.

Future plans: Complete all documentation by the end of April 2026, add more diagrams and detailed explanations.

Recommendation: Worth following for those who want to understand the principles of LLM inference engines.