Building a CUDA Inference Engine from Scratch: A Deep Technical Analysis of the Tiny-Infer Project

Tiny-Infer is a 60-day educational project for building a large language model (LLM) inference engine using CUDA/C++. Its goal is to build a lightweight inference engine supporting the Llama 3.2 1B model from scratch, integrating core optimization technologies such as Flash Attention, paged KV cache, speculative decoding, and INT8 quantization. The project adheres to the principle of "correctness before speed" and helps learners master the underlying principles of LLM inference optimization through a structured learning path. Quantifiable goals include increasing greedy decoding throughput to over 40 tokens/s and reducing memory usage by 50%.

• Original Author/Maintainer: venkatakesavvenna
• Source Platform: GitHub
• Original Link: https://github.com/venkatakesavvenna/tiny-infer
• Release Date: May 26, 2026

Existing LLM inference frameworks (such as vLLM and TensorRT-LLM) are often complex and difficult to use as learning materials. Tiny-Infer fills the gap in the field of LLM inference education by providing a "minimum viable" reference implementation to help learners understand the essence of optimization technologies.

Tiny-Infer adopts a layered architecture, with core code written in CUDA C++ (only the tokenizer uses Python to wrap the HuggingFace implementation). The 60-day plan is divided into two phases:

• First Month: Build the engine foundation, including weight loading, forward propagation (embedding, RMSNorm, RoPE, naive multi-head attention, SwiGLU), static KV cache + autoregressive generation, Flash Attention optimization, and paged KV cache.
• Second Month: Implement speculative decoding and INT8 quantization.

Each phase must ensure that the output is numerically consistent with HuggingFace Transformers.

Flash Attention

Through a block-wise computation strategy, it reduces the memory complexity of attention from O(N²) to O(N), uses GPU SRAM to perform local softmax, and trades recomputation for memory efficiency.

Paged KV Cache

Drawing on the virtual memory mechanism of operating systems, it divides the KV cache into fixed blocks for dynamic management, supports sequence-shared memory and dynamic expansion, and improves memory efficiency for long contexts.

Speculative Decoding and INT8 Quantization

• Speculative Decoding: Uses a small draft model to generate candidate tokens, which are then verified in parallel by the main model, accelerating single-batch processing by more than 1.5x.
• INT8 Quantization: Reduces the precision of KV cache from FP16 to INT8, halving memory usage with minimal quality loss.

The project adopts a structured "learning by doing" design, with clear daily tasks, verification standards, and submission requirements. Core engineering rules:

1. Correctness before speed
2. Each phase ends with data
3. Don't reinvent the wheel
4. Seek help from the community in time
5. Submit code daily

Benchmark scripts record peak memory, first-token latency, and throughput to ensure reproducible optimization results.

Tiny-Infer provides developers, researchers, and students with a starting point to deeply understand LLM inference systems. The author plans to produce 3 technical blogs, 1 public GitHub repository, and 1 benchmark table to promote knowledge sharing in the open-source community. For the Chinese technical community, this project lowers the entry barrier for LLM system programming and helps cultivate talent in underlying optimization.

Tiny-Infer is an open-source project with clear goals and a well-planned schedule. It is both a code repository and a curriculum outline, breaking down complex knowledge into 60 learning units. As the demand for large model deployment grows, there is an urgent need for engineers who master inference optimization technologies. Projects like Tiny-Infer will become important infrastructure for cultivating relevant talent.