NanoLlama: A Bare-Metal Llama 3 Inference Engine Built from Scratch in C++

A Llama 3 8B inference engine written entirely from scratch in C++ without relying on any external machine learning frameworks. It achieves efficient large-model inference on pure CPU through mmap zero-copy, AVX2 SIMD instruction set, and OpenMP multi-threading optimizations.

NanoLlama was born from a simple yet profound question: How exactly do modern large language models work internally? The author aims to thoroughly deconstruct the LLM architecture and understand every step from weight loading to token generation. This "from scratch" methodology makes the project an excellent resource for learning Transformer architecture and inference optimization.

Unlike most projects based on existing frameworks, NanoLlama takes a more challenging but transparent path. It does not rely on any external libraries—all mathematical operations, memory management, and tensor operations are implemented manually. This design philosophy ensures that every line of code directly corresponds to a specific computational step of the neural network, with no black-box abstractions.

The first challenge in large-model inference is model loading. The weight file of Llama 3 8B usually exceeds 5GB, and traditional file reading methods significantly slow down the startup speed. NanoLlama uses the mmap (memory mapping) mechanism of the Linux system to solve this problem.

The core idea of mmap is to directly map the binary file on the disk to the process's virtual address space instead of reading it into RAM all at once. When the CPU actually needs to access a certain block of weight data, it triggers physical memory loading through the Page Fault mechanism. This "on-demand loading" strategy brings several significant advantages:

• Extremely fast startup: No need to wait for the entire model file to be read
• Memory efficiency: The operating system automatically manages the lifecycle of memory pages
• Clean code: Avoids complex buffer management and streaming read logic

This method is similar to the memory strategy of llama.cpp, but NanoLlama's implementation is completely independent, demonstrating how to solve practical problems with the most basic system calls.

The bottleneck of LLM inference lies in matrix-vector multiplication, which is the most frequent operation in the Transformer architecture. NanoLlama deeply explores the AVX2 instruction set of Intel/AMD processors and implements manually optimized vectorized computing.

The specific implementations include:

256-bit register parallel processing: Using __m256 type SIMD registers, it can process 8 single-precision floating-point numbers (float32) or 16 half-precision floating-point numbers (float16) at a time. Compared to scalar operations, the theoretical speedup can reach 8-16 times.

FMA fused multiply-add instruction: Modern CPUs support Fused Multiply-Add operations, which can complete the calculation of a * b + c in a single clock cycle. NanoLlama fully utilizes this feature, merging matrix multiplication and bias addition into one instruction, reducing the number of operation cycles by about one-third.

OpenMP multi-threading parallelism: Through OpenMP compilation directives, computing tasks are automatically distributed to all available CPU cores. Each core independently processes different parts of the tensor, achieving nearly linear multi-core scaling.

To reduce memory usage, modern LLMs usually adopt quantization technology. NanoLlama supports Q4_K quantization in GGUF format, which is a block quantization scheme that compresses weights from 16 bits to 4 bits.

The challenge of dequantization is to real-time restore 4-bit data to 32-bit floating-point numbers for calculation during inference. NanoLlama's implementation strategy is very ingenious:

1. Block-level processing: Q4_K organizes weights into blocks, each containing 32 weights and a scaling factor
2. Half-precision intermediate state: The scaling factor is stored in FP16 format and first converted to FP32
3. Vectorized bit operations: Using AVX2 bit operation instructions to decompress multiple 4-bit values in parallel
4. In-register computation: The entire dequantization process is completed in CPU registers, avoiding frequent memory accesses

This design allows the model to maintain a small memory footprint while achieving inference quality close to that of full-precision models.

NanoLlama is not a simple inference wrapper but a complete reproduction of the Llama 3 architecture. Here are the implementation key points of the core components:

In a stack of 32 Transformer layers, signal attenuation is a serious problem. NanoLlama adopts the Pre-Norm architecture, which applies normalization before each sub-layer (attention or feed-forward network) and then adds the output of the sub-layer to the input. This design ensures the stable propagation of gradients in deep networks and is a standard practice for modern LLMs.

Compared to traditional LayerNorm, the Llama series uses RMSNorm (Root Mean Square Normalization). This normalization method only calculates the root mean square of the input without subtracting the mean, resulting in less computation. NanoLlama's implementation is in the math_utils module, using AVX2 instructions to accelerate sum-of-squares and division operations.