Building an ARM-Native LLaMA Inference Engine from Scratch: Pure C++ Implementation and NEON Acceleration Practice

This article will conduct an in-depth analysis of the arm-llm-core project—a dependency-free LLaMA inference engine optimized for Apple Silicon. Implemented in pure C++, the project covers key technologies such as memory mapping, low-level implementation of Transformer kernels, and ARM NEON SIMD acceleration. It aims to help developers understand LLM inference mechanisms and achieve high-performance deployment from first principles.

Existing frameworks like PyTorch, Transformers, or llama.cpp are powerful, but they encapsulate too many low-level details, making it difficult for developers to deeply understand the inference mechanism. The arm-llm-core project stems from the exploratory spirit of "starting from first principles". By hand-writing core LLaMA components in pure C++, it builds a dependency-free, high-performance inference engine on Apple Silicon, meeting both learning needs and specific hardware optimization requirements.

arm-llm-core is a LLaMA inference engine customized for ARM processors (especially Apple Silicon M2). Its core feature is "zero dependencies"—using only standard C++17 and CMake, with no external deep learning libraries. Advantages include: small compiled output size and simple deployment; transparent code that is easy to learn and debug; ability to deeply optimize for specific hardware without being constrained by general-purpose frameworks.

Traditional frameworks load the entire weight file into memory when loading a model, leading to slow startup and high memory usage. arm-llm-core adopts a memory mapping (mmap) strategy: the ModelLoader component maps the model file to the virtual address space, and with lightweight Tensor views, metadata directly links to disk data. Through the OS page fault mechanism, data is loaded on demand (lazy loading), allowing large models to "load" in sub-seconds, with memory usage only for currently active computations. Resource management follows the C++ RAII principle to eliminate memory leaks.

arm-llm-core implements core components of the LLaMA architecture from scratch:

• RMSNorm: Lightweight normalization to stabilize signal flow in deep networks;
• RoPE: Rotational Position Encoding, integrating relative position information to improve long-sequence extrapolation capabilities;
• Self-Attention and KV Cache: Implements scaled dot-product attention, pre-allocates KV cache to reuse key-value pairs, improving long-sequence generation efficiency;
• Feed-Forward Network: Includes SiLU activation function and uses a gating mechanism;
• Sampling Strategy: Supports temperature adjustment to balance generation diversity and numerical stability.

The project deeply leverages the ARM NEON SIMD instruction set (128-bit vector registers, processing 4 32-bit floating-point numbers simultaneously) to optimize core operations:

• vld1q_f32: Loads 4 float32 values into vector registers;
• vmlaq_f32: Fused Multiply-Add (FMA), completing multiply-add in a single cycle;
• vaddvq_f32: Horizontal summation across vector channels. Combined with compilation options -mcpu=apple-m2 -O3 (loop unrolling, auto-vectorization), it fully utilizes Apple Silicon's superscalar pipeline to improve computational efficiency.

arm-llm-core uses a custom binary format to store weights. It provides a PyTorch conversion script to export HuggingFace-compatible models (e.g., TinyLlama-1.1B) into .bin format, automatically handling differences in attention head grouping. Usage steps:

1. Run build.sh to compile the project;
2. Use export.py to convert the pre-trained model;
3. Execute ./build/llm_engine to start inference.

Roadmap:

• Completed: Zero-copy memory mapping, Transformer core components, NEON acceleration;
• Planned: INT8 quantization (halving memory usage), Python CLI wrapper, and multi-threaded parallelism (multi-core expansion). Conclusion: arm-llm-core is both a usable inference engine and an excellent learning resource. It demonstrates the process of building an LLM system from the ground up, and its optimization for Apple Silicon proves the value of handwritten kernels in specific scenarios—when general-purpose frameworks are insufficient in performance, low-level optimization capabilities are crucial.