AMD-NFS: A Native LLM Inference Stack to Break CUDA Monopoly

AMD-NFS is an LLM inference and service stack built from scratch, designed to bypass CUDA ecosystem lock-in, natively support ROCm/HIP, and replace traditional service software like vLLM and llama.cpp.

The current large language model (LLM) inference ecosystem is almost entirely dominated by NVIDIA's CUDA. From vLLM to llama.cpp, from Triton Inference Server to various optimization frameworks, most open-source projects prioritize or even only support the CUDA platform. This ecosystem lock-in not only limits the diversity of hardware choices but also keeps competitors' GPUs like AMD in a marginal position in the AI inference field for a long time.

For developers using AMD GPUs, this means either giving up performance optimization or struggling with compatibility layers. ROCm, as AMD's open-source GPU computing platform, provides HIP (Heterogeneous-compute Interface for Portability) to simulate CUDA interfaces, but most existing software stacks have not been deeply optimized for AMD hardware.

AMD-NFS (AMD-Native Inference Stack) was born to address this pain point. It is an LLM inference and service stack built from scratch, whose core goal is to completely bypass CUDA ecosystem lock-in, natively support AMD's ROCm/HIP platform, and provide a unified, high-performance alternative.

Unlike the approach of adding HIP compatibility layers on top of existing CUDA code, AMD-NFS has chosen a more ambitious path: redesigning the entire inference stack to be optimized for AMD GPU architecture from the ground up. This includes deep customization at all levels such as memory management, kernel scheduling, and parallel computing modes.

AMD-NFS adopts a clear layered architecture design, dividing the system into multiple independent but collaborative modules:

The bottom layer is implemented in C, including slab allocators and HIP kernel stubs. Slab allocator is an efficient memory management technique that pre-allocates fixed-size memory blocks to reduce runtime allocation overhead, which is crucial for LLM inference that requires frequent memory operations. HIP kernel stubs provide the basic interface for subsequent GPU computations.

The middle layer uses C++ to build the engine core skeleton, responsible for key functions such as model loading, inference scheduling, and batch processing management. C++'s performance advantages and fine-grained control over hardware make it an ideal choice for building high-performance inference engines.

Python bindings are provided via Cython, allowing developers to use familiar Python interfaces to call underlying high-performance implementations. This layer also includes setup.py for easy installation and deployment, lowering the barrier to use.

The top layer uses Go to build the server skeleton, leveraging Go's advantages in concurrent processing and network services to provide high-throughput model service interfaces. Go's lightweight goroutine model is particularly suitable for handling a large number of concurrent inference requests.