Lattice: A Linux Kernel-Level Optimization Engine for LLM Inference

Lattice is an OS support layer based on Linux and Rust, designed specifically for large language model (LLM) inference workloads. It addresses memory fragmentation and GPU utilization bottlenecks in long-context inference through technologies like kernel-level PagedAttention, virtual GPU memory management, coroutine heterogeneous scheduling, and eBPF network offloading.

• Original Author/Maintainer: Vitalrubbish
• Source Platform: GitHub
• Original Title: Lattice
• Original Link: https://github.com/Vitalrubbish/Lattice
• Source Publication/Update Time: 2026-05-29

The inference process of large language models usually consists of two stages: the Prefill stage (computationally intensive) and the Decode stage (memory intensive).

As context length continues to increase, models need to maintain a large KV Cache (key-value cache), which poses significant challenges to system memory management.

Traditional OS memory allocation mechanisms tend to cause severe memory fragmentation when handling such large-capacity, dynamically changing GPU memory demands. Fragmentation not only limits the effective utilization of GPUs but also directly affects inference throughput and latency performance. Existing inference frameworks like vLLM and SGLang have made many optimizations at the application layer, but they are still limited by the underlying memory management mechanisms of the OS.

The core idea of the Lattice project is to push optimizations down to the OS level, fundamentally solving the performance bottlenecks of LLM inference through kernel-level memory management and network optimization.

Lattice is developed using the Rust language, leveraging Rust's memory safety features and zero-cost abstraction capabilities to build a lightweight yet powerful OS support layer. Its technical architecture focuses on four core optimization directions:

Lattice implements a kernel-level PagedAttention mechanism, managing GPU memory through an on-demand physical allocation strategy. When more memory is needed during inference, the system triggers physical memory allocation via the kernel page fault handling mechanism instead of pre-allocating large blocks of contiguous memory.

This design draws on the concept of OS virtual memory, treating GPU memory as a pageable resource. When physical GPU memory is insufficient, the system can automatically offload infrequently used KV Cache pages to host memory and reload them back to the GPU when needed. This flexible memory management strategy significantly reduces memory fragmentation and improves the overall utilization of GPU memory.

In generation scenarios like Beam Search, models need to maintain multiple candidate sequences simultaneously. Lattice introduces a copy-on-write mechanism, allowing multiple candidate sequences to share the underlying physical KV Cache pages.

Specifically, when multiple sequences share the same context prefix, they can reference the same set of physical memory pages. Only when a sequence generates unique new content does the system trigger a page copy operation. This mechanism uses reference counting to manage the lifecycle of shared pages, significantly reducing memory redundancy while ensuring correctness.

Lattice uses eBPF technology to directly parse inference requests at the network card level, bypassing the traditional socket buffer layer to achieve zero-copy data flow. Through XDP (eXpress Data Path) and TC (Traffic Control) hooks, network packets can be processed directly in kernel space without copying to user space.

This design is particularly important for high-concurrency inference scenarios. The processing latency and CPU overhead of the traditional network stack become bottlenecks in high QPS (Queries Per Second) scenarios, while eBPF offloading can reduce network processing latency to the microsecond level.

In distributed inference scenarios, models are split across multiple GPUs for execution, requiring frequent activation value transfers. Lattice implements an NCCL (NVIDIA Collective Communications Library) bypass mechanism via eBPF, using AF_XDP sockets for inter-node communication.

This design avoids the processing overhead of the traditional TCP/IP protocol stack and is particularly suitable for activation value transfers in pipeline parallelism scenarios. By processing network packets directly in user space, Lattice can significantly reduce communication latency in distributed inference.