Spill: An Intelligent Memory Tiering Solution to Break VRAM Limits for Large Model Inference

This article introduces Spill—an intelligent GPU memory tiering plugin for llama.cpp. Using techniques like learning access patterns and prefetching data, it allows large models exceeding VRAM capacity to achieve inference speeds close to full VRAM deployment on consumer hardware. Spill solves the problem of sudden inference speed drops due to insufficient VRAM when deploying large models locally, providing a new path for running large models on consumer-grade hardware.

When deploying large models locally, VRAM capacity is a key bottleneck. For example, the 24GB VRAM of RTX4090 cannot hold a 70B parameter Q4 quantized model (about 35GB). Traditional solutions offload some layers to system memory, leading to a cliff-like drop in inference speed (e.g., 440ms per token transfer). Existing solutions (like llama.cpp, ollama) use simple tiered offloading, lacking intelligent optimizations (no compression, pipelining, or parallelism).

Spill's design philosophy is to use software intelligence to avoid hardware bottlenecks. Its core ideas include: 1. Compressed transmission (compress before PCIe transfer to improve effective bandwidth); 2. Heterogeneous parallel execution (CPU/GPU compute different layers simultaneously); 3. Sparse-aware scheduling (MoE models only transfer the actually activated expert parts); 4. Predictive prefetching (prefetch next layer or predict MoE experts based on access patterns). Additionally, Spill implements a three-tier cache architecture: Tier0 (VRAM, active weights/hot KV), Tier1 (host memory, warm experts/KV pages), Tier2 (NVMe, cold experts/overflow KV).

Spill's workflow has three stages: 1. Cold start: Load the model to the optimal tier and record access conditions; 2. Warm-up phase (2-10 inferences): Accumulate data, optimize prefetching—cache hit rate increases from 60% to 90%; 3. Steady state (after 10 times): Prediction accuracy reaches 90-95%, effective throughput is close to 95% of full VRAM deployment, and workload characteristics can be saved for instant hot start.

Spill has targeted optimizations for MoE models (e.g., Gemma4, Mixtral): intercept routing outputs, predict expert selection for the next token and prefetch—hit rate reaches 90-95%. Additionally, Spill is implemented based on wgpu, supporting NVIDIA, AMD, Intel Arc, Apple Silicon, and browser environments (WebGPU), covering the non-CUDA market.

As a llama.cpp plugin, Spill can be enabled via environment variables or command line:

• Ollama environment: OLLAMA_HELIX=1 ollama serve
• Direct llama.cpp call: ./llama-server --backend spill --model gemma4-26b.gguf
• Instant hot start: HELIX_PROFILE=~/.spill/code-review.json ollama serve

Spill represents a new direction in local LLM inference optimization: shifting from passive hardware adaptation to active software intelligence. Combining multiple technologies, it is expected to achieve a 3-5x inference speed improvement on consumer hardware. It provides users with a path to run large models without professional GPUs. As MoE architectures become mainstream, intelligent memory management will become even more important.