llama-hdd.cpp: A Disk-Persisted Checkpoint Solution for LLM Inference

llama-hdd.cpp is a soft fork of llama.cpp, released by developer LuminaNAO on GitHub (repository link: https://github.com/LuminaNAO/llama-hdd.cpp, MIT License). Its core feature is persisting prompt checkpoints (including KV cache and other states) during inference to disk, solving problems like memory limitations, state loss, and redundant computations faced by traditional LLM inference, while supporting long-context processing and state recoverability.

In practical LLM applications, long-context inference faces many challenges. Traditional inference states (e.g., KV cache) are only stored in volatile memory, leading to: 1. KV cache for long sequences exhausts memory; 2. State loss upon program crash/restart requiring restart from scratch; 3. Redundant encoding of historical context in multi-turn interactions; 4. Context window fragmentation requiring information truncation. The core problem is the lack of an effective state persistence mechanism.

The core of llama-hdd.cpp is the disk-backed checkpoint mechanism:

Checkpoint Architecture

Includes KV cache snapshots, positional encoding states, attention masks, and metadata (e.g., model configurations).

Disk Storage Strategy

Block storage (on-demand loading), compressed encoding, index structure (fast access), incremental updates (only save changes).

State Recovery

Read and validate checkpoint files, reconstruct KV cache, positional encoding, and attention states, and verify version compatibility.

This solution applies to:

1. Ultra-Long Document Processing: Segmented processing + checkpoints to break through model context window limitations;
2. Persistent Conversations: Recover session state after service restart;
3. Resource Optimization: No need to re-encode history in multi-turn interactions, reducing latency and cost;
4. Fault Tolerance & Reliability: Recover from the latest checkpoint after batch processing/long task interruption.

While disk persistence brings advantages, trade-offs need to be considered:

• Storage Space: KV cache checkpoints occupy a large amount of disk space, requiring strategies like automatic cleanup and compression;
• I/O Performance: Disk read/write is slower than memory, requiring optimizations like asynchronous writing, SSD storage, and preloading;
• Consistency: Avoid race conditions leading to state corruption in concurrent scenarios.

As a soft fork, llama-hdd.cpp maintains compatibility with the upstream:

• Easily sync new features and optimizations from upstream;
• API-compatible, allowing existing applications to migrate smoothly;
• Community can choose whether to enable the persistence feature.

llama-hdd.cpp effectively solves problems like long-context processing and state persistence through the disk checkpoint mechanism, providing stronger support for LLM deployment in production environments. As LLM applications become more complex, such persistence and state management technologies will become more important, and this project provides a valuable reference implementation for related directions.