LeanKV: Accelerating LLM Inference via Activation Sparsity and KV Cache Quantization

The LeanKV project innovatively combines activation sparsity and KV cache quantization techniques to increase the inference throughput of large language models (LLMs) by 2-3 times without losing model precision, providing a practical solution for efficient LLM deployment. The project is maintained by asmit383, and its source code is hosted on GitHub.

As the scale of LLM models grows, the memory required for key-value (KV) caching increases linearly, becoming a major bottleneck limiting long-context processing and batch inference. While traditional quantization methods can reduce memory usage, they often sacrifice precision. Balancing precision and efficiency is a key focus in the industry.

Principle of Activation Sparsity

Leveraging the non-full activation characteristic of Transformer attention heads, dynamically skip activation computations with small contributions without modifying model weights.

KV Cache Quantization Mechanism

Perform adaptive quantization on KV vectors (compressing from 16/32 bits to lower bit widths), adjusting based on data distribution to balance compression ratio and precision.

Synergistic Effect

The combination of both: sparsity reduces the number of tokens processed, while quantization lowers the memory usage per token, achieving a 2-3x throughput increase without precision loss.

LeanKV needs deep integration with inference frameworks:

• Dynamic sparsity detection: Efficient heuristic algorithms to real-time determine skippable activations
• Quantization pipeline: Avoid quantization becoming a new bottleneck
• Memory layout optimization: Adapt to the data format after quantization
• Compatibility: Support mainstream engines like vLLM and TensorRT-LLM These solutions make LeanKV practically deployable.

The improvements from LeanKV bring multiple values:

• Cost reduction: Serve more requests with the same hardware
• Latency improvement: Enhance the experience of interactive applications like chatbots
• Long context support: Facilitate document analysis and code understanding scenarios
• Edge deployment: Reduce resource requirements, making large model deployment on constrained devices more feasible

Limitations

• The effect of activation sparsity varies by model architecture and task
• Quantization strategies need to balance compression ratio and precision

Future Directions

• Explore more aggressive quantization (4 bits and below) combined with precision recovery
• Combine sparsity detection with model fine-tuning to train sparsity-friendly models
• Extend to multi-modal model inference optimization

LeanKV demonstrates that through algorithm design and engineering implementation, LLM inference efficiency can be improved without sacrificing quality, providing a practical optimization path for deployment teams. In the future, such technologies will become a key bridge connecting model capabilities and application needs.