Reading
Using E8 lattice quantization and attention-aware token eviction mechanism, NexusQuant compresses the KV cache of large language models by 10-33x without training or calibration data, enabling long-context inference to move from multi-card clusters to single-card deployment.
Section 01
Using E8 lattice quantization and attention-aware token eviction mechanism, NexusQuant compresses the KV cache of large language models by 10-33x without training or calibration data, enabling long-context inference to move from multi-card clusters to single-card deployment.
Section 02
To understand the value of NexusQuant, we first need to grasp why KV cache consumes so much memory. In the Transformer architecture, when processing long sequences, the model needs to store the Key and Value matrices for each layer to perform attention calculations when generating new tokens. The size of these matrices is proportional to the sequence length—the longer the sequence, the larger the cache.
For example, the Mistral-7B model's KV cache for 128K context reaches up to 80GB. This means even a top-tier A100 GPU (80GB memory) will encounter OOM (out-of-memory) when handling 32K context. To process longer sequences, one has to resort to multi-card clusters, which significantly increases deployment costs.
Section 03
NexusQuant adopts a combined strategy to compress KV cache, consisting of two key components:
Section 04
First, the system scores tokens based on attention weights. Tokens with lower attention weights are considered to have less impact on subsequent generation and can thus be safely evicted. The system always retains the BOS (Beginning of Sequence) token and a recent sliding window to ensure key information is not lost.
In this way, the number of tokens can be reduced by 2.5x at a 60% eviction rate, while the impact on model performance is kept within an acceptable range.
Section 05
For the retained tokens, NexusQuant uses a technique called E8 lattice quantization—this is the most ingenious part of the entire scheme.
The E8 lattice is a special 8-dimensional lattice structure in mathematics with extremely high packing density. NexusQuant groups 8 floating-point numbers together, uniformly distributes energy via Hadamard rotation, then maps them to the E8 lattice. This mapping can be represented with very few bits: Keys use 3-bit, Values use 2-bit (since Keys require higher precision to handle the amplification effect of softmax).
Additionally, the system uses differential encoding and zstd compression—adjacent tokens often produce similar lattice indices; storing differences and then compressing can achieve an additional 2-3x compression ratio.
Section 06
NexusQuant's implementation includes several key steps:
Importance Scoring offers two options: fast scoring based on Key-Key proxy (no extra computation) or using a real attention scorer (higher quality but requires an additional forward pass).
RoPE Removal is another key trick. Since Rotary Position Encoding (RoPE) places Keys in different subspaces at different positions, direct quantization does not work well. NexusQuant first 'undoes' RoPE before quantization to bring all Keys back to a common subspace, then restores RoPE after quantization.
Boundary Protection is an optimization for specific model families. Qwen series models are particularly sensitive to quantization in certain layers, so the system provides a protect_boundary parameter that allows selecting to keep the first and last several layers in FP16 precision.
Section 07
NexusQuant provides four preset configurations to adapt to different quality-compression trade-offs:
| Preset | Compression Ratio | Perplexity Loss | Context Supported by 80GB Memory |
|---|---|---|---|
| high | ~9x | <0.5% | ~1.2M tokens |
| asym | ~14x | ~1% | ~1.8M tokens |
| balanced | ~17x | ~1.3% | ~2.2M tokens |
| max | ~33x | +0.66% | ~4.2M tokens |
Actual measurement data shows that NexusQuant achieves significant compression effects on mainstream models like Mistral-7B, Phi-3-mini, and Qwen2.5-7B. Especially when using K3V2 (3-bit Keys + 2-bit Values) with a real scorer, even at a 35% eviction rate, the perplexity loss can be controlled within 1%.
Section 08
NexusQuant's biggest advantage is 'training freedom'. Let's compare it with similar technologies:
In contrast, NexusQuant requires no training or calibration, is ready to use out of the box, yet achieves a compression ratio of 10-33x—this has obvious advantages in practicality.
Keep going with more reads from the same topic.
Splinter is a minimalist, high-performance key-value (KV) and vector storage system enabling zero-latency inter-process communication via shared memory and atomic operations. With only 766 lines of core code, it supports millions of operations per second and 768-dimensional vector storage, offering a new architectural approach for local LLM inference and data-intensive applications.
LiteMind is a Python framework that provides developers with a unified API to integrate mainstream LLM providers such as OpenAI, Anthropic, Google Gemini, and Ollama. It supports multimodal input/output, tool calling, RAG enhancement, and agent construction.
OmniRoute is an open-source AI gateway that integrates over 67 large model providers through a single OpenAI-compatible endpoint. It offers intelligent routing, load balancing, automatic degradation, and quota management, helping developers get the most available AI services at the lowest cost.
This article introduces an open-source multimodal RAG framework based on Google Gemini Embedding 2, which can uniformly handle embedding and retrieval of four media types: text, images, videos, and audio. Combined with Supabase pgvector vector database and OpenRouter large language models, it provides a complete production-level retrieval-augmented generation pipeline.