BlockQuant: A New Block Vector Quantization Method Based on Spherical Geometry

Key Takeaways

• Unified theoretical analysis clarifies: The advantages of rotational quantization methods like EDEN and RabitQ are not absolute but depend on specific distortion criteria (e.g., MSE, inner product distortion, high-probability control).
• Proposes BlockQuant: More faithfully preserves the geometric structure of rotated embeddings via block-level spherical quantization, outperforming baselines like EDEN and RabitQ in both MSE and inner product distortion.
• Applicable scenarios: Long-context LLM inference (KV cache compression), vector database retrieval, edge device deployment, etc.

Importance of Vector Quantization

Vector quantization is the infrastructure for scalable AI, applied in:

• Memory-efficient storage: Compress high-dimensional vectors to reduce storage usage;
• Fast retrieval: Speed up similarity calculation for approximate nearest neighbor search;
• Compressed inference: Reduce memory requirements for large model inference on edge devices (e.g., LLM KV cache can reach tens of GB).

Confusion in Rotational Quantization

Rotational quantization (random orthogonal transformation to distribute errors uniformly) has emerged, with representative methods like EDEN, RabitQ, TurboQuant, but comparison is challenging:

• Different papers use different distortion criteria (MSE, inner product distortion), probability frameworks (expectation vs high probability), and implementation assumptions;
• Practitioners find it hard to determine the optimal method for specific scenarios.

Unified Theoretical Comparison

The research team provides a unified analysis, clarifying that each method's advantages depend on criteria:

Method	MSE	Expected Inner Product	High-Probability Control
EDEN	Excellent	Excellent	Good
TurboQuant	Excellent	Good	Good
RabitQ	Good	Good	Excellent

Conclusion: Method selection should be based on application requirements, not a single metric.

BlockQuant Innovation

Core Idea: Block-level spherical quantization (traditional is coordinate-level):

1. Rotate the vector then split into blocks;
2. Treat each block as a point on a high-dimensional sphere;
3. Spherical quantization preserves intra-block geometric relationships.

Algorithm Flow: Random rotation → Block splitting → Spherical mapping → Spherical quantization → Encoding and storage.

Advantage: More faithfully preserves the spherical geometry of rotated embeddings (high-dimensional vectors tend to distribute on the sphere).

Theoretical Guarantees

Advantages of BlockQuant under key distortion criteria:

• Reconstruction MSE Bound: Given a bit budget, the expected MSE is strictly better than coordinate-level baselines;
• Expected Inner Product Distortion Bound: The expected inner product error of quantized vectors is smaller;
• Theoretical results do not depend on specific data distributions and are applicable to high-dimensional embedding scenarios.

Experimental Validation

Real-World Datasets

On text embeddings (OpenAI, Sentence-BERT), image embeddings (CLIP), and recommendation system embeddings, BlockQuant outperforms baselines in both MSE and inner product distortion.

LLM Long-Context Inference

• Maintains higher inference accuracy at the same bit rate;
• Uses lower bit rate at the same accuracy (e.g., 3-bit vs 4-bit);
• Memory savings in long-sequence scenarios significantly improve throughput.

Computational Efficiency

• Encoding speed is slightly lower than coordinate-level but practical;
• Decoding speed is comparable to baselines;
• Memory bandwidth savings in long-context scenarios outweigh encoding overhead.

Practical Application Scenarios

1. Long-Context LLM Deployment: KV cache quantization (memory bottleneck, accuracy-sensitive; BlockQuant achieves high compression ratio while preserving accuracy);
2. Vector Databases: Reduce storage costs and improve retrieval accuracy (improved inner product distortion guarantee);
3. Edge Device Deployment: Maintain usable accuracy at extremely low bit rates, adapting to resource constraints.

Technical Synergies

BlockQuant can be combined with other compression techniques:

• Quantization Synergy: Mixed use with weight quantization, supporting mixed precision;
• Pruning Synergy: Structured pruning reduces parameter count, BlockQuant compresses remaining representations;
• Distillation Synergy: After distilling a small model, BlockQuant further compresses it.

Current Limitations

• Block Size Selection: Optimal value depends on data and tasks;
• Rotation Overhead: Random orthogonal transformation cost is non-negligible in extremely high-dimensional scenarios;
• Hardware Optimization: Does not fully utilize dedicated instructions like GPU tensor cores.

Future Directions

1. Adaptive Block Size: Dynamically adjust block size;
2. Learned Rotation: Data-driven learning of optimal rotation (non-random);
3. Non-Uniform Quantization: Spherical non-uniform quantization points matching data distribution;
4. End-to-End Training: Integrate BlockQuant into model training process for joint optimization.

Core Recap: BlockQuant breaks through coordinate-level limitations via block-level spherical quantization, demonstrating practical value in multiple scenarios. Future optimization can be done via adaptive block size, learned rotation, etc.