EOQ-Quantization: A New LLM Compression Scheme Based on Entropy-Optimal Polar Quantization

EOQ-Quantization introduces entropy-optimal quantization theory and achieves near-lossless compression of LLM weights using PolarQuant technology. It significantly reduces VRAM usage while maintaining inference speed, offering new insights for edge deployment of large models. This article will elaborate on its background, core methods, technical paths, performance, and other aspects.

The parameter scale of large language models has experienced explosive growth in recent years, from billions to trillions of parameters. The improvement in model capabilities is accompanied by a sharp increase in storage and computing costs. How to reduce deployment costs while maintaining model performance has always been one of the core challenges in the field of machine learning engineering. As a key model compression technology, quantization reduces memory usage and computational overhead by lowering the precision of parameter representation, and has become an essential means for large model deployment. However, traditional quantization methods often face a dilemma: aggressive quantization strategies can bring greater compression benefits but usually come with obvious quality loss; conservative strategies can maintain performance but have limited compression effects. Finding the optimal quantization strategy to achieve the best balance between compression ratio and model quality is the driving force for the continuous evolution of quantization technology.

The EOQ-Quantization (Entropy-Optimal Quantization) project proposes a new quantization method based on information theory principles. The core insight of the project is: the optimal quantization strategy should respect the information distribution characteristics of data—using finer quantization granularity in regions with high information density and coarser representation in sparse regions. This idea originates from the entropy concept in Shannon's information theory. The entropy of data reflects its uncertainty or information content; higher entropy data contains more information and requires more bits to retain it. EOQ-Quantization adaptively allocates quantization bits by calculating the entropy characteristics of model weight distribution, achieving optimal compression in the sense of information theory. The technical implementation of EOQ-Quantization is based on the PolarQuant framework, a transformation technology that maps weights from Cartesian space to polar coordinate space. In polar coordinate representation, the direction information (angle) and magnitude information of weights are separated, which reveals new characteristics of weight distribution. Studies show that neural network weights often exhibit specific distribution patterns after training: the magnitude distribution tends to be sparse (many values close to zero), while the angle distribution is relatively uniform. PolarQuant leverages this characteristic and applies different quantization strategies to magnitude and angle respectively. For the sparse magnitude component, non-uniform quantization and stronger compression can be used; for the information-rich angle component, more quantization bits are allocated to maintain precision. The advantage of this asymmetric quantization strategy is that it can achieve higher information retention rate than traditional uniform quantization at the same average bit rate.

The key to EOQ-Quantization achieving near-lossless compression lies in the collaborative work of multi-layer optimization strategies. First is the layered quantization strategy: the project assigns different quantization parameters to each layer based on the sensitivity differences of different layers in the neural network. The weight distribution characteristics of attention layers and feed-forward layers are different, so they need to be treated differently. Second is group quantization optimization: the project processes weights in groups and independently calculates optimal quantization parameters within each group. This fine-grained processing can better adapt to the local characteristics of weight distribution and avoid efficiency losses caused by global quantization. Third is the error compensation mechanism: EOQ-Quantization introduces error feedback during the quantization process, propagating quantization errors to subsequent processing to reduce the impact of accumulated errors on model performance. This technology significantly reduces the precision loss caused by quantization while maintaining low-bit representation.

The design goal of EOQ-Quantization is not only to compress the model size but more importantly to optimize the memory usage during inference. In large model inference, the memory usage of model weights is often the main resource bottleneck, especially when deploying large models on consumer-grade GPUs. By compressing weights from FP16 or BF16 to 4 bits or even lower, EOQ-Quantization can reduce the model's memory usage by 50-75%. This means that models that originally required high-end GPUs to run can now run smoothly on mid-range or even entry-level GPUs. In terms of inference speed, the memory bandwidth savings from quantization can often offset the overhead of dequantization calculations. The memory bandwidth of modern GPUs is usually the bottleneck of computing power; by reducing data movement, quantization can actually accelerate the inference process. EOQ-Quantization optimizes quantization kernels for different hardware architectures to ensure good performance on various devices.

According to the benchmark test results provided by the project, EOQ-Quantization shows excellent performance on multiple open-source models. On Llama series models, using 4-bit quantization configuration, the model's perplexity loss is controlled within 1%, and its performance on actual downstream tasks (question answering, summarization, code generation) is almost indistinguishable from the original model. Compared with existing quantization methods (such as GPTQ, AWQ, GGUF), EOQ-Quantization usually achieves lower precision loss at the same compression ratio. Especially in very low-bit (3 bits and below) configurations, the advantage of entropy-optimal quantization is more obvious, enabling extreme compression ratios while maintaining usable performance. In terms of inference performance, models optimized by EOQ-Quantization achieve significant throughput improvements on consumer-grade GPUs. Test data shows that when running a 70B parameter model on RTX 4090, the inference speed of the quantized version is 2-3 times faster than the FP16 version, while the memory usage is reduced from over 80GB to about 20GB.

EOQ-Quantization is suitable for various LLM deployment scenarios. For individual users and researchers, it makes it possible to run large models on local workstations without relying on expensive cloud APIs. For enterprise deployments, it can significantly reduce the hardware cost of inference services and improve the utilization of existing infrastructure. In terms of edge device deployment, the extreme compression capability of EOQ-Quantization provides the possibility of running LLMs on resource-constrained devices. Although the project is mainly optimized for GPUs, its core algorithms can be adapted to various hardware platforms, including mobile devices and embedded systems. The project provides integration support with mainstream inference frameworks, including compatibility with popular backends such as vLLM, llama.cpp, and TensorRT-LLM. Users can integrate EOQ-Quantization into existing deployment processes through simple command-line tools or API calls.

Although EOQ-Quantization has achieved significant compression effects, the project documentation also points out some current technical limitations. First, polar coordinate transformation and entropy calculation introduce additional preprocessing overhead; although this overhead is one-time, it may be a consideration for scenarios that require frequent model switching. Second, the current implementation is mainly optimized for Transformer architectures, and its adaptability to other types of neural network architectures (such as CNN, RNN) needs further verification. In addition, the stability of the model still has room for improvement in very low-bit (2 bits and below) configurations. Looking to the future, the development directions of EOQ-Quantization include: supporting activation quantization to further reduce computational overhead, exploring dynamic quantization strategies to adapt to input changes, and developing dedicated optimization kernels for specific hardware.