FOEM: A New First-Order Error Compensation Method for Quantized Large Language Models

FOEM is a new first-order error compensation method for quantized large language models accepted by AAAI 2026. This method significantly improves the performance of quantized models by accurately handling the first-order errors during the quantization process. Core keywords include quantization, large language models, model compression, error compensation, INT4 quantization, etc.

As the scale of large language models (LLMs) expands, inference resources and storage costs grow exponentially. Model quantization technology reduces storage and computational overhead by converting high-precision floating-point numbers into low-precision integers (e.g., INT8/INT4). However, traditional quantization methods only focus on minimizing the overall error magnitude, ignoring the distribution characteristics of errors across different layers and positions, where some errors have a more significant impact on performance.

The core view of the FOEM project is that "first-order errors play a decisive role in quantized large language models". First-order errors refer to linear error terms during the quantization process, which have a more direct and significant impact on model outputs. Traditional rounding/truncation strategies tend to produce systematic first-order error shifts, whose cumulative propagation leads to performance degradation, especially affecting the precise calculation of the attention mechanism.

FOEM proposes a complete first-order error compensation framework, including three key steps: 1. Error Decomposition and Analysis: Split quantization errors into first-order linear errors and high-order nonlinear errors, proving the dominant role of first-order errors; 2. Adaptive Compensation Strategy: Dynamically adjust compensation intensity according to the characteristics of model layers, applying stronger compensation to sensitive layers (e.g., attention projection layers); 3. End-to-End Optimization: Add a first-order error penalty term to the quantization objective function, jointly optimizing storage efficiency and inference accuracy.

FOEM has been validated on mainstream models such as the Llama series and OPT series: 1. Significant Accuracy Improvement: Under INT4 quantization, the perplexity is reduced by an average of more than 5 percentage points, and some tasks are close to the FP16 baseline; 2. Strong Generalization Ability: Stable improvement across models of different architectures and scales; 3. Controllable Computational Overhead: The additional overhead is almost negligible, with high practical value.

The technical significance of FOEM includes: 1. Lowering Deployment Threshold: Improving the usability of low-bit quantized models, making it possible to run large models on consumer GPUs or edge devices; 2. Promoting Quantization Theory Research: Clarifying the differences in the impact of errors of different orders, providing a new perspective for subsequent algorithm design; 3. Practical Application Value: Suitable for high-efficiency inference scenarios such as real-time dialogue systems and mobile AI assistants.

By focusing on first-order error compensation, FOEM opens up a new direction for performance optimization of quantized LLMs and has been accepted by AAAI 2026. In the future, FOEM is expected to combine with technologies such as knowledge distillation and dynamic quantization to further promote the practical application of large models in resource-constrained environments.