VAR-Compressor: Efficient Quantized Deployment of 8-Billion-Parameter Visual Autoregressive Models on Edge GPUs

The VAR-Compressor project uses W4A4 weight and activation quantization and INT8 KV cache quantization techniques to compress the Infinity VAR 8B visual generation model to run natively on 16GB edge devices, providing new ideas for edge AI deployment.

In recent years, visual autoregressive models (VAR) have performed excellently in the field of image generation, but their scale of billions of parameters has extremely high computational and memory requirements. For example, the Infinity VAR 8B model has high memory demand under standard inference, limiting its application on edge platforms like NVIDIA Jetson. Model compression has become a key link.

VAR-Compressor develops a quantization scheme for the Infinity VAR model: 1. W4A4 weight and activation quantization: using SVDQuant to handle extreme activation outliers (maximum-to-median ratio up to 353x) in the FFN down-projection layer, and constructing a high-precision low-rank branch with SVD to mitigate accuracy loss; 2. INT8 KV cache quantization: based on the findings that the channel coefficient of variation >1.2 and skewness ≈0.85, an asymmetric per-channel quantization strategy is adopted to save memory while maintaining performance.

Based on structural analysis of the Infinity VAR architecture: 1. Activation outliers: extreme activation outliers exist in the FFN down-projection layer, with kurtosis higher than Gaussian distribution, prompting the application of SVDQuant; 2. KV cache characteristics: variance is unevenly distributed across different dimensions, so asymmetric per-channel INT8 quantization is chosen to achieve 4x memory savings while maintaining accuracy.

The compressed Infinity VAR 8B model can run natively on 16GB edge devices. Application scenarios include: edge content creation (local image generation on Jetson devices), privacy-sensitive applications (local processing without cloud upload), real-time interactive systems (reducing inference latency), and resource-constrained environments (deploying high-performance models on embedded systems).

The project is custom-developed based on MIT HAN Lab's DeepCompressor framework, integrating the SVDQuant quantization engine. Usage steps: 1. Clone the repository and install dependencies; 2. Download the pre-compressed model or run the quantization process; 3. Deploy inference on the target edge device. A diagnostic toolset is also provided to verify compression effects and performance metrics.

The project's corresponding paper, 'Enabling 8B Bitwise Autoregressive Image Generation on Edge GPUs', elaborates on technical details. As an open-source project, it provides usable compressed models and a complete technical path reference, proving that an 8-billion-parameter model can still maintain usable generation quality under 4-bit quantization, offering a reference for edge deployment of large-scale generation models.

With the improvement of edge AI chip computing power and optimization of quantization algorithms, it is expected that larger-scale generation models can run on smaller devices in the future. The structure-aware quantization strategy and architecture-specific optimizations of VAR-Compressor provide a reference paradigm for this direction.