VAC: Intelligent Neural Network Compression Technology Guided by Fisher Information

Core Overview of the VAC Project

VAC (Variable Allocation Compression) is a structured neural network compression method that combines Fisher information sensitivity analysis and evolutionary strategy search. By allocating optimal compression budgets to each weight matrix, it achieves a compression ratio of up to 2x while maintaining model performance, providing new insights for the efficient deployment of large language models.

Project Source

• Original author/maintainer: asystemoffields
• Source platform: GitHub
• Release time: May 26, 2026
• Project link: https://github.com/asystemoffields/v-a-c

With the parameter scale of large language models like GPT, LLaMA, and OLMo growing to hundreds of billions, deployment faces multiple challenges:

1. Storage and VRAM Pressure: A 7B-parameter model in bf16 precision requires approximately 14GB of VRAM, exceeding the capacity of most consumer GPUs;
2. Limitations of Quantization Technologies: Quantization methods like GPTQ and AWQ only reduce storage bits and do not lower inference computation (FLOPs remain unchanged);
3. Flaws of One-Size-Fits-All Compression: Uniform quantization ignores sensitivity differences between layers/components, easily causing irreversible performance loss on key parameters.

The core of VAC is to find the optimal compression representation for each weight matrix. Key technologies include:

1. Low-Rank Decomposition: Decompose the weight matrix W into B@A, reducing storage (m×n→r×(m+n)) and computation;
2. Fisher Information Sensitivity Analysis: Use a diagonal Fisher matrix to evaluate parameter importance, prioritizing the discarding of low-sensitivity directions via scaled SVD;
3. MCKP Optimization Allocation: Model compression budget allocation as a Multiple-Choice Knapsack Problem (MCKP) to minimize performance loss under the total budget;
4. Sequential Compression: Process layers in an "middle-out" order, adapting to activations after previous layer compression to solve error propagation issues;
5. Evolutionary Strategy: Search for the optimal compression order, Fisher scaling function (cube root is better than square root), and layer allocation ratios.

OLMo-3-7B-Think Experiment Results

Method	Perplexity (PPL)	Compression Ratio	Notes
Naive SVD (uniform 2x)	9739	2.0x	Model completely broken
VAC v1 (sequential Fisher)	144	2.0x	67x improvement
VAC v2 (evolutionary)	90.54	1.8x	39% better than v1
Restored	~27	1.8x	Only 6 PPL away from the teacher model

Inference Performance Comparison

Format	Download Size	VRAM Requirement	Quality	Inference Speed
Original bf16	14.6GB	14.6GB	PPL21	1.0x
GPTQ Q4	4.1GB	~5GB	PPL~23	~1.0x
VAC1.8x(bf16)	8.9GB	8.9GB	PPL27	~1.8x
VAC1.8x(INT8)	8.9GB	~4.5GB	PPL27.3	~1.8x

VAC reduces both storage and computation, outperforming pure quantization methods.

The practical value of VAC includes:

1. Edge Deployment: Consumer GPUs (e.g., RTX4090) can run 7B+ models (~4.5GB VRAM needed for 1.8x compression + INT8 quantization);
2. Inference Acceleration: Reduced FLOPs directly improve throughput and lower latency;
3. Model Customization: Modular design supports experimenting with different compression strategies to adapt to specific tasks/hardware;
4. Academic Benchmark: Provides complete open-source components (Fisher analysis, MCKP optimization, etc.).

Current Limitations

• GGUF/llama.cpp not supported (requires custom inference path);
• Loading requires trust_remote_code=True (restricted in security-sensitive environments);
• 6 PPL gap from the teacher model (exact benchmark may vary);
• Loading requires 16GB system RAM, GPU VRAM requirements: 8.9GB (bf16) or ~4.5GB (INT8).

Future Directions

Explore adaptive compression technology to allow models to dynamically adjust compression levels based on deployment environment and task requirements, balancing quality and efficiency.