SFMP: A Fine-Grained, Hardware-Friendly, Search-Free Mixed-Precision Quantization Framework for Large Language Models

SFMP is a novel mixed-precision quantization framework. Through four key innovations—fractional bitwidth, block-level mixed precision, row-column weight rearrangement, and unified GEMM kernel—it addresses the high search cost and low hardware efficiency issues in traditional methods, achieving an excellent balance between compression ratio and inference efficiency, and is suitable for large language model deployment scenarios.

As the parameter scale of large language models expands, deployment costs rise sharply, making quantization technology a key compression method. Traditional uniform quantization struggles to balance compression ratio and performance; existing mixed-precision methods have two major pain points: first, they require expensive discrete optimization to determine precision allocation, and the search space grows exponentially with model size; second, irregular memory layouts lead to low hardware efficiency.

1. Fractional Bitwidth: Converts discrete precision allocation into continuous optimization, reducing solution complexity without the need for search; 2. Block-Level Mixed Precision: Uses (512,128) blocks as units, balancing fine granularity and hardware friendliness; 3. Row-Column Weight Rearrangement: Aggregates important weights into specific blocks, improving quantization quality with minimal overhead; 4. Unified GEMM Kernel: Supports efficient CUDA kernels for any average bitwidth, enhancing deployment flexibility.

Verified on Llama-3.1-8B and Qwen3-8B models:

• Compression Effect: Under the 3.75-bit configuration for Llama-3.1-8B, the model size is only 5.3GB (about 35% of the original), and the accuracy is close to the FP16 version;
• Cross-Model Consistency: Experiments on Qwen3-8B show similar trends, verifying universality;
• Inference Efficiency: The unified GEMM kernel improves throughput, and the search-free feature shortens deployment preparation time.

SFMP supports integration with multiple quantization methods such as AWQ and GPTQ; the ModelScope platform provides pre-quantized models to lower the usage threshold; it offers an end-to-end toolchain (sensitivity analysis, quantization, evaluation, deployment) and supports exporting BCQ format and custom CUDA kernels.

1. Edge AI Acceleration: Compresses the model size to less than a quarter of the original, facilitating edge device deployment;
2. Cloud Cost Optimization: Reduces memory usage and inference costs, improving service concurrency;
3. Open Source Ecosystem Contribution: All code, models, and tools are open-sourced, promoting technology democratization and subsequent research.

SFMP addresses the pain points of traditional mixed-precision quantization through four innovations, balancing compression ratio, performance, and efficiency, and provides a new path for the widespread deployment of large models. In the future, it will support more architectures, explore lower bitwidth solutions, and deepen hardware cooperation, and is expected to become one of the de facto standards in the quantization field.