SFMP: Introduction to the Search-Free Fine-Grained Mixed-Precision Quantization Scheme for Large Language Models

SFMP (Search-Free Mixed-Precision) is a hardware-friendly, search-free mixed-precision quantization method designed to address the high inference cost of large language models. Its core lies in fine-grained weight grouping and adaptive precision allocation, which significantly reduces inference costs while maintaining model performance, avoiding the drawback of traditional mixed-precision methods that rely on expensive searches.

Background: Dilemmas of Quantization Technology and Challenges of Mixed Precision

The expanding parameter scale of large language models leads to a sharp increase in inference deployment costs. Quantization technology is a core method for model compression, but traditional schemes face a dilemma: uniform low precision is efficient but harms performance, while high precision fails to fully exploit hardware efficiency. Mixed-precision quantization allows different layers/groups to use different precisions, but most existing methods rely on expensive searches, which are time-consuming and difficult to adapt to hardware constraints.

Core Innovations of SFMP

1. Fine-grained weight grouping: Divide weight matrices into small weight groups, select precision independently, and accurately capture local distribution characteristics;
2. Hardware-friendly design: Support native precisions of AI accelerators (INT4/6/8), follow memory alignment requirements, and adapt to mainstream hardware;
3. Adaptive precision allocation: Based on sensitivity analysis of weight groups, use high precision for high-sensitivity groups and low precision for low-sensitivity groups to balance quality and compression efficiency.

Technical Implementation Details of SFMP

SFMP consists of three main components:

• Weight analysis module: Calculate statistical features of weight groups (distribution range, variance, outlier ratio) to determine quantization difficulty;
• Precision decision engine: Analytically decide precision allocation, completed in seconds;
• Quantization execution module: Quantize weights according to configuration, generate hardware-friendly formats, and support uniform/non-uniform quantization.

Experimental Verification: Performance of SFMP

• Model quality: The perplexity and downstream accuracy after quantization are consistent with full precision, outperforming the uniform low-precision baseline;
• Compression efficiency: Model size is reduced by 50%-75%, inference throughput is increased by 1.5-3 times without fine-tuning;
• Computational overhead: Precision allocation takes only a few seconds, which is orders of magnitude faster than traditional searches.

Application Scenarios and Practical Value of SFMP

• Cloud services: Improve inference density and reduce infrastructure costs;
• Edge AI: Run larger models on resource-constrained devices and expand the boundaries of edge intelligence;
• Dynamic quantization: Quickly switch precision configurations based on load/latency to achieve elastic services.

Summary and Outlook

SFMP balances model quality, compression efficiency, and deployment convenience through fine-grained analysis and search-free decision-making, which is an important advancement in mixed-precision quantization. In the future, it is expected to expand to activation quantization, dynamic quantization, and other fields, promoting the popularization of efficient AI.