quantized-SLM: Restoring Inference Fidelity of Quantized Small Language Models via Inference-Time Techniques

The core goal of the quantized-SLM project is to restore the inference fidelity of quantized small language models (SLMs) using pure inference-time techniques (without retraining or increasing model parameters), addressing the key issue of degraded inference performance after quantization. This project provides an efficient and high-performance model deployment solution for edge AI and cost-sensitive scenarios, balancing model compression efficiency and inference capability.

As efficiency concerns for large models grow, SLMs (1B-7B parameters) have gained attention due to their low latency and deployment cost, but their inference capability is inferior to large models. While quantization techniques (PTQ, QAT, GPTQ, etc.) improve efficiency, they cause significant degradation in inference capability (reduced memory and fluency, with the most severe damage to reasoning ability), which has become a core pain point in SLM quantization.

The project proposes a three-stage framework: 1. Inference pattern analysis (comparing differences between full-precision and quantized models to locate key layers/tokens); 2. Key token identification (logical connectives, numerical values, reasoning step markers, etc.); 3. Inference-time intervention (adaptive temperature scaling, confidence-guided decoding, reasoning chain verification, layered precision restoration). Adaptive temperature reduces the temperature for key tokens to enhance certainty, while layered precision restoration improves precision for key middle/deep layers.

In benchmark tests like GSM8K and MATH, after applying technical interventions to 4-bit quantized models, the GSM8K accuracy increased from 45% to 65% (close to the full-precision 70%), and MATH Pass@1 rose from 28% to 42%. The additional computational overhead is controllable (e.g., reasoning chain verification adds 20-30% time), and it is effective across models like Llama-2-7B and Mistral-7B. Ablation experiments show that each component contributes positively, with the complete method achieving the best results.

Applicable to local inference on edge devices (smartphones, IoT) (quantization saves resources + techniques restore performance), real-time interaction systems (balancing speed and accuracy), cost-sensitive applications (aggressive quantization reduces inference costs), and AI research (providing a benchmark for quantization impact analysis).

Current limitations: Some techniques are task-specific, hyperparameters are sensitive, and restoration effects for extreme quantization (below 2-bit) are limited. Future directions: Adaptive hyperparameter tuning, neuron-level precision control, integration with advanced quantization algorithms, establishment of theoretical frameworks, hardware co-design, multimodal expansion, and application in federated learning scenarios.

The project open-sources core algorithms (adaptive temperature, confidence-guided decoding, etc.), evaluation tools, pre-configurations for mainstream small models, and documentation/tutorials, providing the community with plug-and-play inference enhancement tools, a benchmark platform for quantization research, and a basic framework for further development.