EdgeRazor: A New Paradigm for Lightweight Large Models on Edge Devices

The EdgeRazor framework, open-sourced by the Nanjing University team, enables efficient deployment of large language models (LLMs) on edge devices through mixed-precision quantization-aware distillation technology. It supports multiple quantization precisions from 1.58-bit to 4-bit, significantly improving compression rates while maintaining performance, and provides a complete and easy-to-use engineering solution for edge AI scenarios.

Background: Urgent Needs of Edge AI

With the improvement of large language model (LLM) capabilities, deploying LLMs on edge devices (smartphones, IoT devices, etc.) faces resource constraints. Traditional cloud-based inference has issues like network latency, privacy risks, and cost pressures, making direct deployment of large models impractical. Model compression technologies (quantization, knowledge distillation) have become the key bridge connecting LLM capabilities and edge applications.

EdgeRazor Framework Overview

EdgeRazor is a lightweight open-source framework for edge AI. Its core strategy is "Quantization-Aware Distillation (QAD)", which integrates quantization and distillation to compress model size while maintaining performance. The design philosophy is "plug-and-play", allowing low-intrusive integration into existing training workflows.

Mixed-Precision Quantization

It supports matrix-level mixed-precision mechanisms, where different layers/matrices can use different precisions. It supports weight quantization (embedding layer, lm_head), activation quantization, and KV cache quantization. Multiple mixed-precision schemes (e.g., 2.79-bit, 1.88-bit) are provided to facilitate the trade-off between compression rate and performance.

Multi-Dimensional Knowledge Distillation

EdgeRazor provides three complementary distillation methods that can be flexibly combined:

1. Logits Distillation: Align the output distribution of student and teacher models
2. Feature Distillation: Align intermediate layer features
3. Attention Distillation: Transfer Transformer attention patterns

Managed through a unified configuration interface, developers can choose the optimal strategy based on their tasks.

Performance and Experimental Results

Taking Qwen3-0.6B as an example under the W-A8-KV8 configuration:

Configuration	Average Score	Compression Rate
Original Model (W16-A16-KV16)	47.35	1×
4-bit EdgeRazor	47.80	3.94×
2.79-bit EdgeRazor	44.10	5.05×
1.88-bit EdgeRazor	41.76	6.40×
1.58-bit EdgeRazor	39.81	7.03×

The 4-bit configuration model outperforms the original full-precision model. At the same compression rate, its performance is better than traditional methods, and the 2-bit level still maintains usable accuracy.

Application Scenarios and Ecosystem Development

The EdgeRazor team has built a complete ecosystem:

• Pre-quantized model collections (zhangsq-nju/edgerazor-nbit) are released on Hugging Face, including multiple precision versions of Qwen3-0.6B/1.7B
• Supports GGUF format conversion, compatible with llama.cpp, and can run on pure CPU
• Launched EdgeRazor Playground, an interactive demo platform running on CPU, lowering the technical threshold

Developers can directly use the optimized models to experience edge AI technology.

Technical Significance and Future Outlook

EdgeRazor promotes the advancement of edge LLM deployment technology, encapsulating complex technologies into simple interfaces to realize model compression and implementation.

• Mobile developers: Run AI functions locally without network dependency, protecting privacy
• Edge computing: A feasible path to deploy large models in resource-constrained environments
• Researchers: Open-source code and experimental data provide benchmarks

As the demand for edge AI grows, EdgeRazor will become a key infrastructure for AI democratization.