Reading
The EdgeRazor framework, open-sourced by the Nanjing University team, enables efficient deployment of large language models (LLMs) on edge devices via mixed-precision quantization-aware distillation. It supports quantization precisions ranging from 1.58-bit to 4-bit, achieving a maximum compression ratio of 7.03x on the Qwen3-0.6B model.
Section 01
The EdgeRazor framework, open-sourced by the Nanjing University team, enables efficient deployment of large language models (LLMs) on edge devices via mixed-precision quantization-aware distillation. It supports quantization precisions from 1.58-bit to 4-bit, achieving a maximum compression ratio of 7.03x on the Qwen3-0.6B model, effectively balancing extreme compression and model capability.
Section 02
In recent years, large language models have demonstrated strong capabilities across multiple domains, but their parameter scales often reach billions or even tens of billions, making them difficult to run on resource-constrained environments like mobile devices and edge nodes. Traditional model compression methods (pruning, quantization, distillation) are independent of each other, requiring developers to piece together tools; low-bit quantization is often accompanied by significant performance loss, so balancing compression and capability is a focus in academia and industry.
Section 03
EdgeRazor is an open-source lightweight framework developed by the Nanjing University team, designed specifically for edge AI scenarios. Its core idea is to seamlessly integrate model compression techniques into the full-precision training process, enabling efficient lightweighting with minimal code modifications. Unlike the two-stage process of quantization followed by fine-tuning, EdgeRazor adopts Quantization-Aware Distillation (QAD), which considers both quantization noise and knowledge transfer during training, allowing low-bit student models to inherit the capabilities of the teacher model.
Section 04
EdgeRazor supports independent quantization of weights, activations, and KV caches, and provides a matrix-level mixed-precision mechanism (e.g., 2.79-bit: 50% of weights at 4-bit +50% at1.58-bit;1.88-bit:12.5% of weights at4-bit +87.5% at1.58-bit) to adapt to the quantization sensitivity of parameters in different layers.
Supports YAML, JSON, and Python dictionary formats for declarative configuration of quantization strategies, distillation methods, etc., reducing learning costs and reproducibility difficulty.
Section 05
EdgeRazor in Qwen3-0.6B model experimental results:
| Configuration | Average Score | Compression Ratio |
|---|---|---|
| 4-bit | 47.80 | 3.94× |
| 2.79-bit | 44.10 | 5.05× |
| 1.88-bit | 41.76 | 6.40× |
| 1.58-bit | 39.81 | 7.03× |
The 4-bit configuration score (47.80) exceeds the full-precision baseline (47.35); at the same bit width, EdgeRazor's scores in the 3-bit/2-bit range are significantly ahead of traditional methods, proving the effectiveness of the mixed-precision strategy.
Section 06
EdgeRazor lowers the threshold for edge deployment:
Section 07
The team plans to support lightweight ViT-S/16 and ResNet-18 image classification models, as well as compression of the multimodal model Qwen2.5-Omni-7B.
Section 08
EdgeRazor is an important advancement in edge deployment of LLMs. Through the combination of quantization and distillation, as well as the mixed-precision strategy, it maintains model capability under extreme compression. Being open-source (MIT license) and community-maintained facilitates its implementation, and it will play an important role in the model democratization process, providing a solution for running LLMs on resource-constrained devices.
Keep going with more reads from the same topic.
Splinter is a minimalist, high-performance key-value (KV) and vector storage system enabling zero-latency inter-process communication via shared memory and atomic operations. With only 766 lines of core code, it supports millions of operations per second and 768-dimensional vector storage, offering a new architectural approach for local LLM inference and data-intensive applications.
Folkering OS is the world's first AI-native bare-metal operating system, entirely written in Rust no_std without relying on Linux, POSIX, or libc. It can generate commands from scratch, compile them into WASM, and run them in 10 seconds, achieving true self-evolution.
This project explores how to use Large Language Models (LLMs) to detect logical vulnerabilities in Ethereum smart contracts, especially business logic flaws that are difficult for traditional static analysis tools to capture, such as economic manipulation and execution order errors.
This article introduces an open-source project for building modern large language models from scratch. The project evolves from a GPT-2-style basic architecture to a Llama 2/3-style production-level implementation, covering complete tutorial implementations of key technologies such as RMSNorm, RoPE, SwiGLU, GQA, and MoE.