Practical Exploration of Efficient Large Language Model Inference: Integration of INT4 Quantization and MoE Architecture

This article introduces a practical project on efficient inference based on the LLaMA 3.2-1B model, exploring the implementation methods and effects of INT4 weight quantization and Mixture of Experts (MoE) architecture, providing references for deploying large models on edge devices. Key findings include: INT4 quantization can reduce model memory to 1/4 of the original FP16 with controllable increase in perplexity; in the MoE architecture, the LoRA mode performs better than the slicing mode under limited fine-tuning budget, maintaining generation quality while improving computational efficiency.

With the widespread application of LLMs, inference efficiency optimization has become a core challenge (inference cost is dominant). This project was completed by the EE508 course team at the University of Southern California, using LLaMA 3.2-1B as the experimental platform to explore two optimization techniques—INT4 quantization and MoE—aiming to solve the resource constraints of deploying LLMs on edge devices.

The project adopts a three-stage framework:

1. Theoretical review (Transformer mechanism, GQA, RoPE, etc.);
2. INT4 weight quantization (group quantization, each group uses 4-bit integers + independent scaling factors);
3. MoE architecture exploration (two modes: slicing mode initializes experts by slicing FFN weights; LoRA mode initializes experts using LoRA adapters on frozen dense weights).

• INT4 quantization: The module llama/quantize.py uses group quantization, focusing on the numerical stability of quantization and dequantization;
• MoE architecture: The module llama/moe.py introduces configurable expert layers and gating networks to achieve load balancing;
• Training and evaluation: Fine-tuned using the Alpaca-500 dataset, recording loss curves and expert loads; evaluation metrics include perplexity, downstream accuracy, and tok/s.

• INT4 quantization: Memory is reduced to 1/4 of the original FP16, with a slight increase in perplexity; inference acceleration is significant in scenarios with limited memory bandwidth;
• MoE comparison: The slicing mode leads to perplexity degradation due to expert homogenization; the LoRA mode requires only a small number of additional parameters, maintaining generation quality while significantly improving computational efficiency.

• Engineering value: The code is modular (e.g., llama/model.py implements the LLaMA architecture, benchmark_inference.py performs performance testing), facilitating reuse and expansion;
• Industry insights: INT4+MoE provides a feasible path for edge deployment; optimization requires balancing implementation complexity, cost, and quality; university course practices can produce high-quality reproducible research.

• Limitations: Verified only on a 1B model, lacking end-to-end evaluation in real scenarios;
• Future directions: Explore dynamic quantization strategies, optimize MoE routing (e.g., load balancing regularization), and integrate INT4 and MoE architectures.

This project verifies the effects of INT4 quantization and MoE on LLaMA 3.2-1B: INT4 aggressive compression still maintains usable quality, and LoRA mode MoE is a practical solution. It provides reference implementations and experience for deploying large models on edge devices, and the open-source code supports further iterative optimization by the community.