Edge-LLM: A Large Language Model Inference Framework for Mobile and Embedded Devices

Edge-LLM is a specialized inference framework designed for mobile and embedded devices. It addresses key challenges in edge deployment and supports hardware acceleration for Qualcomm QNN/HTP, MediaTek Neuron/APU, and CUDA GPU platforms. Key features include INT8/INT4 quantization, a unified ELM model format for cross-platform deployment, and a complete toolchain covering model conversion, compilation, and runtime execution.

With the growing demand for deploying large language models (LLMs) on edge devices, several unique challenges exist:

• Computational Power Limitation: Mobile/embedded CPU/GPU performance lags behind data center servers.
• Memory Constraints: LLMs often require gigabytes of memory, exceeding the capacity of edge devices.
• Power Sensitivity: Mobile devices have strict power limits, preventing long-term high-load operation.
• Heterogeneous Hardware: Different chip architectures require targeted optimization. Edge-LLM was created to address these issues, supporting quantization and hardware-specific acceleration.

ELM Model Format

Edge-LLM uses ELM (Edge Language Model) as a unified format, enabling single-file deployment and zero-copy loading via memory mapping. An ELM file includes:

• Computation graph (operator definitions and connections).
• Quantized weights (INT8/INT4 with scaling factors and zero values).
• Quantization metadata (parameters and calibration information).
• Optional hardware compilation products (e.g., QNN context binaries).

Layered Architecture

The framework's layered design:

1. Model Parsing Layer: Reads HuggingFace models (safetensors + config.json) to build a unified computation graph IR.
2. Quantization Layer: Applies PTQ/QAT to compress FP32 weights into INT8/INT4.
3. Serialization Layer: Converts quantized graphs/weights into ELM format.
4. Graph Partitioning Layer: Assigns subgraphs to optimal execution backends (falls back to CPU if unsupported).
5. Backend Compilation Layer: Compiles ELM subgraphs into hardware-specific products.
6. Unified Runtime: Manages memory, schedules tasks, and executes inference across backends.

Hardware Backend Support

• Qualcomm Platform: Uses QNN SDK and HTP for acceleration, including QualcommBackend (public interface), QNN compiler (ELM → QNN graph), and runtime (load/execute QNN products).
• MediaTek Platform: Leverages Neuron SDK and APU, including MediaTekBackend, Neuron compiler, and runtime.
• CUDA Backend: Supports NVIDIA GPUs, including CudaBackend, CUDA compiler (operator → CUDA kernel), and runtime.

Model Support

Edge-LLM currently supports:

• Qwen3/Qwen3.5 (Alibaba Tongyi Qianwen series).
• Gemma4 (Google open-source model). Models are located in the models/ directory and use a modular design (shared KV cache, attention mask), making it easy to add new models.

Deployment Workflow

1. Model Conversion: Convert HuggingFace models to ELM format: HF model → common/graph (build IR) → quantization (FP32→INT8/INT4) → elm/writer (output .elm)
2. Hardware Compilation: Compile ELM for target hardware: .elm → common/partitioner (split subgraphs) → backend compiler (hardware products) → elm/writer (update .elm)
3. Inference Execution: Run on target devices: Compiled .elm → common/runtime (load) → backend runtime (execute) → result

CLI Tools

The project provides:

• convert: Convert HuggingFace models to ELM.
• compile: Compile ELM to hardware-specific products.
• run: Execute inference on target devices. These tools support automated deployment and can be integrated into CI/CD pipelines.

Application Scenarios

• Mobile Local Assistant: Run lightweight LLMs on smartphones to provide offline, privacy-preserving smart assistant services.
• IoT Gateway: Deploy models on edge gateways for local sensor data analysis and decision-making.
• Offline Document Processing: Provide AI capabilities (understanding, summarization) in offline environments (e.g., planes, remote areas).
• Industrial Quality Inspection: Use vision-language models on production line edge devices for real-time defect detection.

Technical Highlights

• Cross-Platform: Unified ELM format and backend abstraction enable seamless deployment across hardware.
• Extreme Quantization: INT8/INT4 support reduces model size to 1/4 or 1/8 of FP32.
• Flexible Graph Partitioning: Automatically assigns subgraphs to optimal backends.
• Zero-Copy Loading: Memory mapping reduces startup latency.
• Modular Design: Easy to extend with new models or hardware backends.

Summary & Future Outlook

Edge-LLM provides a complete solution for edge LLM inference, covering model conversion, quantization, compilation, and runtime. Its unified ELM format and cross-backend architecture enable "train once, deploy anywhere", lowering the barrier to edge AI development.

Future Outlook: As mobile chip AI capabilities improve and quantization technology advances, the performance of edge LLMs will further enhance. Edge-LLM will play a key role in democratizing LLMs (making AI accessible to all).