Arm KleidiAI Empowers Edge Devices: LLM-Runner Enables Large Language Models to Run on Arm Chips

Arm officially launched the open-source LLM-Runner project, which efficiently runs large language models (LLMs) on Arm architecture devices through KleidiAI backend optimization, addressing challenges like computing power, latency, and privacy in edge AI deployment. The project supports optimization strategies such as quantized inference, providing a technical path for LLM deployment on mobile and embedded devices.

As LLM capabilities improve, deploying LLMs on edge devices has become an industry focus. Traditional cloud-based inference has issues with latency, privacy, and cost, while terminal deployment needs to address computing power efficiency and energy consumption optimization. As the mainstream architecture for mobile/embedded devices, optimizing the Arm ecosystem is crucial for the popularization of edge AI.

LLM-Runner is an open-source project by Arm, built on the CMake build system, with a core highlight of natively supporting the KleidiAI backend. KleidiAI is a collection of microkernels designed by Arm specifically for AI, deeply optimized for operators like matrix multiplication and attention mechanisms. It uses SIMD instruction sets such as NEON/SVE and dynamically selects the optimal implementation based on Arm processor characteristics, resulting in several times better performance than pure software implementations.

LLM-Runner adopts a modular design, with core components including: model loader (supports multi-format weights and quantization), computing backend (KleidiAI as default, extensible), memory management (optimizes tensor allocation), and inference engine (supports autoregressive generation and KV-Cache). The project is built with CMake and compatible with Linux, Android, and embedded Linux distributions.

LLM-Runner uses multiple optimizations: 1. Quantized inference (INT8/INT4 low precision, reducing memory usage and computation); 2. Operator fusion (merging consecutive operations to reduce bandwidth pressure); 3. Multi-threaded parallelism (dynamically adjusting thread pools to maximize CPU utilization); 4. Memory reuse (tensor pools reduce dynamic allocation overhead).

LLM-Runner enables possibilities in multiple scenarios: offline intelligent assistants (providing translation/QA without network); industrial IoT (real-time fault diagnosis at factory edge nodes); privacy-sensitive scenarios (local processing of sensitive data in healthcare/finance); low-latency interactions (millisecond-level responses for voice assistants/real-time translation).

As an official Arm example, LLM-Runner demonstrates how to build efficient AI applications in the Arm ecosystem. Arm will continue to iterate KleidiAI, and the popularization of the SME instruction set in the next-generation processors will further lower the threshold for edge LLM deployment. Developers can practice mobile/embedded LLM deployment through this project.