Argus Engine: A High-Performance Rust LLM Inference Engine for ARM64 Edge Devices

Argus Engine is a Rust-based large language model (LLM) inference engine specifically designed for ARM64 edge devices, aiming to address resource constraints in edge-side LLM inference. Key features include support for Q4_0/Q8_0 quantization, OpenCL/CUDA heterogeneous acceleration, intelligent KV cache eviction, and a zero-copy memory architecture. Leveraging Rust's zero-cost abstractions and memory safety features, it enables efficient operation of large models on consumer-grade ARM64 devices, representing an important exploration in edge AI inference technology.

Edge devices (smartphones, embedded devices, etc.) face constraints such as limited memory, tight power consumption, high real-time response requirements, and diverse hardware architectures. Traditional cloud-based inference solutions rely on sufficient GPU resources and cannot adapt to edge environments. Deep innovations are needed across algorithm optimization, system architecture, and hardware adaptation to enable smooth operation of billion-parameter models on ARM64 devices.

Quantization Technology

Supports Q4_0 (4-bit, 8:1 compression ratio) and Q8_0 (8-bit, 4:1 compression ratio) quantization, combined with ARM NEON instruction set optimization for dequantization calculations.

Heterogeneous Computing

Supports OpenCL (cross-mobile GPU) and CUDA (NVIDIA devices), dynamically scheduling CPU/GPU tasks to achieve optimal resource allocation.

KV Cache Management

Intelligent eviction strategy retains key historical context based on rules like attention scores, maintaining over 90% generation quality when only 20% of KV cache remains.

Zero-Copy Memory

Reduces data transfer via memory mapping, with Rust's ownership system ensuring memory safety.

Adopts a modular architecture:

• Model Loader: Parses quantized formats like GGUF and integrates with the Hugging Face ecosystem;
• Computation Backend Abstraction Layer: Encapsulates differences between CPU/OpenCL/CUDA and supports extending new backends;
• Memory Manager: Custom memory pool to optimize inference loads;
• Scheduler: Coordinates task execution to achieve overlap between computation and transfer.

Applicable to:

• Local smartphone assistants (privacy protection, offline processing);
• Embedded smart devices (real-time natural language interaction);
• Offline document processing (AI functions in network-free environments);
• Robots and drones (onboard decision-making to enhance autonomy).

Limitations:

• Limited model ecosystem compatibility (mainly supports GGUF format);
• Dynamic shape processing efficiency needs improvement;
• Extreme quantization may lead to accuracy degradation. Development Directions:
• Introduce advanced quantization algorithms like AWQ/GPTQ;
• Support hardware such as Apple Neural Engine and Qualcomm Hexagon NPU;
• Implement speculative decoding acceleration;
• Improve the model conversion toolchain.

Argus Engine provides a feasible solution for running large models on resource-constrained devices through technologies like Rust performance optimization and fine-grained quantization strategies. As demand for edge-side AI grows, dedicated inference engines will become increasingly important. We look forward to the project's continued development and its contribution of more innovations to the edge AI ecosystem.