IntAttention: A Pure Integer Attention Inference Acceleration Scheme for Edge Devices

IntAttention is the open-source implementation of an MLSys 2026 paper. It proposes an all-integer attention pipeline to enable high-fidelity and high-speed inference of Large Language Models (LLMs) and Vision Transformers (ViTs) on ARM CPUs, aiming to address the computational power bottleneck of deploying Transformer models on edge devices.

With the popularization of LLMs and ViTs, edge device deployment faces issues like high floating-point computation overhead, high latency, and high energy consumption. While quantization techniques can optimize these, existing solutions often ignore the complex operations of the attention mechanism; matrix multiplication and Softmax in the attention mechanism are prone to precision loss and numerical overflow under integer quantization. Balancing precision and efficiency remains an open problem.

The core of IntAttention is an all-integer attention pipeline that covers the entire process of Query-Key dot product, Softmax normalization, and Attention-Value multiplication. Key optimizations include: 1. Integer Softmax replaces floating-point exponentiation and division with Look-Up Tables (LUTs) and fixed-point operations; 2. Layer-wise dynamic quantization, adjusting scaling factors and zero points based on the activation distribution of each layer; 3. Blocked memory layout optimization to improve cache hit rate.

Tests were conducted on models like LLaMA, BERT, ViT, and ARM CPUs such as Qualcomm Snapdragon and Apple M-series: Compared to floating-point baselines, inference speed increased by 2-4 times, and memory usage decreased by approximately 50%; in terms of precision, the accuracy difference from floating-point models was less than 1% in benchmark tests like GLUE and ImageNet.

IntAttention can be applied to: 1. Mobile intelligent assistants, running LLMs locally to achieve privacy protection and low latency; 2. Real-time visual understanding, running ViTs on camera terminals for security and autonomous driving assistance; 3. IoT devices, running Transformer models on embedded devices to upgrade smart homes and industrial inspection.

IntAttention's code is fully open-source, providing model conversion tools for PyTorch and ONNX formats; it supports optimized kernels for ARM NEON and x86 AVX2; the official team provides tutorials, pre-trained models, and complete deployment processes, and the community is actively expanding support for multimodal models.

IntAttention represents the hardware-aware direction of edge AI inference optimization. In the future, it will expand to platforms like RISC-V and NPU, and combine with sparsification and pruning techniques to further unleash the AI potential of edge devices.