llada.cpp: NPU Acceleration Solution for Diffusion Large Model Inference on Mobile Devices

llada.cpp is the first inference framework for diffusion large language models (dLLMs) specifically designed for mobile NPUs. It addresses the inference challenges of diffusion LLMs on mobile devices through three core technologies: multi-block speculative decoding, dual-path progressive correction, and swap-optimized memory runtime. This reduces the generation latency of the LLaDA-8B model by 17-42x while maintaining generation quality.

Diffusion language models (dLLMs) theoretically reduce latency by generating multiple tokens in parallel via denoising, but face three major obstacles on mobile devices:

1. Workload Shrinkage: The effective computation volume decreases in the late stages of block-level decoding, leading to underutilization of NPU parallel capabilities;
2. Token Correction Complexity: Token revisions make KV cache reuse difficult, and frequent refreshes increase overhead;
3. Memory Address Space Limitation: Mobile NPUs have limited accessible addresses, resulting in high costs for data remapping and transmission.

Multi-block Speculative Decoding

When the workload decreases in the late stages of current block decoding, it proactively speculates tokens for future blocks and fills the computation pipeline, fully utilizing NPU parallel capabilities and smoothing the workload curve.

Dual-path Progressive Correction

Submitted tokens remain revisable until stable, and unstable token refreshes are handled on the CPU side, enabling CPU-NPU collaboration: NPUs focus on matrix operations, while CPUs handle correction logic, and parallel pipelines improve efficiency.

Swap-optimized Memory Runtime

It compactly manages the address layout visible to the NPU, overlaps data staging with NPU computation, and reduces data remapping and transmission overhead.

The research team evaluated llada.cpp on various hardware platforms and dLLM workloads. The results show that after enabling prefix KV cache reuse, the generation latency of the LLaDA-8B model is reduced by 17-42x while maintaining generation quality.

Technical Significance: It demonstrates the deep co-design between the diffusion model architecture and the hardware characteristics of mobile NPUs. The three technologies provide reusable patterns for computation scheduling, heterogeneous collaboration, and memory management in mobile inference.

Future Outlook: Explore the parallel potential of mobile NPUs (under low power consumption), extend optimization strategies to more model architectures, and provide directions for large model deployment on mobile phones.

• Problem: Diffusion LLMs in mobile NPU inference are limited by workload shrinkage, complex token correction, and memory address constraints;
• Solution: llada.cpp addresses these issues through three core technologies;
• Outcome: LLaDA-8B model latency reduced by 17-42x while maintaining generation quality;
• Value: The first NPU-aware complete solution for mobile large model inference.

• Original Author/Maintainer: Paper author team (arXiv:2606.13740v1);
• Source Platform: arXiv;
• Original Title: Efficient On-Device Diffusion LLM Inference with Mobile NPU;
• Original Link: http://arxiv.org/abs/2606.13740v1;
• Release Time: June 11, 2026.