Ryuu_AI: An Edge AI Solution for Running Large Language Models Locally on Raspberry Pi 5

The Ryuu_AI project is maintained by RJSLabbert and open-sourced on GitHub (link: https://github.com/RJSLabbert/Ryuu_AI, updated on 2026-05-24). This solution shows how to run large language models locally on Raspberry Pi 5 with Hailo 10H NPU (AI HAT 2+ expansion board), without cloud APIs or token consumption, providing a practical reference for edge AI deployment and solving cloud dependency issues such as privacy, cost, and availability.

Mainstream use of large language models relies on cloud APIs, but there are issues like privacy leakage risks (data sent to third parties), high costs (token-based billing), availability affected by network conditions, and dependence on service provider policies. Edge AI solves these problems by running models on local devices, but large models have high resource requirements—how to run them on edge devices is a challenge. Ryuu_AI is exactly the solution to this challenge.

• Raspberry Pi 5: Broadcom BCM2712 quad-core ARM Cortex-A76 (2.4GHz), VideoCore VII GPU (800MHz), 4/8GB LPDDR4X memory, dual 4K output—performance is significantly improved compared to previous generations.
• Hailo 10H NPU: Designed specifically for AI inference, providing 10 TOPS of computing power with low power consumption—it is the core accelerator.
• AI HAT 2+: Official Raspberry Pi expansion board, integrating Hailo NPU, connected via PCIe interface—plug-and-play reduces integration complexity.

Running LLMs on edge devices requires solving multiple problems:

1. Model Quantization and Compression: Apply low-precision quantization (like INT4), weight pruning, and knowledge distillation to large models to adapt to edge resources.
2. Memory Management Optimization: Use techniques such as memory mapping, layered loading, and dynamic unloading to handle the limited memory of Raspberry Pi.
3. NPU Compilation and Deployment: Use Hailo SDK to convert models into NPU-executable formats, completing quantization and optimization.
4. Inference Pipeline Design: Use techniques like streaming output and speculative decoding to improve response speed.

The Ryuu_AI solution applies to multiple scenarios:

• Smart Home/Voice Assistant: Local operation protects privacy and has no cloud dependency.
• Industrial IoT: Deploy LLMs on edge gateways for log analysis, fault diagnosis, etc.—still usable in network-isolated environments.
• Education and Research: Low-cost hardware to experience LLMs, lowering the threshold for AI learning.
• Offline Environments: Provide intelligent assistance (document analysis, knowledge query, etc.) in network-free scenarios like the wild or on ships.

Edge deployment has the following limitations:

• Model Size Limitation: Only quantized models with 7B or fewer parameters can be run, whose capabilities are weaker than large models like GPT-4.
• Slow Inference Speed: Although accelerated by NPU, latency and throughput are still not as good as cloud GPU clusters.
• Limited Model Selection: Only supports models compiled and optimized via Hailo SDK, with limited choices of open-source models.
• Simplified Functions: Advanced functions need to be cut to adapt to edge resources.

• Community Ecosystem: The open-source code of Ryuu_AI provides a reproducible reference for the community; developers can extend support for more models, optimize speed, etc. This idea can be migrated to other NPU platforms like Intel Movidius and Google Coral.
• Future: Advances in model compression technology and improvements in edge hardware computing power will promote the expansion of edge AI capabilities. It is expected to run larger models on Raspberry Pi-level devices, promoting the democratization and popularization of AI.

The Ryuu_AI project proves that running LLMs locally on resource-constrained devices is feasible, opening up new possibilities for privacy-first and cost-sensitive AI applications. For developers exploring edge AI, it is an open-source project worth paying attention to and learning from.