Running a 30-Billion-Parameter Large Model on Raspberry Pi Cluster: A Low-Cost Practice of Distributed Inference

Against the backdrop of rising inference costs for Large Language Models (LLMs), the Hermes Cluster project demonstrates an efficient inference solution using extremely low-cost hardware: through a distributed cluster composed of 4 Raspberry Pi 5s, it successfully runs the Qwen3-30B-A3B MoE model with 30 billion parameters, achieving an inference speed of 13.82 tok/s, providing a feasible reference for edge AI deployment.

Large language models usually rely on expensive GPU servers. The maturity of model quantization technology and distributed inference frameworks has made it possible for consumer/embedded hardware to run large models. The Hermes Cluster project is based on the distributed-llama framework, and through 9 key patches to optimize multi-node communication efficiency, it enables ARM devices like Raspberry Pi to achieve usable inference performance.

Hardware Configuration: 4 Raspberry Pi 5s (each with 8GB RAM), leveraging their advantages in CPU performance and memory bandwidth improvement; Architecture Design: Master-slave architecture + high-speed network connection, optimizing communication primitives and memory layout to reduce data transmission overhead between nodes; Model Selection: Qwen3-30B-A3B MoE model (total parameters: 30 billion, about 3 billion parameters activated each time; its sparse characteristics are suitable for distributed deployment); Quantization Strategy: 4-bit or lower-precision quantization, splitting the model and storing it in the memory of 4 devices.

Inference speed reaches 13.82 tok/s, which is sufficient to support interactive scenarios such as document summarization, code completion, and dialogue; Significant advantages in power consumption and cost: total power consumption <50 watts, total cost only a few hundred dollars, which has extremely high cost-effectiveness compared to GPU solutions (thousands of dollars in cost + high power consumption).

Contributed 9 patches to the upstream of distributed-llama (covering communication optimization, memory management, and ARM architecture adaptation); Released a complete technical report, recording the cluster setup, performance tuning process, and challenges, providing valuable materials for reproducing/improving the solution.

• Edge AI Gateway: Private AI services for scenarios like factories, farms, and retail stores;
• Educational Research: Allowing students/researchers to access large model technology at low cost;
• IoT Hub: Localized intelligent decision-making for smart homes and smart cities;
• Emergency Backup: Providing limited inference services when the main server fails.

Limitations: Raspberry Pi's CPU performance is limited; it cannot compete with GPUs in compute-intensive tasks, and is suitable for scenarios with low latency sensitivity and moderate throughput; Future Improvements: Introduce NPU acceleration modules, optimize load balancing strategies, explore hybrid parallel modes, and develop intelligent caching mechanisms.

The Hermes Cluster project proves the possibility that "large models don't need large hardware". Reducing hardware thresholds is as important as optimizing algorithms. This solution provides a practical reference for edge AI deployment, and we look forward to more similar innovations to promote AI democratization.