Running Quantized Large Models on Raspberry Pi 4: A Practice of Local RAG Chatbot for Edge Devices

Exploring how to deploy a complete LLM+RAG system on the resource-constrained Raspberry Pi 4, using the 390MB Qwen2.5-0.5B quantized model to implement a local AI chatbot with end-to-end response times of 3-6 seconds. The project covers key technologies such as model quantization, lightweight inference engine optimization, and RAG retrieval integration, verifying the feasibility of running an AI system locally on edge devices.

With the rapid improvement of large language model (LLM) capabilities, how to achieve efficient local inference on edge devices has become a hot topic in the developer community. Traditionally, running LLMs requires expensive GPU servers, but the development of quantization technology and lightweight inference engines has made it possible for consumer-grade hardware and even embedded devices to run AI. This project focuses on the Raspberry Pi 4 (4GB RAM, ARM processor), with the core challenge of realizing an end-to-end AI dialogue experience including Retrieval-Augmented Generation (RAG).

The project adopts a modular design, with core components including:

1. Quantized Language Model: The Qwen2.5-0.5B model is selected, compressed to 390MB via GGUF format 4-bit quantization (Q4_K_M), balancing memory usage and inference quality.
2. Lightweight Inference Engine: Built based on llama-cpp-python, tuned to 3 threads, achieving a generation speed of 3-8 tokens per second on the ARM Cortex-A72 processor.
3. RAG Retrieval Pipeline: Uses all-MiniLM-L6-v2 to generate text embeddings, FAISS library for vector search, and preloaded Vietnamese electric vehicle consultation documents.

The measured data on Raspberry Pi 4 are as follows:

Stage	Time Consumption
RAG Vector Retrieval	10-15 ms
First Token Generation	1-2 seconds
Complete LLM Inference	3-5 seconds
End-to-End Total Latency	3-6 seconds
Generation Speed	3-8 tokens/second
This performance meets the basic requirements for real-time dialogue on edge devices and has practical value.

Optimization strategies for low-resource environments:

• Memory Optimization: Q4 quantization controls the model size within 400MB, adapting to the Raspberry Pi's 4GB memory limit.
• Computation Optimization: Limiting to 3 threads avoids CPU preemption, and a small context window reduces KV cache usage.
• Retrieval Optimization: The lightweight implementation of FAISS makes retrieval take only 10-15 ms, reducing end-to-end latency.
• Localization Design: Natively supports Vietnamese scenarios, adapting to specific language and cultural needs.

Typical application scenarios:

• Offline customer service system: Providing AI consultation in network-free environments
• Privacy-sensitive scenarios: Local data processing without cloud upload
• IoT intelligent interaction: Providing natural language interaction for smart home/industrial devices
• Educational experiment platform: Teaching cases for edge AI and model deployment Expansion possibilities: Flexible configuration interfaces support model replacement, thread count adjustment, and RAG document library modification to adapt to different hardware and business needs.

The project verifies the feasibility of the 'small model + optimized architecture' in the edge AI field. With the progress of quantization technologies (GPTQ, AWQ, GGUF) and inference engines, future Raspberry Pi-level devices can run larger-scale models. This project provides a full-link reference implementation for edge AI beginners, covering practices from model selection, quantization conversion, inference optimization to RAG integration.