AMD ROCm Local GPU Voice Assistant: Fully Offline Real-Time Streaming LLM Interaction Solution

This project is based on the AMD ROCm platform, integrating the vLLM inference engine, Whisper speech recognition, and Edge-TTS speech synthesis to achieve a fully local, cloud-independent real-time AI dialogue experience. Key advantages include privacy protection (local data processing), offline availability, and GPU-accelerated edge-side inference, providing an alternative for users who value privacy or require offline scenarios.

Most AI assistants rely on cloud APIs, which pose privacy leakage risks and cannot be used offline. This project aims to build a fully privatized AI assistant where all data processing steps (from audio collection to voice output) are completed locally. Choosing AMD ROCm as the platform provides an open-source alternative to avoid vendor lock-in, suitable for enterprise intranets or privacy-sensitive scenarios.

The system uses a pipeline architecture: Microphone audio → Whisper speech recognition to text → vLLM (PagedAttention algorithm) inference to generate responses → Edge-TTS speech synthesis. vLLM supports streaming output to reduce perceived latency; Gradio provides a browser UI supporting text/voice input and auto-play; remote access is possible via SSH port forwarding.

Tested hardware includes AMD Radeon AI PRO R9700 (RDNA4), W7900 (RDNA3), and Ryzen AI MAX 300 series APU (with quantization optimization support). Recommended environment: ROCm7.2 + PyTorch2.11 preview + vLLM0.14. Deployment uses Docker containerization: Pull ROCm vLLM image → Install Gradio/Whisper/Edge-TTS → Download main script (standard/optimized/Ryzen AI version) → Auto-download models (Llama/Whisper/TTS) on first launch.

By default, it uses the DavidAU community's Llama3.3 8B Instruct model (concise responses, strong reasoning), configured with a short output length (160 tokens) and temperature 0.8. System prompts give the assistant the personality "Eva" (witty, dry humor, brief responses), following the principle of "help first, then humor". Users can modify prompts to customize the assistant's personality or replace the model.

Applicable scenarios: Personal privacy protection (local data retention), enterprise intranet deployment (compliance requirements), technical learning platform (friendly for secondary development). The project demonstrates the maturity of AMD ROCm in the AI inference field, providing users with an alternative to NVIDIA CUDA.

Current limitations: Speech synthesis starts only after the full text is generated, causing latency; the 8B model has limited ability to handle complex tasks; multilingual support needs optimization. Improvement directions: Implement streaming speech synthesis (output while generating); support larger models (13B/70B); enhance multilingual support; rely on community contributions to improve functions.