Voice-Assistant: An End-to-End Voice Dialogue System Based on Local Large Language Models

Voice-Assistant is a local end-to-end voice dialogue system developed and open-sourced by FredieBrunn. It integrates Whisper speech recognition, Ollama local large language model, and pyttsx3 speech synthesis, and achieves a complete voice interaction loop via Flask REST API and web interface. Its core concept is full-process local operation to ensure user data privacy, supporting flexible configuration and function expansion.

The project aims to build an intelligent voice assistant that does not rely on cloud services, solving the data privacy issues of cloud services. All AI components are executed locally to ensure user data privacy and system accessibility, realizing a complete interaction loop from voice input collection, recognition to text, LLM inference to generate responses, and speech synthesis output, providing a natural dialogue experience similar to mainstream instant messaging tools. The project has been open-sourced on GitHub.

The system adopts a modular design, divided into three core service layers:

1. Speech Recognition Layer: Uses OpenAI's open-source Whisper model, supporting multiple sizes (from 75MB tiny to 3GB large-v3), allowing users to flexibly configure based on hardware conditions and accuracy requirements;
2. Language Understanding and Generation Layer: Runs large language models locally via Ollama, supporting open-source models like Llama and Mistral (e.g., 640MB TinyLlama, 4GB Llama3). French users can choose the optimized Mistral model;
3. Speech Synthesis Layer: Uses the cross-platform pyttsx3 library, supporting multi-language voice packs (e.g., French), with good compatibility on Windows, Linux, and macOS.

Environment Dependencies

Requires Python 3.9+, Ollama, ffmpeg, espeak (install via package manager on Linux/macOS).

Installation Process

An install.sh automatic installation script is provided; manual installation requires cloning the repository → creating and activating a virtual environment → installing Python dependencies → installing and starting Ollama and pulling models → starting the Flask backend → accessing the frontend interface.

Configuration Options

You can switch Whisper models (MODEL_STT/WHISPER_MODEL) and LLM models (OLLAMA_MODEL) by modifying variables or environment variables, and customize the service port.

The Flask backend exposes RESTful APIs:

• GET /health: Health check, returns Whisper model version, Ollama connection status, and list of available models;
• POST /transcribe: Receives audio data and returns recognized text;
• POST /chat: Receives text and language parameters and returns LLM responses;
• POST /transcribe_and_chat: Directly receives audio and returns recognition results and AI responses. The layered design supports full use or independent integration.

Advantages of full-process local operation:

• Data Privacy: Voice input and dialogue content are not uploaded to external servers;
• Offline Availability: No internet connection required after models are downloaded;
• Cost Control: No API call fees, suitable for high-frequency use;
• Customizability: Users can freely modify and expand the system without being restricted by commercial services.

Application scenarios include:

• Personal intelligent assistant (desktop voice interaction entry);
• Accessibility tools (voice control for visually impaired or mobility-impaired individuals);
• Educational assistance (pronunciation practice and dialogue simulation for language learning);
• Smart home control (integration with Home Assistant);
• Enterprise private deployment (customized voice services for internal networks). Outlook: With the performance improvement and size optimization of local large language models, local-first AI applications will become more practical and popular.

Voice-Assistant integrates three mature open-source components: Whisper, Ollama, and pyttsx3, and implements end-to-end local voice dialogue capabilities with concise code. It is a reference case for developers to understand the architecture of voice AI systems, and a practical choice for users who need to deploy voice assistants locally.