Mithrandir: A Complete Guide to Building a Local Privacy-First AI Assistant Based on Gemma 4

Mithrandir is an open-source local AI assistant project with a core focus on privacy-first and local hosting. It uses Gemma 4 for local inference (with Claude as a backup engine) and aims to replicate the capabilities of cloud-based AI assistants on consumer-grade hardware. This project is not just usable software but also a detailed build log with comprehensive documentation, providing learners with a reference for the actual building process.

The project name is derived from Gandalf in The Lord of the Rings, symbolizing wise guidance. Its core positioning is a privacy-first, locally hosted AI assistant, emphasizing building a 'system that runs LLMs' rather than the LLM itself to lower the entry barrier: no need to understand the mathematical details of Transformers—just integrate existing models into the application stack.

Adopts a hybrid local+cloud architecture:

• Local inference: Gemma 4 26B (MoE) runs via Ollama, achieving 144 tokens/sec on RTX4090 (4x faster than cloud APIs)
• Cloud backup: Complex tasks are routed to the Claude API
• Agent framework: ReAct loop + Pydantic tool calling
• Memory system: ChromaDB (vector database) + SQLite (long-term memory + code RAG)
• Market module: HMM market state recognition + EDGAR quantitative screening (covers 9.8K SEC filings)
• Voice interaction: faster-whisper recognition + F5-TTS cloning + Kokoro TTS
• UI: React/FastAPI/WebSocket frontend + Telegram bot

Hardware requirements are user-friendly:

Component	Minimum Configuration	Recommended Configuration
GPU	NVIDIA 8GB VRAM	NVIDIA 20GB+ VRAM
Memory	16GB	32GB+
Storage	50GB	100GB+
OS	Win10/11 or Linux	Win11 or Ubuntu
Model selection: Gemma4 26B runs on 24GB VRAM; the e4b variant (4B active parameters) runs on 8GB VRAM.

Core capabilities include:

• SEC filing analysis: Reads public company filings to assist fundamental research
• Market tracking: Real-time data + HMM market state recognition
• Persistent memory: Maintains dialogue context coherence
• Voice cloning: Uses F5-TTS technology for voice cloning
• Codebase RAG: Understands personal codebases to provide programming assistance

The project documentation has great learning value:

• JOURNEY.md records the complete build process (steps, bugs, solutions)
• Key FAQ questions:
  1. Accessible to ordinary users (requires NVIDIA GPU with 8GB+ VRAM)
  2. Cost is only electricity (zero API fees)
  3. Performance on daily tasks is comparable to cloud-based assistants; Claude is better for complex reasoning
  4. Data does not leave the device by default

The deployment guide is complete, taking an estimated 1-2 hours (mainly for model downloads). Access methods: Browser (custom frontend) or iPhone (Telegram bot).

It represents the direction of AI democratization: enabling individuals to run powerful AI systems on local hardware without relying on cloud service providers. The improved capabilities of open-source models (Gemma/Llama/Qwen) plus advances in consumer-grade GPUs make local AI more practical. As a complete reference implementation, this project lowers the barrier for developers to enter the agentic AI field.