VoxVision.ai: Architecture Design and Intelligent Routing Strategy of a Multimodal AI Assistant

VoxVision.ai is a multimodal AI assistant launched by Oxlo, integrating voice, visual, text, and image generation capabilities. It achieves natural multimodal interaction through intelligent model routing and multi-model degradation mechanisms. This article will analyze its architectural design, core capabilities, and innovative points.

Traditional AI systems are mostly unimodal (e.g., chatbots handle text, speech recognition handles audio), which struggle to meet users' complex needs. Human cognition is inherently multimodal, so VoxVision.ai mimics natural interaction methods, with the ability to listen, see, speak, and generate visual content—distinguishing it from unimodal applications.

Covers four interactive modes:

1. Voice Mode: Dual-engine STT (Sarvam Saaras v3 prioritizes Indian languages, Groq Whisper v3 Turbo serves as a backup for general languages), intelligent TTS routing (Kokoro 82M for English/Latin languages, gTTS for Indian languages), supporting composite request processing
2. Visual Mode: Personalized greetings (generated by Kimi K2.5 analyzing the first frame), intelligent intent routing (captures new frames for analysis of visual questions; skips the camera for non-visual questions), real-time object detection (YOLOv11)
3. Creative Visual Features: What If (scene re-imagination), Biographies (fictional biographies of objects), Director (generates movie posters)
4. Image Generation: img2img (style transfer), text2img (text-to-image generation)

• Multi-model degradation chain: The large language model layer includes Kimi K2.5 (primary), Qwen3 32B (voice-specific), DeepSeek R1 70B (backup), etc., ensuring high availability
• Voice processing flow: User voice → WebM recording → STT engine selection → Text cleaning → Intent classification → Model selection → Anti-hallucination check → TTS engine selection → Audio playback
• Visual processing flow: Camera activation → Capture first frame → Kimi K2.5 analysis → Personalized greeting → Listening → Voice input → STT → Intent routing (visual/non-visual branch) → TTS output
• Tech stack: Backend Python3.11 + FastAPI; Frontend React19 + TypeScript + Vite + Tailwind CSS

• Native local language support: Indian languages (e.g., Kannada) output in native scripts instead of Latin transliteration
• Optimized intelligent intent routing: Skips the camera for non-visual questions, reducing response time by 2-5 seconds
• Recapture feedback mechanism: Proactively requests users to adjust their position when images are blurry
• Single API key convenience: Access multiple models via Oxlo.ai's multi-model API

• Limitations: Heavy reliance on Oxlo API, limited offline capabilities, insufficient complex visual reasoning, weak multi-user support
• Improvement suggestions: Enhance local model support, deepen visual reasoning capabilities, expand multi-user session context memory

• Application scenarios: Education (multimodal homework feedback), creative industry (concept map generation), assistive technology (environment description for visually impaired), customer service (photo + voice question support)
• Future outlook: Multimodal AI will better adapt to natural human interaction; VoxVision.ai serves as a reference architecture to promote more intuitive AI interaction experiences