ARC-AI: Real-Time Multimodal AI Assistant Integrating Audio Streams, RAG Memory, and Autonomous Workflows

ARC-AI (Autonomous Real-time Conversational AI) is a full-stack digital assistant that breaks through the traditional chatbot paradigm. Developed using the MERN tech stack, it has proactive and autonomous features, capable of executing background tasks, conducting real-time web research, remembering user preferences, and actively controlling the front-end interface. Its core functions include real-time voice interaction, RAG vector memory, web search, scheduled task scheduling, dynamic front-end UI control, and WhatsApp automation. It adopts an intelligent provider routing architecture to support dynamic switching between Gemini and Mistral models, and implements interruption-safe streaming and multi-agent workflows.

The project's back-end architecture has transitioned from a tightly coupled single-provider system to an extensible provider-agnostic AI runtime. The core is an intelligent provider routing mechanism that dynamically selects models like Gemini (good at multimodal reasoning) or Mistral (fast and low-cost) based on task characteristics. This architecture addresses the security issues of multimodal capabilities, gracefully degrades when no compatible provider is available, and interruption-safe streaming ensures proper management of token lifecycles and resource cleanup when users interrupt conversations.

The real-time interaction pipeline uses a low-latency design, combining REST API and WebSocket, supporting voice (Web Speech API)/text input and transmitting via Socket.IO. The visual capability can capture camera frames and dynamically understand them via the Pixtral model. The RAG memory system uses Mistral to generate vector embeddings stored in Pinecone, supporting semantic search, remembering user information and preferences to achieve personalized interaction. Semantic matching is more flexible and intelligent than keyword search.

A complete tool calling framework is implemented, which autonomously decides to call tools like web search (scraped via Cheerio) and weather queries. It supports converting natural language into scheduled tasks and automatically creating cron jobs for background execution. The dynamic front-end UI control capability can switch themes, open websites, play media, etc., transforming the AI from an information provider to an agent that proactively operates the user's environment.

WhatsApp automation achieves reliable message delivery via Google Apps Script webhooks. The persistent workspace supports cross-session conversations, message history storage and retrieval, real-time switching, and pagination. The MongoDB architecture manages the conversation lifecycle (asynchronous title generation, soft deletion archiving, metadata tracking), and the responsive UI adapts to multi-device layouts.

Tech Stack: Back-end: Node.js + Express + Socket.IO + node-cron; Front-end: React18 + Vite + Tailwind; Databases: MongoDB Atlas + Pinecone; AI Models: Mistral/Pixtral; Infrastructure: Google Apps Script. Application scenarios include personal assistants, smart home control, automated workflows, research agents, etc. The open-source MIT license provides a fully functional reference implementation.

Plans include adding multi-agent cluster workflows, GPT-4o real-time vision, a richer tool ecosystem, and deep third-party service integrations. The intelligent routing architecture lays the foundation for accessing more providers like OpenAI, Claude, Groq, Ollama, etc., representing the trend of AI evolving from question-answering systems to intelligent agents.