Infer-App: A Native macOS Local Large Model Chat App Integrating Voice, RAG, and Agent Runtime

Infer-App is a native local large model chat application designed specifically for macOS, integrating the dual inference engines of llama.cpp and MLX. It supports on-device speech recognition, RAG retrieval augmentation, and an Agent runtime based on the MCP protocol. Key advantages include fully offline operation to ensure privacy, native macOS experience, and a flexible technical architecture, providing users with a one-stop local AI assistant solution.

Infer-App is positioned as a native macOS application, aiming to combine LLM local execution capabilities with modern interactive experiences. The interface is built using SwiftUI, follows macOS design guidelines, and supports native features such as dark/light mode, multi-window layout, and system shortcuts. It deeply integrates with the macOS ecosystem, enabling interaction with Spotlight and Shortcuts, supporting drag-and-drop file import, and providing a seamless system-level experience.

Infer-App adopts a dual-engine inference solution: the llama.cpp engine provides wide model compatibility and efficient quantized inference, supporting open-source models like Llama and Mistral; the MLX engine is optimized for Apple Silicon, leveraging the unified memory and Neural Engine of M-series chips to enhance performance. Additionally, it includes built-in on-device speech recognition (completed locally to ensure privacy), a RAG system (supporting local document import and semantic retrieval), and an Agent runtime based on the MCP protocol (capable of calling local operations and external tools).

In terms of privacy, all inference, speech recognition, and document processing are completed offline, data is stored locally, users can import models on their own, and the code is fully open-source and auditable. For performance, it is optimized for Apple Silicon: using unified memory to reduce copying, Metal-accelerated computing, intelligent scheduling of CPU/GPU resources, and a low-power mode to extend battery life, ensuring smooth operation on consumer-grade Macs.

Infer-App is suitable for multiple types of users: privacy-first groups (such as lawyers and doctors who handle sensitive information), offline workers (usable even without a network), developers and tech enthusiasts (who want to deeply understand LLM architecture), and power macOS users (who pursue native experiences).

Technical implementation highlights include cross-language binding (Swift/C++/Python interaction), intelligent resource management (dynamic model loading, memory pressure awareness), and a modular architecture (customizable function plugins). The project provides a reference for local LLM applications; its architectural design and system optimization experience are valuable for developers and promote the development of the local AI assistant ecosystem.

Infer-App integrates a variety of advanced technologies, providing a level of feature richness close to cloud services while ensuring privacy. As an open-source project with a rich tech stack and advanced design concepts, it proves that consumer-grade devices can run a complete AI assistant, offering a high-quality choice for macOS users who value data sovereignty and native experiences.