Hermit: An Open-Source Chat App for Running Local Large Language Models on Mobile Phones

Hermit is an open-source mobile chat application developed with React Native and Expo. Its core feature is supporting the local execution of GGUF-format large language models on devices via llama.rn, while also being compatible with remote OpenAI-compatible APIs. It balances privacy protection (data remains local) and usage flexibility, providing users with an offline AI chat experience.

• Original Author/Maintainer: stargazer617
• Source Platform: GitHub
• Project Positioning: Designed specifically for users who want to experience large language model chat on mobile devices, enabling local AI chat functionality without relying on cloud services.

Local Model Inference Support

By integrating the llama.rn library, it enables local LLM inference on mobile devices, supports GGUF-format models, and keeps chat data local to ensure privacy.

Dual-Mode Architecture

1. Local Mode: Uses device NPU/CPU for inference, suitable for offline or high-privacy scenarios;
2. Remote Mode: Compatible with OpenAI-format APIs, allowing connection to self-hosted services or third-party providers.

Advantages of Development Framework

Using React Native + Expo, it has cross-platform capabilities (iOS/Android) and simplifies the build and deployment process.

llama.rn Integration

llama.rn is a React Native binding for llama.cpp, encapsulating the C++ inference engine into JS interfaces to balance performance and development experience.

Model Format Support

Supports the GGUF format, which has moderate file size, fast loading speed, and low memory usage—ideal for mobile devices.

• Privacy-First Scenarios: Handling sensitive information (medical consultation, legal advice, etc.) where chat content never leaves the device;
• Offline Environments: Providing continuous AI services when the network is unstable or unavailable (long flights, remote areas);
• Development and Testing: Quickly testing the performance of different GGUF models on mobile devices, evaluating the balance between quantization accuracy and inference speed.

Addressing Mobile Resource Constraints

• Supports 4/5/8-bit quantized models to reduce memory usage;
• Optimizes loading strategies (on-demand loading + caching);
• Provides model size recommendations to help users select models suitable for their devices.

Inference Performance Optimization

Under the hood, it uses the NEON instruction set (ARM architecture) and Metal GPU acceleration (iOS) to efficiently utilize resources and control power consumption.

Model Ecosystem

Compatible with GGUF models from platforms like Hugging Face, including series such as Llama2/3, Mistral, and Qwen.

API Compatibility

Supports OpenAI-compatible APIs, allowing integration with services like OpenRouter, Together AI, and local vLLM.

Hermit represents an important direction for mobile AI applications: bringing LLM capabilities to mobile devices while protecting privacy. As mobile chip performance improves and model quantization technology advances, the local running experience will continue to improve. It provides a fully functional, easy-to-use open-source solution for developers and users exploring local AI.