oMLX: A Local LLM Inference Server Optimized for Apple Silicon

oMLX is a local LLM inference server designed for macOS and Apple Silicon. It optimizes performance using hierarchical KV caching and continuous batching technologies. It supports multiple types including text LLMs, vision-language models (VLM), and embedding models. It provides menu bar management and a Web UI, enabling privacy protection and convenient operation for local deployment, suitable for developers, researchers, and AI enthusiasts.

Existing LLM server solutions compromise between convenience and control—either they are simple but lack configuration options, or complex requiring command-line operations. oMLX aims to solve these issues: it supports fixed memory for commonly used models, on-demand switching of large models, flexible context limits, and all operations can be done via the menu bar. Its hierarchical KV caching strategy stores hot data in memory and offloads cold data to SSD, reuses historical context across requests, and adapts to programming scenarios (e.g., working with Claude Code).

1. Hierarchical KV caching architecture: Block-level management inspired by vLLM. Hot cache (RAM) stores frequently accessed blocks to ensure response speed; cold cache (SSD) stores overflow blocks in safetensors format, which can be restored after restart, breaking through memory limits.
2. Continuous batching: Dynamically optimizes prefill/generation batch sizes via mlx-lm's BatchGenerator, supporting concurrent requests.
3. Context scaling: Adapts to Claude Code scenarios, adjusts token count to trigger automatic compression, and uses SSE keep-alive to prevent timeouts.

• Multi-model support: Text LLMs, VLMs (multi-image dialogue, OCR optimization), embedding models, re-ranking models;
• Intelligent model management: LRU eviction, manual loading/unloading, model pinning, per-model TTL, process memory limit (system RAM minus 8GB);
• Web management panel: Real-time monitoring, model management, built-in chat interface (supports VLM image upload), model downloader (HuggingFace), benchmark testing, tool integration configuration, supports multi-language and offline operation.

Installation methods: 1. DMG package: Drag to Applications, supports auto-update; 2. Homebrew: Tap the repository then install, can manage background running via services; 3. Source code: Clone the repository then pip install. System requirements: macOS15.0+, Python3.10+, Apple Silicon. Quick start: Set model directory → Start server → Download model. Compatible with OpenAI API clients (address: http://localhost:8000/v1), built-in chat interface at /admin/chat.

Application scenarios: Local AI-assisted programming (privacy protection, usable without network), offline document processing (VLM/OCR analysis of sensitive documents), private knowledge base Q&A (RAG technology), model development and testing (quick switching of models and parameters). Best practices: Cache tuning (increase hot cache for short conversations, use cold cache for long contexts), model selection (7B-13B suitable for daily use, large models use hierarchical cache), concurrency configuration (conservative settings for M1/M2, aggressive attempts for M3/M4).

Future directions: Multi-device distributed inference, support for more model formats like GGUF, advanced quantization compression technology, plugin ecosystem. Community contributions: Open-source under Apache2.0 license. We welcome performance testing, multi-language translation, document improvement, bug reports, model compatibility testing. You can participate via GitHub Issues/Discussions or submit PRs.