mlx-serve：Apple Silicon上的纯Zig原生LLM推理服务器

mlx-serve is a pure Zig-language LLM inference server optimized for Apple Silicon (M1/M2/M3/M4), free of Python dependencies. It provides OpenAI and Anthropic compatible APIs, and includes a macOS GUI app MLX Core. Key features include lightweight deployment, high performance, tool calling, and multi-modal support.

Python's dominance in AI inference brings deployment complexity and bloated dependencies. mlx-serve uses Zig to directly call Apple's MLX C interface, avoiding Python runtime overhead. Its "No Python" design ensures native execution from model loading to token generation, aiming for faster startup, lower memory usage, and simpler deployment.

mlx-serve offers HTTP APIs compatible with OpenAI (supports /v1/chat/completions, /v1/completions, /v1/models) and Anthropic (/v1/messages). It supports both streaming and non-streaming responses, plus tool calling for external tool integration.

Key optimizations include KV cache reuse (boosts multi-turn dialogue speed), full sampling parameter control (temperature, top-k, top-p, etc.), and integration of Gemma4's SigLIP encoder for multi-modal reasoning via image_url content blocks. It also supports reasoning/thinking mode with configurable token budget.

MLX Core is a macOS menu bar app with: model browser (HuggingFace download with breakpoint续传, architecture detection), multi-session chat (Markdown rendering), agent mode (10 built-in tools like shell, file operations, web search), customizable system prompts, persistent memory, and skill system (add .md files to ~/.mlx-serve/skills/).

Supported architectures: Gemma4/3, Qwen3/3.5/3.6, Nemotron-H, Llama, Mistral (examples: gemma-4-e2b-it-4bit, Llama3). Installation options: Homebrew (brew tap ddalcu/mlx-serve then install mlx-core and mlx-serve) or source build (zig build -Doptimize=ReleaseFast, then run with model path and port). API example via curl is provided.

mlx-serve represents a direction of using system-level languages (Zig) to rebuild AI infrastructure,摆脱Python dependency. For Apple Silicon users, it's a high-performance local LLM solution. Its agent and tool calling features make it a full local AI assistant platform. As model quantization and Apple Silicon improve, such native solutions will play a bigger role in edge AI.