asiai-inference-server: Fleet Management Hub for Local LLM Inference on Apple Silicon

asiai-inference-server is a management tool for LLM inference engines designed specifically for Apple Silicon. Its core purpose is to address the pain point of unreleased VRAM caused by macOS's unified memory compressor. It provides installation, startup, stop, uninstallation, and orchestration functions, supporting single-machine or multi-machine cluster control. It is the control plane companion of the asiai observation tool, facilitating efficient operation and maintenance of local AI workflows.

When running local LLMs on Apple Silicon Macs, the compressor of macOS's unified memory architecture causes VRAM to remain reserved even after the process terminates, leading to memory shortages when switching models frequently. Additionally, installing and managing multiple inference engines (such as Ollama, LM Studio) involves tedious command-line operations and configurations, with no unified control plane available.

asiai-inference-server is the control plane project of asiai (Apple Silicon AI Observation/Benchmarking CLI). It is responsible for managing the full lifecycle of inference engines (installation, startup, stop, uninstallation, orchestration). Its core mission is to deterministically reclaim memory through engine uninstallation APIs, LaunchDaemon restarts, and the sudo purge command, while supporting single-machine/multi-machine cluster management.

Key requirements summarized from practical experience:

1. Simplify engine lifecycle management, avoiding tedious commands and configurations;
2. One-click configuration file switching for quick model changes;
3. Truly release VRAM instead of relying on the system compressor;
4. Unified cluster dashboard to manage multiple Mac devices;
5. Support MCP protocol for integrating AI agents to autonomously manage clusters.

Adopting a layered architecture, core features include:

• Dual CLI modes: independent aisctl tool and asiai engine subcommands;
• Pure Python standard library: only depends on Python standard libraries, with optional MCP support;
• Apple Silicon-specific: relies on macOS tools like launchctl, vm_stat, sudo purge;
• SSH-prioritized cluster operations: v0.3 implements SSH-based multi-Mac inventory management and command distribution;
• Configuration formats: TOML (human-editable) and JSON (runtime state).

Three key scenarios:

1. Rapid switching in development environments: One command to switch models and release memory;
2. Multi-machine cluster inference: Unified task scheduling, assigning devices based on model size and load;
3. Autonomous management by AI agents: Through the MCP protocol, AI assistants automatically select models, start services, and clean up resources.

Currently in the v0.0.1 pre-alpha phase, the roadmap is as follows:

Version	Function Scope	Status
v0.0	Repository skeleton + packaging	In progress
v0.1	Installation/uninstallation/startup/stop + memory cleanup	Next version
v0.2	Configuration file switching (TOML application/rollback)	Planned
v0.3	Cluster manager (multi-Mac inventory, SSH distribution)	Planned
v0.4	Web cockpit + optional HTTP proxy	Planned
v1.0	MCP writing tool + PyPI/Homebrew release	Planned

The project uses the Apache-2.0 license and was created by Jean-Marc Nahlovsky. As part of the Apple Silicon AI ecosystem, it complements the asiai observation tool, addresses the operational challenges of local LLM deployment, and provides key infrastructure supplements for macOS local large model users.