Mesh LLM: A Multi-Machine Distributed Inference Framework Based on llama.cpp, Enabling GPU Resource Pooling and Sharing

Mesh LLM is an open-source distributed inference framework based on llama.cpp. Its core goal is to enable multi-machine GPU resource pooling and sharing, support pipeline parallelism and expert parallelism strategies, and provide an OpenAI-compatible API, allowing multiple machines to collaboratively run ultra-large models. It aims to address the pain point where single-GPU or single-machine GPUs cannot meet the inference requirements of large models, and lower the technical threshold for distributed inference.

As the scale of large language models continues to expand, single-GPU or single-machine GPUs can no longer meet inference requirements. Traditional distributed inference solutions are complex to configure and require professional cluster management experience. Mesh LLM addresses this pain point by allowing users to pool the GPU capacity of multiple machines, expose a unified OpenAI-compatible API endpoint externally, and follow a simple design philosophy—after starting one node, machines can be added at any time, and the system automatically handles load balancing and model sharding.

Mesh LLM is built on llama.cpp and adopts different parallelism strategies for different models: pipeline parallelism for dense models (model layers are distributed across different nodes based on memory), and expert sharding for mixture-of-experts models (zero cross-node inference traffic). Key design points include: each node locally provides the same API endpoint to simplify access; intelligent routing prioritizes local execution (when the model can run on a single machine), and only triggers distributed sharding when capacity is exceeded; for latency optimization, llama-server runs on the same machine as the GPU, so cross-network latency only affects the generation of the first token and does not impact subsequent throughput.

Mesh LLM implements several performance optimizations: model loading uses zero-transfer GGUF loading technology, reducing loading time from 111 seconds to 5 seconds; RPC communication reduces round trips per token from 558 to 8 via caching and skipping intermediate lookups; supports direct server-to-server tensor transfer; speculative decoding can increase throughput by 38% in code generation scenarios (with a 75% acceptance rate).

Mesh LLM supports simultaneous multi-model services: the API proxy routes requests via the model field, and the /v1/models endpoint lists available models. The system has demand-aware dynamic rebalancing capabilities, propagating demand signals via the gossip protocol (with TTL decay). When a model loses its serving node, a standby node automatically takes over within approximately 60 seconds.

Mesh LLM offers multiple usage modes: for beginners, use mesh-llm serve --auto for automatic configuration; create a private mesh using mesh-llm serve --model to generate an invitation token; GPU-less machines can join as pure clients; supports named mesh collaboration; provides macOS launchd and Linux systemd background services; configuration files use TOML format to preset models and plugins.

Mesh LLM supports multimodal inference, including visual models like Qwen3-VL and audio models like Qwen2-Audio, and supports image/audio/file attachment requests (large attachments use range blob uploads). For ecosystem integration, it has built-in support for AI Agent tools like Goose and Claude Code. These tools can reuse existing meshes or automatically start client nodes to seamlessly use distributed inference capabilities.

Mesh LLM is a practical and easy-to-use open-source distributed inference solution. Through resource pooling, flexible parallelism strategies, and simple deployment, it lowers the threshold for multi-machine collaborative inference. The project is built with Rust and Node.js, supports backends like CUDA and ROCm, and has good cross-platform compatibility. For researchers and developers, it is an excellent choice for utilizing scattered GPU resources. Its open-source nature facilitates continuous community improvement and promotes the development of distributed AI infrastructure.