Groove：去中心化大模型推理网络的架构与实践

Groove is an open-source decentralized LLM inference network that aggregates computing resources from multiple machines into a distributed cluster. This post will break down its architecture, communication protocols, deployment practices, and application prospects.

With the growing scale of large language models (LLMs), single-machine inference faces dual bottlenecks of memory and computing power. Groove proposes an innovative solution: using a decentralized network to aggregate resources from multiple machines for distributed model inference. This reduces reliance on high-performance hardware and opens new possibilities for edge computing scenarios.

Groove uses a three-layer architecture:

1. Relay Layer: Coordination center for routing and task distribution, bound to port 0.0.0.0:8770, with centralized coordination and distributed execution (only relay exposes ports).
2. Compute Node Layer: Work units executing inference, loading partial model layers (via --layers parameter, e.g., 0-11 for Qwen2.5-0.5B), supporting CPU, CUDA, MPS backends.
3. Consumer Layer: Client initiating inference requests, abstracting model distribution details for scalability.

Groove implements custom Wire Protocol v2 (msgpack serialization, envelope routing) addressing distributed challenges:

• Tensor transfer optimization for model weights/activations.
• KV cache management for multi-turn dialogues.
• Optional speculative decoding for acceleration. All traffic routes via relay (no direct node communication), simplifying security (only protect relay port; nodes can be behind NAT/firewall).

Deployment steps:

1. Env prep: bash setup.sh to install virtual env and dependencies.
2. Start relay: Activate env and run relay service.
3. Start compute nodes: Launch nodes with specified layers and relay address.
4. Initiate inference: Use consumer client to send requests. Auxiliary functions: --status (health check), --test (test suite), --smoke (light model test), --info MODEL (model info & layer split recommendations).

Key technical choices:

• Model parallelism: Unlike data parallel training, Groove uses model parallelism for inference (distribute different layers to nodes, suitable for sequential forward propagation).
• Zero-config network: Compute nodes only make outbound connections (no port forwarding/firewall complexity).
• Cross-platform support: Works on Linux, macOS, Windows; supports CPU, CUDA, MPS backends.

Groove is ideal for:

• Edge computing clusters (aggregate edge devices into inference pools).
• Heterogeneous hardware utilization (mix GPU servers and CPU workstations).
• Privacy-sensitive scenarios (local data processing, no cloud upload).
• Model-as-service (build decentralized inference markets).

Groove provides a lightweight, easy-to-deploy solution for distributed LLM inference. Though in early stages, its clear architecture and practical engineering choices are noteworthy. It's a valuable reference implementation for developers/researchers exploring decentralized AI infrastructure.