Plexus: Heterogeneous GPU Mesh Enables Verifiable LLM Inference, Allowing Home Labs to Run Data Center-Scale Large Models

Plexus is an open-source Rust-based project that aggregates GPU resources from ordinary machines by building a heterogeneous GPU mesh network to enable verifiable LLM inference. It addresses the cost, privacy, and hardware limitations of large model inference, allowing developers to run data center-scale large language models within the budget of a home lab.

As the capabilities of large models like GPT-4 improve, the hardware requirements for running them increase. Relying on cloud APIs has issues with data privacy, network latency, and ongoing costs; local operation faces GPU resource barriers—data center-grade graphics cards are expensive and power-hungry, while consumer-grade cards lack sufficient VRAM.

The core of Plexus's innovative solution includes: 1. Mesh Network Layer: Connects heterogeneous GPUs to form a unified computing pool with automatic load balancing and scheduling; 2. Model Sharding and Parallel Inference: Intelligently shards model parameters across multiple GPUs and coordinates communication for parallel computing; 3. Verifiable Inference: Generates cryptographic proofs to verify the correctness of inference and ensure results are untampered.

Plexus is written in Rust, leveraging its memory safety and concurrent performance advantages; communication optimization uses gradient compression, pipeline parallelism, and topology-aware scheduling to reduce overhead; fault tolerance mechanisms save state via checkpoints, allowing recovery from node failures.

1. Home Labs: Connect graphics cards like RTX 4090/3090 to run billion-parameter models for research; 2. Privacy-Focused Enterprises: Industries like finance and healthcare can deploy locally, keeping data within internal networks, with verifiable inference enhancing security; 3. Edge and Offline Environments: Run AI services in network-free scenarios such as ships and remote areas.

Plexus is open-sourced under the Apache 2.0 license, with its GitHub repository providing build guides and example configurations; as a Rust ecosystem project, it encourages community contributions of code, documentation, and usage experience.

Plexus represents the direction of LLM infrastructure expanding from centralized data centers to distributed edges, and the "divide and conquer" inference model may become mainstream; its codebase is a valuable resource for learning distributed AI systems and verifiable computing.