High-Performance Distributed LLM Inference Gateway Built with C++17

This article introduces a high-performance distributed LLM inference gateway built with C++17, aiming to address challenges such as high-concurrency request handling and streaming generation failover in LLM deployment. The gateway uses gRPC for streaming transmission, SWIM protocol for decentralized member management and fault detection, supports weighted least connection load balancing and mid-stream failover, and provides a lightweight yet fully functional basic framework for production-grade LLM inference services.

With the widespread deployment of LLMs in various applications, traditional monolithic deployment struggles to handle high-concurrency requests, and simple load balancing cannot deal with failover during streaming generation. The distributed LLM inference gateway project addresses these challenges by providing load balancing, fault tolerance, and token streaming support to ensure users' real-time generation experience.

The system adopts a two-layer communication model: the first layer is gRPC over TCP, used for inference traffic transmission between clients and the gateway, as well as between the gateway and replicas, leveraging gRPC's server-side streaming capability to reduce latency; the second layer is the UDP-based SWIM gossip protocol, which implements peer-to-peer fault detection and member management between replicas without the need for a centralized health monitor, ensuring efficient fault detection through indirect probing, suspicion mechanisms, etc. In the architecture, clients connect to the gateway via gRPC streams; the gateway includes a load balancer, request queue, and member management subscriber; replicas synchronize status via UDP gossip.

The gateway implements several key features: 1. Weighted least connection load balancing: routes requests to the least loaded instance based on load metadata propagated via gossip; 2. Token streaming: forwards tokens generated by replicas to clients in real time; 3. Mid-stream failover: transparently reroutes requests when a replica goes down, with no user perception; 4. Backpressure mechanism: prevents system overload via FIFO queues and per-replica concurrency limits; 5. Rolling updates: gracefully drains a replica before restarting with a new version, ensuring zero request loss.

The project uses C++17 for ultimate performance, gRPC and Protobuf for RPC communication, and raw UDP sockets with Protobuf encoding for gossip transmission. The build system uses CMake and Makefile, and documentation is generated via Doxygen. The LLM backend is a simulated implementation: after receiving a prompt, it returns tokens with a configured delay, which is convenient for testing and demonstration, and reserves an interface for real LLM engines.

This gateway is suitable for high-availability LLM service scenarios, such as chatbots, code completion tools, real-time writing assistants, and other applications that need to handle a large number of concurrent streaming requests. For operation and maintenance teams, decentralized fault detection eliminates single points of failure, and weighted load balancing and backpressure mechanisms optimize resource utilization, avoiding replica overload or idleness.

The distributed LLM inference gateway combines gRPC's efficient transmission, SWIM protocol's decentralized coordination, and carefully designed load balancing strategies to address modern AI infrastructure challenges, providing a lightweight and complete framework for production-grade LLM inference services. For teams building their own LLM inference infrastructure, it is an open-source project worth studying and referencing.