distributed-llm-simulation: Simulation and Implementation of a Distributed LLM Inference System

This is an open-source project simulating a distributed large language model (LLM) inference system, developed by mariamtarek7115. It implements load balancing, GPU worker node management, and fault tolerance mechanisms, adopts the classic Master-Worker architecture, and includes a RAG module and client SDK, providing a reference architecture for building production-grade distributed AI services. Its core goal is to solve the problem that single-machine deployment cannot meet LLM inference requirements, and to provide an efficient, stable, and scalable distributed inference simulation solution.

Modern LLMs have billions or even hundreds of billions of parameters, so a single GPU's memory cannot hold the complete model weights; even if it can, inference latency and throughput will become bottlenecks. Distributed inference breaks through hardware limitations by splitting/replicating the model across multiple GPU nodes. In addition, distributed systems have advantages such as horizontal scaling, redundant deployment (improving availability), and load balancing (making full use of resources), but they also introduce complexities like node communication, task scheduling, and fault recovery.

Adopting the Master-Worker architecture, the core components include:

1. Master Node: The control plane, responsible for receiving requests, assigning tasks, monitoring node status, and maintaining global topology and load information.
2. Worker Node: GPU nodes that execute inference tasks, register capabilities and load with the Master, and manage GPU resources.
3. Load Balancer: Intelligent task distribution, considering factors such as node load, GPU memory, network latency, and task priority.
4. RAG Module: Integrates external knowledge bases and supports enhanced generation based on private data.
5. Client SDK: Encapsulates communication details and simplifies the calling of inference requests.

The project implements multi-level fault tolerance strategies:

• Heartbeat Detection: The Master sends heartbeats periodically, marks unresponsive Workers as offline, and reschedules tasks.
• Task Retry: Automatically retries tasks when they fail (configurable maximum number of retries and backoff interval).
• Checkpoint Mechanism: Saves the intermediate state of long-running tasks and resumes from the checkpoint after a crash.
• Graceful Degradation: Reduces concurrency and limits queue length when nodes are insufficient to ensure core functions are available.

The project uses a modular design with a clear directory structure:

• master/: Master node implementation (HTTP API, scheduler, registry)
• workers/: Worker node implementation (model loading, inference execution, resource management)
• load_balancer/: Load balancing algorithms (round-robin, least connections, weighted, etc.)
• rag/: RAG functions (document indexing, vector retrieval, context assembly)
• client/: Client SDK
• common/: Shared data structures and utility functions
• llm/: Core LLM inference logic (can interface with different backends) The modular design facilitates understanding, maintenance, and component replacement.

Applicable Scenarios:

1. Distributed LLM Service Builders: Reference architecture design and implementation details (task scheduling, fault handling, etc.).
2. Distributed System Learners: Case study to understand principles like heartbeat detection, task queues, and fault handling through code.
3. Experimenters: Test distributed LLM strategies (sharding schemes, scheduling algorithms) locally without expensive GPU clusters.

Current Limitations:

• Uses simplified models or simulated latency, not connected to real LLM inference engines; performance data is for reference only.
• Network communication uses local inter-process communication or simple HTTP, which may become a bottleneck under high concurrency. Improvement Directions:
• Integrate with real inference backends (vLLM, TensorRT-LLM).
• Implement complex model parallelism strategies (tensor parallelism, pipeline parallelism).
• Add monitoring and observability.
• Support dynamic scaling.

distributed-llm-simulation provides a clear reference implementation for distributed LLM inference systems, integrating functions like load balancing, fault tolerance, and RAG. Its core design ideas (modularity, fault tolerance first, separation of concerns) are best practices for production-grade systems. Despite its limitations, it has important reference value for developers exploring distributed AI infrastructure.