Practice of Building a Distributed Large Language Model Inference System Based on Slurm and Ray

This article explores how to build a multi-node GPU distributed large language model inference system on an HPC cluster by combining Slurm resource scheduling, Ray distributed computing framework, and vLLM inference engine. It addresses the problem of insufficient GPU memory on a single node, enables cross-machine GPU collaborative computing, improves inference throughput, and maintains model accuracy.

Modern large language models (such as GPT-4, LLaMA-3) have parameter scales ranging from tens of billions to hundreds of billions. A single GPU's memory (40GB-80GB) cannot accommodate the complete model weights and activation values. Traditional solutions (quantization, sharding) either sacrifice accuracy or increase latency. Although distributed inference can solve this problem, it faces challenges such as complex resource scheduling, high network communication overhead, difficulty in fault isolation, and environment consistency.

A layered and progressive architecture is adopted:

1. Slurm Resource Scheduling Layer: Responsible for node allocation, resource isolation, queue management, and environment preparation. Nodes are applied via sbatch.
2. Ray Cluster Management Layer: The Head node manages the global state, Worker nodes report GPU resources, and cluster communication and scheduling are verified.
3. vLLM Inference Engine Layer: Supports tensor parallelism (multi-GPU on a single node) and pipeline parallelism (cross-node) to improve memory utilization and throughput.
4. Distributed Model Execution Layer: Combines Ray and vLLM to implement multi-node inference and dynamic scaling.

Divided into three phases:

1. Ray Cluster Verification: Submit the sbatch script, start Head/Worker nodes, test cluster functions, and store logs in results/logs.
2. vLLM Single Node Verification: Install dependencies, load models (such as LLaMA, Qwen), monitor GPU usage, and perform benchmark tests.
3. Multi-node Integration: Configure vLLM to use the Ray backend, set parallel parameters, start the inference service, and conduct end-to-end testing and tuning.

Key Practices:

• Progressive Verification: Verify each layer independently before integration to avoid debugging difficulties.
• Centralized Logging: Aggregate logs from all nodes to shared storage.
• Resource Monitoring: Real-time monitoring of GPU, network, and memory status.
• Fault Tolerance Design: Graceful degradation when nodes fail.
• Performance Benchmarking: Establish single/multi-node baselines to quantify distributed benefits.

Future Plans:

• Support more models (Mistral, Falcon, etc.).
• Optimize dynamic batching and request scheduling.
• Explore load balancing for heterogeneous GPU clusters.
• Integrate model quantization to reduce resource requirements.
• Develop an automated deployment toolchain. Distributed large language model inference is an important direction for AI infrastructure, and this combination can provide support for large-scale AI applications.