NCCL Collective Communication Benchmark: Performance Analysis of H100 NVSwitch for Tensor Parallel LLM Inference

This project targets the tensor parallel (TP) LLM inference scenario, systematically testing the performance of NCCL collective communication primitives (all-reduce/all-gather/reduce-scatter) on an 8× H100 NVSwitch host. It covers data sizes from 8B to 8GB, compares algorithms like NVLink SHARP, Ring, and Tree, and provides quantitative references for communication optimization in TP LLM inference.

As LLM parameters exceed hundreds of billions, a single GPU's memory can no longer hold the complete model, making distributed inference inevitable. Tensor parallelism (TP) splits model layers across multiple GPUs, but each layer's computation requires collective communication to aggregate results, where all-reduce is often the bottleneck for inference latency. NVIDIA's NCCL library optimizes collective communication for multi-GPU scenarios, providing algorithms like Ring, Tree, and NVLink SHARP to adapt to different topologies and data sizes.

Test Environment: 8× NVIDIA H100 GPUs, NVSwitch fully connected topology (900GB/s NVLink 4.0 bandwidth per GPU pair), tested 2/4/6/8 GPU configurations, data sizes from 8B to 8GB. Core Tests: Focus on key communication primitives for TP (all-reduce/all-gather/reduce-scatter), with metrics including effective bandwidth (busbw), link budget, and algorithm performance comparison.

• Impact of Message Size: For small messages (<1MB), NVLS is optimal; for medium messages (1MB-512MB), Tree has the highest bandwidth utilization; for large messages (>512MB), Ring shows stable performance.
• GPU Scalability: The 8-GPU NVSwitch topology still maintains efficient all-reduce performance; the advantages of the Tree algorithm become apparent as the number of GPUs increases; the algorithm selection for reduce-scatter operations has a significant impact on scalability.
• Bandwidth Utilization: The gap between theoretical peak and actual throughput reflects the combined effects of protocol overhead, synchronization delays, etc.

Optimization Strategies: 1. Dynamically switch NCCL algorithms (based on activation value size); 2. Use CUDA Graph and NCCL asynchronous operations to overlap communication and computation; 3. Merge small messages to improve bandwidth utilization. Hardware Guidance: NVSwitch fully connected topology has high ROI for TP scenarios; H100 NVLink 4.0 bandwidth has a significant improvement; intra-node TP communication overhead for 8-GPU nodes is no longer the main bottleneck. Collaboration Suggestions: Combine pipeline parallelism (PP) to determine the optimal TP degree (e.g., TP=4/8).

The project provides complete scripts and visualization tools. Users can: 1. Reproduce tests on their own hardware; 2. Compare performance of GPUs like A100/H100; 3. Analyze behavior of specific NCCL versions; 4. Generate bandwidth-latency curves. Reproduction depends on nccl-tests and Python visualization scripts.

Limitations: Only tests intra-node GPU communication, not covering multi-node networks (e.g., InfiniBand/RoCE); uses synthetic loads, which differ from real LLM inference communication patterns. Future Directions: Introduce end-to-end testing with real LLM workloads; compare communication characteristics of different sequence lengths/batch sizes; explore the impact of FP8 quantization on communication bandwidth.

The waynehacking8/nccl-collectives-bench project provides valuable performance data for the LLM inference community. Through systematic testing in an 8× H100 NVSwitch environment, it quantifies the performance of different NCCL algorithms across various data sizes, providing empirical basis for TP communication optimization and helping engineers make informed design decisions.