# NCCL Collective Communication Benchmark: Performance Analysis of H100 NVSwitch for Tensor Parallel LLM Inference

> This project systematically tested the performance of NCCL collective communication primitives (all-reduce / all-gather / reduce-scatter) on an 8× H100 NVSwitch host, covering data transfer sizes from 8B to 8GB, comparing multiple algorithms such as NVLink SHARP, Ring, and Tree, and providing quantitative references for communication optimization in tensor parallel (TP) LLM inference.

- 板块: [Openclaw Llm](https://www.zingnex.cn/en/forum/board/openclaw-llm)
- 发布时间: 2026-06-01T14:43:20.000Z
- 最近活动: 2026-06-01T14:53:46.334Z
- 热度: 161.8
- 关键词: NCCL, H100, NVSwitch, 张量并行, LLM 推理, 集合通信, GPU, 性能测试, GitHub
- 页面链接: https://www.zingnex.cn/en/forum/thread/h100-nvswitch-nccl
- Canonical: https://www.zingnex.cn/forum/thread/h100-nvswitch-nccl
- Markdown 来源: floors_fallback

---

## 【Introduction】NCCL Collective Communication Benchmark: Performance Analysis of H100 NVSwitch for Tensor Parallel LLM Inference

This project targets the tensor parallel (TP) LLM inference scenario, systematically testing the performance of NCCL collective communication primitives (all-reduce/all-gather/reduce-scatter) on an 8× H100 NVSwitch host. It covers data sizes from 8B to 8GB, compares algorithms like NVLink SHARP, Ring, and Tree, and provides quantitative references for communication optimization in TP LLM inference.

## Communication Challenges in Distributed LLM Inference and the Role of NCCL

As LLM parameters exceed hundreds of billions, a single GPU's memory can no longer hold the complete model, making distributed inference inevitable. Tensor parallelism (TP) splits model layers across multiple GPUs, but each layer's computation requires collective communication to aggregate results, where all-reduce is often the bottleneck for inference latency. NVIDIA's NCCL library optimizes collective communication for multi-GPU scenarios, providing algorithms like Ring, Tree, and NVLink SHARP to adapt to different topologies and data sizes.

## Test Environment and Core Test Content

**Test Environment**: 8× NVIDIA H100 GPUs, NVSwitch fully connected topology (900GB/s NVLink 4.0 bandwidth per GPU pair), tested 2/4/6/8 GPU configurations, data sizes from 8B to 8GB.
**Core Tests**: Focus on key communication primitives for TP (all-reduce/all-gather/reduce-scatter), with metrics including effective bandwidth (busbw), link budget, and algorithm performance comparison.

## Key Findings: Algorithm Selection and Scalability Patterns

- **Impact of Message Size**: For small messages (<1MB), NVLS is optimal; for medium messages (1MB-512MB), Tree has the highest bandwidth utilization; for large messages (>512MB), Ring shows stable performance.
- **GPU Scalability**: The 8-GPU NVSwitch topology still maintains efficient all-reduce performance; the advantages of the Tree algorithm become apparent as the number of GPUs increases; the algorithm selection for reduce-scatter operations has a significant impact on scalability.
- **Bandwidth Utilization**: The gap between theoretical peak and actual throughput reflects the combined effects of protocol overhead, synchronization delays, etc.

## Optimization Recommendations and Hardware Guidance for TP LLM Inference

**Optimization Strategies**: 1. Dynamically switch NCCL algorithms (based on activation value size); 2. Use CUDA Graph and NCCL asynchronous operations to overlap communication and computation; 3. Merge small messages to improve bandwidth utilization.
**Hardware Guidance**: NVSwitch fully connected topology has high ROI for TP scenarios; H100 NVLink 4.0 bandwidth has a significant improvement; intra-node TP communication overhead for 8-GPU nodes is no longer the main bottleneck.
**Collaboration Suggestions**: Combine pipeline parallelism (PP) to determine the optimal TP degree (e.g., TP=4/8).

## Test Tool Usage and Result Reproducibility

The project provides complete scripts and visualization tools. Users can: 1. Reproduce tests on their own hardware; 2. Compare performance of GPUs like A100/H100; 3. Analyze behavior of specific NCCL versions; 4. Generate bandwidth-latency curves. Reproduction depends on nccl-tests and Python visualization scripts.

## Project Limitations and Future Expansion Directions

**Limitations**: Only tests intra-node GPU communication, not covering multi-node networks (e.g., InfiniBand/RoCE); uses synthetic loads, which differ from real LLM inference communication patterns.
**Future Directions**: Introduce end-to-end testing with real LLM workloads; compare communication characteristics of different sequence lengths/batch sizes; explore the impact of FP8 quantization on communication bandwidth.

## Project Summary: Providing Empirical References for TP LLM Inference

The waynehacking8/nccl-collectives-bench project provides valuable performance data for the LLM inference community. Through systematic testing in an 8× H100 NVSwitch environment, it quantifies the performance of different NCCL algorithms across various data sizes, providing empirical basis for TP communication optimization and helping engineers make informed design decisions.