llm-d-diagnostics: A Diagnostic Tool for Distributed Inference of Large Language Models

This article introduces the open-source toolkit llm-d-diagnostics, designed specifically for distributed inference scenarios of large language models. It helps developers diagnose and optimize performance bottlenecks and system issues, covering core capabilities such as monitoring, bottleneck localization, and report generation, and is suitable for various deployment modes.

As the scale of large language models grows, single GPU/server can hardly meet inference demands, making distributed inference the mainstream. However, distributed systems introduce challenges like network latency, uneven load, difficulty in fault localization, and resource contention, which call for professional diagnostic tools.

llm-d-diagnostics is an open-source diagnostic toolkit designed for the llm-d distributed inference framework, providing: 1. Real-time monitoring of performance metrics across nodes; 2. Localization of issues like communication latency and computation bottlenecks; 3. Generation of structured diagnostic reports; 4. Adaptation to deployment scenarios such as single-machine multi-card, multi-machine multi-card, and cloud.

1. Real-time performance monitoring: Tracks fine-grained metrics like inference latency, throughput, memory usage, communication overhead, and queue depth. The lightweight agent collection has minimal impact on performance; 2. Automatic bottleneck diagnosis: Detects communication bottlenecks (e.g., excessive activation value transmission), uneven computation load (pipeline bubbles), and memory pressure warnings; 3. Visualization and reporting: Output formats include console views, Prometheus time-series data, JSON reports, and flame graphs.

1. Low-intrusiveness design: Bypass architecture that intervenes in the inference process via hooks without modifying core code, with minimal impact, easy integration, and dynamic start/stop; 2. Cross-platform compatibility: Supports NVIDIA/CUDA, AMD/ROCm GPUs, NCCL/Gloo/MPI communication backends, and deployment on bare metal, Docker, and Kubernetes; 3. Extensible metrics system: Plugin-based design supporting custom metrics, adjustment of sampling frequency, and configuration of alarm thresholds.

Scenario 1: Benchmark testing before new model launch—simulate load, identify performance inflection points, verify resource configuration, and establish baselines; Scenario 2: Production fault troubleshooting—real-time monitoring of anomalies, compare metric differences, locate root causes, and generate reports; Scenario 3: Architecture optimization verification—compare data before and after modifications, quantify optimization effects.

Feature	llm-d-diagnostics	General Profiler	Cloud Vendor Monitoring
LLM-specific Optimization	✅ Optimized for Transformer architecture	❌ General design	⚠️ Partial support
Distributed Awareness	✅ Natively supports multi-node	⚠️ Requires additional configuration	⚠️ Depends on infrastructure
Deployment Flexibility	✅ Lightweight, runs anywhere	✅ Runs locally	❌ Tied to cloud platform
Open Source & Free	✅ Fully open source	Partially open source	❌ Commercial service

Future plans: Automatic tuning suggestions, historical trend analysis, multi-framework support (vLLM/TensorRT-LLM), and integrated test suites. Summary: llm-d-diagnostics fills the gap in diagnostic tools for LLM distributed inference, which is crucial for ensuring service stability and optimizing resource utilization. It is recommended that teams deploying distributed LLM services include it in their tech stack.