Shennong: LLM Inference Performance Profiling and Tracing Analysis Tool

Shennong is an open-source CLI tool designed to address the core challenge of LLM inference performance optimization in production. It helps developers identify hidden bottlenecks across complex software stacks, enabling data-driven optimization decisions for model deployment efficiency. Its key value lies in providing end-to-end tracing and multi-granularity analysis with minimal overhead.

LLM inference optimization is complex due to three main factors:

1. Multi-layer software stack: Issues can arise in model frameworks (PyTorch/TensorFlow), inference engines (vLLM/TensorRT-LLM), hardware abstractions (CUDA/ROCm), or system services (batch scheduling/caching).
2. Multiple performance dimensions: Latency, throughput, TTFT, ITL, and resource utilization—each with varying priorities across use cases.
3. Dynamic characteristics: Variable input/output lengths, attention complexity (sequence length squared), changing memory access patterns, and dynamic batch scheduling make static analysis insufficient.

Shennong's design focuses on:

• End-to-end tracing: Captures the full lifecycle from request input to output to identify pipeline bottlenecks.
• Multi-granularity analysis: Supports request, stage, operator, and memory-level insights.
• Low-overhead collection: Uses sampling/tracing techniques to minimize impact on actual inference.
• Extensible architecture: Plugin-based system for community contributions to support new engines/hardware.

Shennong offers four core features:

1. Trace collection: Integrates with PyTorch Profiler, inference engine hooks (vLLM/TensorRT-LLM), system monitoring (GPU/PCIe metrics), and custom code markers.
2. Visualization: Timeline view (flame graphs), aggregate stats (percentiles/SD), comparison view (different configs), and hotspot analysis.
3. Bottleneck diagnosis: Detects compute (GPU utilization), memory (bandwidth), parallel (sync/wait times), and scheduling (batch overhead) bottlenecks.
4. Report generation: Interactive HTML, structured JSON, and concise text reports for various use cases.

Shennong is useful in several scenarios:

• Engine selection: Compare vLLM vs TensorRT-LLM using commands like shennong profile --engine vllm --model llama-7b --dataset eval_prompts.json and shennong compare vllm_trace.json trtllm_trace.json.
• Performance regression: Integrate into CI/CD to detect >5% performance drops with shennong compare trace_baseline.json trace_new.json --threshold 5%.
• Optimization verification: Validate effects of quantization/operator fusion via trace comparison.
• Production diagnosis: Profile real traffic with shennong profile --endpoint http://prod-llm-api:8000 --duration 60s to find root causes.
• Capacity planning: Predict hardware performance and identify bottleneck resources.

Shennong's implementation includes:

• Trace format: Uses Chrome Trace Event for interoperability, with events containing timestamps, types, thread IDs, and metadata.
• Low-overhead techniques: Async writing, sampling, zero-copy transfer, and compile-time optimization for production.
• Analysis engine: Supports streaming processing, parallel analysis, and index building for large trace files.

Future plans for Shennong include: supporting more engines/hardware, real-time monitoring, ML-assisted anomaly detection, cloud service integration, and distributed multi-GPU analysis. As LLM applications grow, Shennong will play an increasingly critical role in helping teams optimize inference performance with data-driven insights.