Server-Inspector: An All-Round Hardware Detection and Capacity Evaluation Tool for LLM Inference Servers

Server-Inspector is a general-purpose hardware detection and capacity evaluation tool designed specifically for LLM inference scenarios. It supports in-depth hardware profiling analysis and performance evaluation in multi-accelerator environments, and helps solve key hardware capacity evaluation problems in deployment decisions through a profile-driven approach.

With the increasing popularity of Large Language Model (LLM) inference services today, accurately evaluating the hardware capacity and inference capability of servers has become a core challenge in deployment decisions. Currently, there is a lack of dedicated hardware evaluation tools for LLM inference scenarios.

1. Multi-Accelerator Hardware Detection

Automatically identifies various AI accelerators such as NVIDIA GPU, AMD GPU, Intel Gaudi, and extracts key information like memory capacity, number of computing units, and bandwidth specifications

2. Profile-Driven Evaluation Framework

Users can define evaluation profiles to simulate specific model loads, and quantify the server's inference throughput and latency performance through standardized tests

3. Capacity Prediction and Planning

Generates capacity reports, providing recommendations for the number of concurrent requests, maximum loadable model size, and optimal configuration under latency constraints

Hardware Information Collection

Obtains hardware data through low-level system calls and vendor SDKs, such as using the NVML library to read real-time information like memory, temperature, and power consumption of NVIDIA GPUs

Inference Load Simulation

Built-in multiple inference load simulators, controlling parameters like batch size and sequence length to simulate the LLM forward propagation process

Performance Indicator Calculation

Processes raw data to generate quantitative indicators such as tokens/second, latency percentiles, and memory utilization, providing a basis for capacity planning

Data Center Capacity Planning

Evaluates whether existing hardware meets LLM service requirements and guides equipment procurement decisions

Heterogeneous Cluster Management

Provides unified hardware profiling capabilities to assist scheduling systems in intelligently assigning tasks

Performance Benchmarking

Serves as a standardized suite to compare LLM inference performance of different hardware configurations, supporting hardware selection

Server-Inspector fills the gap in hardware evaluation tools for LLM inference scenarios. As model sizes grow and hardware diversification trends intensify, its importance will continue to increase. Future plans include expanding support for more accelerator types and integrating more complex load models to simulate real production environments.