AI-SSD Benchmark: Analysis of the Large Language Model Inference Performance Evaluation Tool v2.1

This article introduces the AI-SSD Benchmark version 2.1, a tool specifically designed to evaluate the inference performance of large language models (LLMs) on SSD storage devices. It fills the gap in traditional evaluation tools' assessment of storage subsystems, helping developers quantify the impact of SSD performance on LLM inference, guide SSD selection, optimize inference system architecture, and verify deployment configurations to improve model deployment efficiency.

Optimizing the inference performance of large language models is a core issue in AI infrastructure. However, the growth in model size has led to prominent data I/O bottlenecks—especially the storage access efficiency of weights and KV caches directly affects latency and throughput. Traditional tools focus on GPU computing power and have weak assessment of storage subsystems. Different SSDs vary significantly in random read, sequential read, and latency. Thus, the AI-SSD Benchmark was developed to address the problem of SSD performance evaluation in LLM inference scenarios.

The core features of version 2.1 include:

1. Real workload simulation: Based on real LLM inference I/O characteristics, supporting model loading (multiple formats), KV cache access during inference (random features), and concurrent access scenarios;
2. Multi-dimensional performance metrics: Covering latency (loading time, first token latency, etc.), throughput (sequential bandwidth, random IOPS, etc.), and resource utilization (SSD queue depth, CPU I/O wait time, etc.);
3. Flexible test configuration: Adjustable model parameters (size, quantization precision, etc.), load parameters (concurrency count, generation length, etc.), and storage parameters (read-ahead strategy, etc.);
4. Comparison and reporting functions: Supporting multi-device/version comparison and trend analysis, outputting JSON, chart PDFs, and HTML reports.

The technical implementation uses a modular architecture:

• Workload generator: Generates request sequences by referencing I/O patterns of mainstream inference engines;
• I/O execution engine: Based on asynchronous I/O and multi-threading, supporting Direct I/O;
• Performance sampler: Collects time-series data with microsecond-level precision;
• Analysis engine: Statistically analyzes raw data, calculates metrics, and identifies anomalies. Key optimizations include zero-copy technology, intelligent read-ahead strategy, NUMA awareness, etc.

Usage scenarios and best practices:

1. SSD selection evaluation: Compare the performance of candidate products under different model sizes and concurrent loads;
2. Performance bottleneck localization: Isolate the responsibility of the storage subsystem to determine if the bottleneck lies in storage;
3. Configuration optimization verification: Verify the effects of changes to file system parameters, I/O schedulers, etc.;
4. Capacity planning: Test the performance of different model configurations to develop storage expansion plans.

AI-SSD Benchmark is an active open-source project. The improvements in v2.1 come from community contributions. Contributions such as adding support for new model formats, expanding storage medium testing, and optimizing report visualization are welcome. Future directions include multi-modal support, distributed testing, cloud-native integration, and AI-assisted analysis.

AI-SSD Benchmark v2.1 provides a professional storage evaluation tool for optimizing LLM inference performance. In today's complex AI infrastructure, such evaluation tools tailored to AI workloads will play an important role in helping large language models achieve optimal performance on various hardware platforms.