InferenceX: Open-Source Continuous Inference Benchmarking Platform — Real-Time Tracking of LLM Inference Performance Evolution

InferenceX, launched by SemiAnalysis, is an open-source automated benchmarking platform designed to address the problem that traditional fixed-point benchmarking becomes outdated quickly. It continuously tracks the actual performance of mainstream inference frameworks on the latest hardware (such as NVIDIA Blackwell, AMD MI355X, etc.), provides transparent and reproducible data to support AI infrastructure decision-making, and its core value lies in capturing inference performance leaps in near real-time, breaking information lag.

LLM inference performance improvement relies on hardware innovation (NVIDIA and AMD release new GPUs every year) and software optimization (SGLang, vLLM, etc., are updated on a daily basis). The results of traditional static benchmarking are prone to becoming invalid due to software updates, leading to misallocation of enterprise resources. InferenceX provides continuously updated performance metrics to solve this dilemma.

InferenceX covers:

• Inference frameworks: SGLang, vLLM, TensorRT-LLM
• Hardware: NVIDIA GB200 NVL72/B200/GB300 NVL72/H100, AMD MI355X (TPU v6e/v7, etc., will be added soon)
• Models: Qwen3.5, DeepSeek series, etc., which are close to production environments.

InferenceX evaluates from multiple dimensions:

• Tokens per Second: Basic metric for generation speed
• Throughput per Dollar: Performance-cost ratio, assisting hardware selection
• Tokens per Megawatt: Energy efficiency
• Latency Distribution: Tail metrics such as P99 latency to ensure service stability.

InferenceX has gained industry recognition:

• Peter Hoeschele (OpenAI): Provides a real-time performance landscape
• Tri Dao (Together AI): Demonstrates the actual effects of software optimization
• Simon Mo (vLLM): Supports publicly reproducible benchmarks The platform uses the Apache 2.0 license. Only results from the official repository are authoritative, and data is traceable. Users can view real-time data through the open-source dashboard. Manufacturers such as NVIDIA and AMD, as well as cloud service providers, provide resource support.

Value:

• Architects: Evaluate the cost-effectiveness of hardware-software combinations
• ML engineers: Reference optimal inference configurations
• Researchers: Standardized evaluation platform
• Cloud service providers: Showcase performance advantages Future: Expand hardware coverage (such as TPU), introduce long-context and multi-modal inference, and keep up with the latest versions of software frameworks.

Through continuous testing, open-source transparency, and ecological cooperation, InferenceX has become a trusted performance reference for the AI community. Whether enterprises are planning infrastructure or researchers are understanding technological progress, they can gain insights. With more hardware and frameworks joining, it is expected to become the standard measurement in the field of LLM inference.