OpenVINO GPU Inference Performance Evaluation Tool: ov-impact-bench - Real-World Testing of Intel GPU LLM Inference Performance

ov-impact-bench is a tool specifically designed to measure the inference performance of large language models (LLMs) using OpenVINO on Intel GPUs. It can quantify the real performance differences between GPU and CPU fallback, covering key metrics such as latency, energy consumption, and throughput.

With the widespread adoption of large language models (LLMs) across various application scenarios, optimizing inference performance has become a key challenge. Intel's OpenVINO toolkit provides robust support for deploying AI models on Intel hardware, but in practical applications, developers often face a core question: How big is the performance difference between GPU inference and CPU fallback? The ov-impact-bench project developed by pjordanandrsn was created to address this pain point. It aims to provide an accurate, repeatable benchmarking tool specifically for measuring the real-world performance of OpenVINO when running LLM inference on Intel GPUs.

The core value of ov-impact-bench lies in its comprehensive performance measurement capabilities. Instead of focusing solely on traditional throughput metrics, the project conducts an in-depth analysis of inference performance from multiple dimensions:

The tool can accurately measure the full latency from input submission to output generation, including preprocessing, model inference, and postprocessing stages. This is crucial for real-time interactive applications (such as chatbots), as user experience is directly affected by response speed.

In addition to speed metrics, ov-impact-bench also focuses on energy efficiency. In edge device and data center scenarios, performance per watt is a key factor in evaluating the economic viability of a solution. The tool records energy consumption data during inference via Intel GPU's power monitoring interface.

For batch processing scenarios, the tool supports throughput testing with multiple concurrent requests, helping developers understand the system's performance under high load and the maximum utilization efficiency of GPU resources.

The project's unique feature is its ability to compare performance differences between native GPU inference and CPU fallback (automatic switch to CPU when GPU resources are insufficient or unsupported operations are encountered). This comparison is crucial for understanding OpenVINO's heterogeneous execution strategy.

ov-impact-bench is built on OpenVINO's Python API, fully leveraging the optimized support for LLMs in OpenVINO version 2024.x. The project has a clear code structure, mainly including the following components:

• Benchmark Engine: Coordinates the testing process, manages model loading, input preparation, and result collection
• Performance Analyzer: Integrates Intel's power monitoring and performance counters to capture fine-grained performance data
• Report Generator: Converts raw test data into structured JSON reports and visual charts
• Configuration Manager: Supports flexible configuration of test parameters (e.g., model path, input sequence length, batch size) via YAML files

The project also validates the optimizations in openvinotoolkit/openvino#35712, ensuring that test results reflect the latest performance improvements of OpenVINO.