model-speed-test: A Comprehensive Evaluation Tool for LLMs with OpenAI-Compatible APIs

Core Points

model-speed-test is an open-source LLM benchmarking tool that supports comprehensive evaluation of speed, visual understanding, tool calling, and reasoning capabilities for any OpenAI-compatible API, helping developers objectively compare the performance of different models and providers.

Original Author & Source

• Original Author/Maintainer: 1chenmm
• Source Platform: GitHub
• Original Link: https://github.com/1chenmm/model-speed-test
• Release Time/Update Time: 2026-06-13T15:10:11Z

Project Overview

model-speed-test focuses on LLM performance evaluation, with the design goal of providing objective and reproducible benchmark results. Unlike tools that only focus on generation speed, it uses a multi-dimensional evaluation system, measuring model capabilities from four key aspects: inference speed, visual understanding, tool calling, and logical reasoning—closer to real-world application scenarios.

Core Features

The project's biggest feature is supporting any OpenAI-compatible API endpoint, including OpenAI services, third-party providers like Azure, and locally deployed inference servers such as vLLM and TGI. It allows horizontal comparison using the same set of standards, making it highly versatile.

Speed Test

Using tokens per second (TPS) as the metric, it measures the text generation throughput of the model. It supports configuring different concurrency levels and input/output lengths to simulate real-scenario load patterns.

Visual Understanding Evaluation

Evaluates the model's accuracy in understanding image content and response speed. By sending image-containing inputs, it checks the accuracy and completeness of descriptions, testing the quality of the visual encoder and multi-modal fusion efficiency.

Tool Calling Test

Simulates real scenarios and evaluates three aspects: calling accuracy (correctly identifying tools and generating formatted parameters), parameter extraction precision (extracting structured parameters from natural language), and calling timing judgment (only calling tools when necessary).

Reasoning Capability Evaluation

Through math calculation, logical reasoning, and common sense judgment questions, it distinguishes between memory models and reasoning models, helping developers assess whether a model is suitable for specific scenarios (e.g., math tutoring).

Applicable Scenarios

• Technical decision-makers: Data-driven selection to avoid marketing misinformation;
• Operations engineers: Regular benchmarking to detect service degradation in time;
• Researchers: Standardized results for paper citations and peer comparisons.

Practical Recommendations

• Establish a fixed test baseline: Test the main model weekly with the same parameters and record TPS trends;
• Customize test cases based on business scenarios: Add business-related samples to get targeted evaluation results.

Architecture Design

Adopts a modular design where the four test dimensions are relatively independent. They can be enabled/disabled on demand, lowering the barrier to use and facilitating the expansion of new dimensions.

Technical Implementation

Developed based on Python with clear dependency management and simple deployment. Test results are output in a structured format, making it easy to integrate into CI/CD pipelines or data visualization platforms.

Summary

The emergence of model-speed-test reflects the shift of the LLM ecosystem from 'wild growth' to 'rational evaluation'. It provides an objective performance benchmark and advocates a data-driven selection culture.

Industry Significance

It is recommended that developers include it in their technical research process, obtain first-hand data through actual testing instead of relying on vendor promotions or community reputation, maintain an objective understanding of model capabilities, and make correct technical decisions.