LLM Stress Tester: Comprehensive Analysis of an Open-Source Local Stress Testing Tool

This article will comprehensively analyze LLM Stress Tester—an open-source local load testing tool based on Streamlit. Designed specifically for OpenAI-compatible endpoints, it takes "local-first, data security" as its core concept, supporting progressive stress testing, multi-model traffic distribution, and seven real-scenario benchmark suites. It helps developers and operation teams accurately evaluate the performance and stability of LLM inference services.

With the popularization of LLMs in production environments, traditional binary "available/unavailable" tests cannot reveal system details under real loads. LLM Stress Tester emerged to solve this problem: it can simulate real usage scenarios in a fully offline environment, providing a professional local load testing solution for OpenAI-compatible endpoints.

The core features of this tool include:

1. Endpoint Compatibility: Supports any inference endpoint following the OpenAI API specification; can batch import API keys (for private endpoints) or skip authentication (for public endpoints);
2. Progressive Stress Testing: Simulates scenarios with gradually increasing traffic through configurations of initial rate, maximum rate, growth multiple, and duration to identify performance inflection points;
3. Flexible Rate Units: Supports switching between RPS/RPM, with internal automatic conversion to ensure accuracy;
4. Multi-Model Traffic Distribution: Allows setting traffic weights for multiple models (e.g., 70% Model A +30% Model B), or sending requests to all models simultaneously for horizontal comparison.

The tool has seven optimized prompt collections built-in:

• Code Generation: Focuses on algorithm implementation and programming problem-solving;
• Mathematical Reasoning: Includes logic tests like word problems and probability calculations;
• Knowledge Q&A: Covers knowledge reserve evaluation in fields like science and history;
• Instruction Following: Tests the ability to execute formatted requirements and multi-step instructions;
• Multi-Turn Dialogue: Checks context coherence;
• Long Text Processing: Targets document summarization and long text analysis;
• Text Processing: Covers tasks like editing, rewriting, and classification.

During testing, the interface refreshes key metrics every 2 seconds (phase progress, rate comparison, request counter); after completion, a results dashboard is generated, including latency percentiles (P50/P95/P99), error rate tracking, phase/model-wise statistics, and dual-axis rate comparison charts. Additionally, it supports export to Excel (configuration, raw data, phase/model summaries, error details) and PDF (integration of core charts), with rate columns automatically matching the test unit.

Deployment methods are flexible:

• Developers: After cloning the repository, execute pip install -e ".", then run streamlit run src/llm_stress_tester/app.py to start the service;
• End Users: Download the pre-built binary file for the corresponding platform (Windows/macOS/Linux). macOS users need to first lift the Gatekeeper restriction, then run directly to launch the browser.

LLM Stress Tester is suitable for multiple scenarios:

• Model service providers: Verify the performance of new inference endpoints;
• Application developers: Test response latency of different models to optimize selection;
• Operation and maintenance teams: Determine the upper limit of system capacity through progressive testing;
• Researchers: Use standardized benchmark suites to compare model capabilities.

LLM Stress Tester, with its local-first design, comprehensive feature coverage, and user-friendly experience, fills the gap in the open-source community for stress testing of LLM inference services. Whether for individual developers or enterprise teams, this tool can help gain deep insights into model service performance and provide data support for the stable operation of production environments.