tps.sh: A Performance Benchmarking Tool for Local and Cloud Large Language Models

tps.sh is an open-source tool dedicated to performance testing of large language models (LLMs). Its core goal is to compare the tokens per second (TPS) performance of local Ollama models and the cloud-based Claude API through 147 tests, helping users make optimal deployment decisions on Apple Silicon devices. This tool eliminates technical barriers to performance evaluation by encapsulating complex testing logic into a concise command-line interface, supporting cross-platform operation, and providing data-driven references for LLM deployment to developers and users.

In practical LLM applications, performance is a key consideration, but traditional performance evaluation requires complex scripts and technical background. The design goal of tps.sh is clear: to provide a simple and intuitive tool that allows users to easily compare the performance of different LLMs. The tool is specifically optimized for the Apple Silicon architecture, leveraging its neural engine and unified memory architecture; it also supports the Windows system, expanding its scope of application and lowering the threshold for performance evaluation.

tps.sh has built-in 147 test cases covering various scenarios (including text generation, code completion, logical reasoning, etc.) and uses 21 typical sample prompt inputs. Its core function is to perform parallel tests on local Ollama models (on Apple Silicon devices) and cloud-based Claude API services. Evaluation dimensions include processing speed (TPS), generation quality, cloud cost analysis, and resource usage (CPU/memory/GPU), enabling multi-dimensional performance comparison.

tps.sh is optimized for Apple Silicon: it automatically uses the neural engine to accelerate inference and leverages the unified memory architecture to reduce data transfer overhead. The Windows version supports testing of various local LLM runtimes or cloud APIs, with system requirements including Windows 10+, 8GB RAM, and a 2GHz processor. The tool supports testing of 7 models, with flexible and extensible configuration mechanisms that allow specifying model paths, API parameters, etc., via configuration files.

Installation: Download precompiled binaries or installation packages from GitHub; Windows users can choose between .exe or .zip files. Running: Enter tps.sh in the command line, which automatically loads the configuration and executes 147 tests for 7 models, displaying progress in real time. Results: Generate a detailed report including overall performance ranking, task type analysis, cost-benefit tradeoff, and raw data tables, helping users quickly identify performance bottlenecks and deployment pros and cons.

The application scenarios of tps.sh include: hardware selection decisions (assessing whether existing devices meet local deployment requirements), model selection references (objectively comparing the performance, quality, and cost of open-source/commercial models), deployment mode evaluation (scenario adaptability of local vs. cloud), and performance monitoring and optimization (establishing baselines and regularly detecting performance changes).

As an open-source project, tps.sh has an active GitHub community where users can submit issues, share results, and contribute code. Future plans include adding support for more models and platforms, expanding test coverage, and introducing advanced performance analysis features. The tool fills the gap in LLM performance evaluation, promotes ecosystem optimization, and is expected to become one of the standard tools for LLM performance evaluation.