Abacus：OpenAI兼容推理API的轻量级基准测试工具

Abacus is a command-line tool designed to benchmark OpenAI-compatible inference APIs. It helps developers evaluate and compare the performance (latency, throughput, etc.) and response quality of different API endpoints, supporting service selection, performance monitoring, and capacity planning. Key features include multi-dimensional testing, multi-endpoint comparison, and a developer-friendly CLI interface.

With the popularity of LLM services, developers face diverse API choices (OpenAI official, Together AI, Groq, self-hosted vLLM/TGI). However, actual performance varies by factors like latency (TTFT, full response time), throughput (TPS/RPS), availability, cost, and output quality. A systematic benchmarking tool is essential for informed decisions—this is where Abacus comes in.

Abacus supports multiple test dimensions:

1. Latency: TTFT (time to first token), full response time, inter-token delay.
2. Throughput: TPS (tokens per second), RPS (requests per second), concurrency testing.
3. Load: Batch requests, success/error rates, response time distribution (P50/P95/P99), bottleneck identification.
4. Multi-endpoint comparison: Test multiple providers/models to support load balancing or service selection.

Abacus has three key technical features:

• OpenAI Compatibility: Follows OpenAI API format (uses /v1/chat/completions), supports any compatible endpoint (OpenAI, Azure OpenAI, open-source托管 services) with custom base URL and API key.
• CLI Interface: Simple commands for testing (e.g., abacus benchmark --endpoint ...), supports config files, concurrency settings, and structured output (JSON).
• Lightweight: Minimal dependencies, easy installation, suitable for CI/CD integration.

Abacus applies to several scenarios:

1. Service Selection: Compare latency/throughput/cost of different APIs to choose the best fit.
2. Performance Monitoring: Regularly test APIs to detect performance degradation or trigger alerts.
3. Capacity Planning: Determine optimal concurrency and resource needs based on test results.
4. Regression Testing: Verify performance after service upgrades or provider switches.

Abacus differs from other tools:

• vs curl/httpie: Automates performance metrics collection, statistical analysis, and batch testing (not manual).
• vs k6/Apache Bench: Focuses on LLM-specific metrics (token-level, streaming response) instead of generic API testing.
• vs lm-evaluation-harness: Lighter, focuses on API performance (not model capability) with simpler configuration.

Design Philosophy:

1. Single Responsibility: Only tests OpenAI-compatible API performance.
2. Embrace Standards: Uses OpenAI API format for wide compatibility.
3. Developer-Friendly: CLI, minimal dependencies, clear output.

Potential Extensions:

• Output quality assessment (similarity to reference, task accuracy).
• Continuous monitoring (trend analysis, anomaly alerts).
• Visual reports (HTML charts, historical comparisons).
• Advanced config management (YAML templates, multi-environment support).

Usage Suggestions:

1. Establish Baselines: Test current APIs to set performance thresholds.
2. Control Variables: Use same prompts/parameters for fair comparisons.
3. Simulate Real Scenarios: Test representative prompt lengths and concurrency.
4. Regular Retesting: Track performance trends over time.

Summary: Abacus is a practical tool for LLM API benchmarking. It helps developers make informed decisions in a diverse API ecosystem, with a focus on simplicity and utility. As LLM applications grow, such tools will become increasingly important for technical decision-making.