CacheOn: An Arena Platform for Large Language Model Inference Server Optimization

CacheOn is an open-source arena platform focused on performance optimization of large language model (LLM) inference servers. It provides researchers and developers with a standardized testing environment and comparison benchmarks to help identify optimal inference optimization strategies. Its core goal is to address the problem that different optimization techniques perform differently under varying hardware and model architectures, providing a unified and fair comparison platform.

With the widespread deployment of large language models (LLMs) in various application scenarios, performance optimization of inference servers has become a key factor affecting user experience and operational costs. However, different optimization techniques—whether quantization, speculative decoding, or caching strategies—often perform differently under varying hardware environments and model architectures. Researchers and engineers are in urgent need of a unified and fair platform to compare the actual effects of various optimization solutions. The CacheOn project was born to address this need; it provides a standardized arena environment where different LLM inference optimization implementations can compete fairly and be performance-compared under the same conditions.

The design philosophy of CacheOn revolves around "reproducible benchmarking", with core capabilities including:

1. Standardized Testing Environment

The project has established a unified testing framework to ensure that all optimization solutions participating in the comparison run under the same input distribution, load pattern, and hardware configuration, eliminating evaluation bias caused by inconsistent testing conditions.

2. Multi-dimensional Performance Metrics

In addition to traditional metrics such as throughput and latency, it also measures key dimensions like time-to-first-token, memory usage, and GPU utilization, providing data support for comprehensive evaluation.

3. Extensible Architecture

Adopting a modular design, it allows users to easily integrate new inference engines and optimization techniques (such as vLLM, TensorRT-LLM, or custom implementations) and conduct comparative tests through a unified interface.

The implementation of CacheOn involves multiple technical aspects: In terms of load generation, it simulates request distributions in real scenarios (varying input sequence lengths, diverse output requirements); in terms of measurement accuracy, it uses high-precision timers and controls measurement overhead to ensure data accuracy; additionally, it considers performance differences between cold start and warm cache states to help understand the performance of optimization strategies in different operation stages.

For LLM inference service providers:

• Quantify the actual benefits of different optimization techniques
• Identify the optimal configuration for specific hardware and model combinations
• Track performance improvements of new versions of inference engines
• Provide data basis for capacity planning and cost estimation

For academic researchers: It provides a reproducible experimental environment to promote standardized research in the field of LLM inference optimization.

With the rapid development of LLM inference technology, CacheOn is expected to become a community-driven benchmark center. Possible future directions include: supporting more model architectures, introducing distributed inference scenarios, and providing automated optimization suggestion functions.