BatchBench: A Workload-Aware Benchmark Framework for Auto-Scaling Strategies in Big Data Batch Processing

BatchBench is an open auto-scaling benchmark framework designed to address the fragmentation issue in the evaluation of auto-scaling for big data batch processing. Through workload classification, parameterized generators, a five-axis evaluation system, and standardized agent interfaces, it provides a fair experimental comparison platform for rule-based, learning-based, and large-model agent-based auto-scaling strategies, promoting the establishment of common measurement standards in the field.

Auto-scaling has become a baseline capability for cloud-native big data processing, but evaluation practices are fragmented. Existing studies use different benchmarks (synthetic queries, proprietary baselines, domain-specific traces) and varying comparison conditions (baselines, workloads, cost models), making cross-paper comparisons almost impossible. This prevents researchers from judging the merits of new methods, makes it difficult for practitioners to select suitable solutions, and leaves the field without a common language and standards.

The core goal of BatchBench is to provide an equal experimental comparison platform for three types of auto-scaling strategies (rule-based, learning-based, large-model agent-based). Its key principles include: equal footing (a neutral environment without presupposing the superiority of any method) and workload awareness (covering the diversity of real scenarios through classification and parameterized generation to avoid over-optimization for a single workload).

One of BatchBench's core contributions is a six-category batch processing workload classification system (ETL, analytical queries, machine learning training, graph computing, streaming micro-batch processing, mixed workloads), based on published benchmarks and analysis of public cluster traces. Another contribution is a parameterized generator that allows adjusting parameters such as job arrival patterns, scale distribution, and resource requirements to generate diverse workloads, with similarity to real trace distributions verified via KS test and Earth Mover's Distance.

BatchBench proposes a five-axis evaluation system to comprehensively measure strategy performance: Cost Axis (computing/storage/network costs, including LLM inference costs), SLA Achievement Axis (job completion time/latency/success rate), Scaling Responsiveness Axis (scaling-up/scaling-down latency, decision frequency), Scaling Oscillation Axis (frequent switching, resource fluctuations), and Decision Interpretability Axis (transparency, log richness).

BatchBench provides a standardized agent interface that unifies the evaluation APIs for rule-based, learning-based, and large-model agents. The interface defines state observation formats (cluster/job/historical metrics), action spaces (scale up/scale down/maintain), reward signals (cost/SLA/comprehensive utility), and interaction protocols, reducing the threshold for integrating new methods and improving reproducibility.

BatchBench is currently in the framework design phase, with a reference implementation under development and planned for open-source release. Future plans include expanding workload classification (real-time inference, federated learning), integrating more real traces, developing automated tuning tools, and establishing a leaderboard. We call on the community to participate and jointly improve the framework.

BatchBench provides the field with a much-needed open benchmark framework, which is expected to end evaluation fragmentation. The implications for research include: emphasizing the importance of fair evaluation, focusing on workload diversity to improve robustness, and advocating an open and collaborative research culture. The framework will drive auto-scaling research toward maturity and practical application.