Hybrid Batching Isn't Always the Optimal Solution: EB+ Dynamic Scheduling Achieves 41.9% Throughput Improvement on Bandwidth-Constrained GPUs

Source: Paper published on arXiv on May 30, 2026: Threshold-Based Exclusive Batching for LLM Inference (Link: http://arxiv.org/abs/2606.00516v1) Core Insight: Hybrid Batching (MB) isn't a one-size-fits-all solution for LLM inference; its performance is significantly impacted by GPU memory bandwidth. On bandwidth-constrained GPUs like the RTX PRO 6000, prefill-decode interference leads to decreased MB efficiency. The proposed Threshold-Based Exclusive Batching (EB) and dynamic hybrid scheduler EB+ can achieve up to 41.9% throughput improvement. Subsequent floors will cover background, core findings, methods, performance evaluation, deployment implications, limitations, and future directions.

LLM inference efficiency is a core challenge for AI infrastructure. Hybrid Batching (MB) is the current mainstream strategy—maximizing resource utilization by interleaving prefill (compute-intensive) and decode (bandwidth-intensive) phases. However, research finds: MB's prefill-decode interference increases the marginal cost per step, even exceeding the cost of pure decoding. This problem is particularly prominent in bandwidth-constrained scenarios.

Experimental comparison between high-bandwidth GPU (H200, 4.8TB/s) and bandwidth-constrained GPU (RTX PRO6000, 1.792TB/s):

• On H200, the threshold where MB is worse than pure decoding is 80% (decode token ratio);
• On RTX PRO6000, the threshold is only 20%. Reason: Decoding is a bandwidth-intensive task. On bandwidth-constrained devices, the prefill phase of MB consumes the bandwidth required for decoding, leading to a sharp drop in efficiency.

1. Exclusive Batching (EB): Strictly separate prefill and decode phases to avoid interference, but requires balancing resource utilization.
2. Closed-form Conditions: Derive mathematical conditions for performance crossover between EB and MB, considering bandwidth, model size, and workload distribution.
3. EB+ Dynamic Scheduler: Monitor GPU bandwidth and workload online, switch between EB/MB strategies in real time; under non-stationary traffic, it achieves a 36.4% throughput improvement compared to fixed MB.

• Bandwidth-Constrained GPUs: EB achieves a 41.9% throughput improvement on the RTX PRO6000;
• High-Bandwidth GPUs: MB still maintains an advantage on the H200;
• Adaptive EB+: Automatically adapts to bandwidth scenarios without manual parameter tuning, always approaching optimal performance.

• Hardware Selection: Use MB for high-bandwidth GPUs (H200/H100); use EB/EB+ for bandwidth-constrained GPUs (RTX series);
• System Optimization: Analyze GPU bandwidth utilization, monitor prefill-decode interference, implement EB+ dynamic switching;
• Cost-Effectiveness: EB+ only requires adjusting scheduling logic, no model modifications, and has a significant ROI.

Limitations:

• Experiments only cover H200 and RTX PRO6000; validation on other GPUs is needed;
• Ultra-large models (100B+ parameters) were not fully tested;
• Strategies need adjustment for multi-GPU parallel scenarios. Future Directions:
• Predictive scheduling (based on request features);
• Multi-objective optimization (throughput + latency + fairness);
• Heterogeneous hardware cluster optimization.