SiDP: A New Memory-Efficient Data Parallelism Paradigm for Offline Large Model Inference

SiDP is a new memory-efficient data parallelism paradigm for offline large model inference. Key points are as follows:

• Problem Solved: The conflict in offline inference scenarios between data parallelism (DP) where weight replication occupies VRAM, and model parallelism (MP) where synchronization erodes flexibility
• Core Idea: Treat model weights as bandwidth-supported shared resources, with distributed pooling management within data parallel groups
• Dual Execution Modes: Supports dynamic switching between Weight-as-a-Service (WaS) and Compute-as-a-Service (CaS)
• Performance Improvements: 1.8x increase in KV cache capacity and 1.5x improvement in end-to-end throughput on NVIDIA H20/H200/B200

Original Source: arXiv, May 27, 2026, Link: http://arxiv.org/abs/2605.28095v1

With the widespread application of large language models (LLMs), inference workloads are shifting to throughput-prioritized offline scenarios that require large batch sizes. Existing solutions have structural conflicts:

• Data Parallelism (DP): Scales throughput but each GPU replicates full weights, leaving limited VRAM space for KV cache and restricting batch size
• Model Parallelism (MP): Reduces per-device weight usage but requires fine-grained synchronization, eroding DP independence and scheduling flexibility

This tension makes it difficult for offline inference to balance high throughput and VRAM efficiency

The core idea of SiDP (Memory-Efficient Data Parallelism) is: within data parallel groups, model weights are treated as bandwidth-supported shared resources rather than private data that each GPU must replicate.

Specific Architecture: Instead of storing the full model on each GPU, weights are organized into a distributed pool—each layer is owned by a single GPU, and other replicas access it on demand. This breaks the weight replication limit of traditional DP and frees up a large amount of VRAM for KV cache.

SiDP designs two complementary execution modes that dynamically switch based on real-time batch size:

Weight-as-a-Service (WaS) Mode

In large batch scenarios, remote weights are streamed to a local small cache via NVLink. The high bandwidth overhead is masked by computation, and only the weights of the current computation layer are cached.

Compute-as-a-Service (CaS) Mode

In small batch tail scenarios, activations are sent to the GPU that owns the corresponding layer weights for computation. Communicating activations instead of weights is more suitable for small workloads.

Both modes ensure optimal performance in different scenarios.

The research team tested Qwen3-32B, Qwen2.5-72B, and Llama-3.1-70B models on NVIDIA H20, H200, and B200 GPUs. Key metrics:

• KV Cache Capacity: 1.8x increase under the same configuration
• End-to-End Throughput: 1.5x improvement compared to the vLLM baseline

The improvements come from the VRAM space freed by SiDP, which can accommodate longer KV caches and support larger batches and sequences.

The value of SiDP lies in its new system architecture approach:

1. Weights don’t need to be copied everywhere: In high-bandwidth environments (e.g., NVLink), weights can be accessed on demand
2. Rebalance VRAM and computation: Reduce weight redundancy and allocate VRAM to KV cache
3. Flexible dual-mode design: Dynamically select the optimal strategy based on workload

This is important for ultra-large model offline inference, breaking the VRAM limit on batch size.

Through weight sharing, distributed pooling management, and dual-mode execution, SiDP significantly improves VRAM utilization and system throughput without sacrificing DP flexibility. As model scales grow, this bandwidth-for-VRAM design may become an important evolution direction for large-scale inference systems.