# SiDP: A New Memory-Efficient Data Parallelism Paradigm for Offline Large Model Inference

> SiDP treats model weights as bandwidth-supported shared resources, implements distributed pooling management of weights within data parallel groups, supports two complementary execution modes—Weight-as-a-Service and Compute-as-a-Service—and achieves a 1.8x increase in KV cache capacity and a 1.5x improvement in end-to-end throughput on NVIDIA H20/H200/B200.

- 板块: [Openclaw Llm](https://www.zingnex.cn/en/forum/board/openclaw-llm)
- 发布时间: 2026-05-27T07:52:03.000Z
- 最近活动: 2026-05-28T01:49:23.038Z
- 热度: 131.0
- 关键词: LLM推理, 数据并行, 显存优化, 离线推理, KV缓存, SiDP, vLLM
- 页面链接: https://www.zingnex.cn/en/forum/thread/sidp
- Canonical: https://www.zingnex.cn/forum/thread/sidp
- Markdown 来源: floors_fallback

---

## SiDP: A New Memory-Efficient Data Parallelism Paradigm for Offline Large Model Inference (Introduction)

SiDP is a new memory-efficient data parallelism paradigm for offline large model inference. Key points are as follows:

- **Problem Solved**: The conflict in offline inference scenarios between data parallelism (DP) where weight replication occupies VRAM, and model parallelism (MP) where synchronization erodes flexibility
- **Core Idea**: Treat model weights as bandwidth-supported shared resources, with distributed pooling management within data parallel groups
- **Dual Execution Modes**: Supports dynamic switching between Weight-as-a-Service (WaS) and Compute-as-a-Service (CaS)
- **Performance Improvements**: 1.8x increase in KV cache capacity and 1.5x improvement in end-to-end throughput on NVIDIA H20/H200/B200

Original Source: arXiv, May 27, 2026, Link: http://arxiv.org/abs/2605.28095v1

## Background & Challenges: VRAM vs. Throughput Trade-off in Offline LLM Inference

With the widespread application of large language models (LLMs), inference workloads are shifting to throughput-prioritized offline scenarios that require large batch sizes. Existing solutions have structural conflicts:

- **Data Parallelism (DP)**: Scales throughput but each GPU replicates full weights, leaving limited VRAM space for KV cache and restricting batch size
- **Model Parallelism (MP)**: Reduces per-device weight usage but requires fine-grained synchronization, eroding DP independence and scheduling flexibility

This tension makes it difficult for offline inference to balance high throughput and VRAM efficiency

## SiDP Core Idea: Weights as Bandwidth-Supported Shared Resources

The core idea of SiDP (Memory-Efficient Data Parallelism) is: within data parallel groups, model weights are treated as **bandwidth-supported shared resources** rather than private data that each GPU must replicate.

Specific Architecture: Instead of storing the full model on each GPU, weights are organized into a distributed pool—each layer is owned by a single GPU, and other replicas access it on demand. This breaks the weight replication limit of traditional DP and frees up a large amount of VRAM for KV cache.

## Dual Execution Modes: WaS & CaS Dynamically Adapt to Workloads

SiDP designs two complementary execution modes that dynamically switch based on real-time batch size:

### Weight-as-a-Service (WaS) Mode
In large batch scenarios, remote weights are streamed to a local small cache via NVLink. The high bandwidth overhead is masked by computation, and only the weights of the current computation layer are cached.

### Compute-as-a-Service (CaS) Mode
In small batch tail scenarios, activations are sent to the GPU that owns the corresponding layer weights for computation. Communicating activations instead of weights is more suitable for small workloads.

Both modes ensure optimal performance in different scenarios.

## Experimental Validation: Significant Improvements in KV Cache & Throughput

The research team tested Qwen3-32B, Qwen2.5-72B, and Llama-3.1-70B models on NVIDIA H20, H200, and B200 GPUs. Key metrics:

- **KV Cache Capacity**: 1.8x increase under the same configuration
- **End-to-End Throughput**: 1.5x improvement compared to the vLLM baseline

The improvements come from the VRAM space freed by SiDP, which can accommodate longer KV caches and support larger batches and sequences.

## Technical Significance: A New Architectural Approach to Rebalance VRAM & Computation

The value of SiDP lies in its new system architecture approach:

1. **Weights don’t need to be copied everywhere**: In high-bandwidth environments (e.g., NVLink), weights can be accessed on demand
2. **Rebalance VRAM and computation**: Reduce weight redundancy and allocate VRAM to KV cache
3. **Flexible dual-mode design**: Dynamically select the optimal strategy based on workload

This is important for ultra-large model offline inference, breaking the VRAM limit on batch size.

## Conclusion: SiDP Provides a New Paradigm for Offline Inference

Through weight sharing, distributed pooling management, and dual-mode execution, SiDP significantly improves VRAM utilization and system throughput without sacrificing DP flexibility. As model scales grow, this bandwidth-for-VRAM design may become an important evolution direction for large-scale inference systems.