Decoupled Large Model Inference Validation Framework on AWS EFA v2: End-to-End Practice from NCCL to SGLang PD

This article introduces the open-source AWS EFA v2 Decoupled Large Model Inference Validation Framework by KevinZhao. Targeting the AWS EKS environment with EFA v2 RDMA network, it validates the full-chain feasibility from underlying network performance to upper-layer SGLang Prefill-Decode decoupled deployment. The framework adopts a four-layer progressive validation architecture, providing reproducible testing methods and performance benchmarks for production-grade decoupled LLM inference deployments.

Traditional single-node inference faces memory bottlenecks and throughput limitations. Decoupled inference distributes the compute-intensive Prefill phase and memory-bandwidth-intensive Decode phase across different nodes to optimize resource utilization. However, this architecture needs to address the low-latency and high-bandwidth requirements for cross-node KV Cache transmission. AWS EFA v2 provides high-performance RDMA networks, but systematic validation of multi-layer software stack performance from hardware to applications is required.

The first phase of the framework uses NCCL to test all-reduce/all-to-all operations on p5.48xlarge instances, with the measured all-reduce bandwidth reaching 476.91 GB/s (exceeding the 320GB/s threshold). The second phase validates the low-latency dispatch/combine operations of UCCL-EP: 16 ranks passed the correctness test, with each rank achieving a throughput of approximately 7GB/s, meeting the functional validation requirements.

The third phase uses the Mooncake KV transmission engine, with DRAM-DRAM write throughput at 19.31 GB/s (there is a gap from the 150GB/s target, requiring tuning). The fourth phase of SGLang Prefill-Decode decoupled deployment (1P:1D) shows that TPOT drops to 0.53 times the single-node baseline (Decode acceleration), but TTFT increases to 7.7 times (needing optimization of Prefill overhead or scheduling strategies).

The framework is built on Kubernetes and depends on EKS 1.35+, NVIDIA GPU Operator v24.9.2, MPI Operator v0.6.0, LeaderWorkerSet v0.7.0, etc. It provides 5 dedicated container images; it is recommended to build the images on EC2 m7i.4xlarge+ instances to avoid network bottlenecks.

The project's RUNBOOK.md records complete test logs (including failure and repair methods). Workflow: Configure .env parameters → Build images → Create K8s resources → Test phase by phase. Best practices include: Ensuring correct EFA/OFI configuration, monitoring bandwidth utilization, and adjusting KV Cache transmission buffers.

The value of the framework lies in providing reproducible phased testing methods, lowering the threshold for AWS decoupled inference validation. Applicable scenarios include EFA applicability assessment, SGLang PD performance validation, software stack baseline establishment, etc. Currently, NCCL/UCCL-EP have passed validation, while Mooncake/SGLang need optimization; in the future, with component iterations, the production readiness of the decoupled architecture will be further improved.