Practical Guide to LLM Inference Performance Optimization: Quantitative Evaluation of PD Disaggregation Architecture in Code Assistant Scenarios

This article deeply analyzes the PD-Disaggregation-Eval project, comparing the performance of single-GPU homogeneous deployment and dual-GPU PD disaggregation architecture under code completion workloads through end-to-end experiments. Key findings include a ~50% reduction in P99 Time to First Token (TTFT), more stable Time per Output Token (TPOT), etc., providing quantitative decision-making basis for computing scheduling in production environments.

Modern LLM inference consists of two stages: Prefill (compute-intensive with high parallelism) and Decode (limited by memory bandwidth). In traditional single-GPU deployment, resource sharing between the two stages leads to increased Decode latency (head-of-line blocking) when long-context inputs cause Prefill to occupy resources. Code assistant scenarios have strict requirements for TTFT (first response speed) and TPOT (completion fluency), and rising concurrency amplifies resource competition issues. The PD disaggregation architecture decouples the two stages into independent computing units, eliminating resource competition.

The PD-Disaggregation-Eval project was completed by the wang-zr12 team at the end of 2024, with two phases of experiments: 1. Establish a baseline using a single A100 80GB GPU, scanning concurrency, input sequence length (ISL), and output sequence length (OSL); 2. Implement PD disaggregation with dual A100 40GB GPUs, using the experimental KV transfer feature of vLLM 0.7.3. The model used is Qwen2.5-Coder-7B-Instruct (optimized for code scenarios), and the workloads reference HumanEval and SWE-bench, covering three types of tasks: inline completion, code explanation, and function generation.

Under a 20 QPS mixed workload, the PD disaggregation architecture shows significant performance: P99 TTFT is reduced by approximately 50% (more immediate long-tail first responses); TPOT is more stable, avoiding jitter in single-GPU mode; although the improvement in end-to-end latency (E2E) is smaller, the KV transfer overhead under NVLink 3.0 is offset by parallelization benefits, resulting in an overall positive ROI.

The team built a benefit boundary framework based on the Roofline model, integrating variables such as QPS, ISL, and interconnection bandwidth to predict the break-even point. Verification with 30 configuration combinations shows: when the product of QPS and ISL exceeds the threshold, the benefits of PD disaggregation become apparent; single-GPU deployment is better for low-concurrency and short-input scenarios. This provides a quantitative basis for dynamic scheduling strategies.

PD disaggregation relies on the extensibility of the vLLM framework, using the KV cache transfer mechanism of PyNcclConnector (NCCL for inter-GPU communication). Attention should be paid to max_model_len settings and KV transfer buffer tuning. The project provides a complete reproducible workflow (from Colab single-GPU baseline to cloud dual-GPU PD deployment), and the documentation details steps for model download, environment configuration, and benchmark testing.

The benefits of PD disaggregation depend on workload characteristics; it is recommended to conduct small-scale experiments first to determine the break-even point. Interconnection bandwidth is critical—KV transfer overhead is high under PCIe 4.0, requiring NVLink or high-speed interconnection. Dynamic scheduling (switching between single-GPU and PD modes based on load) is a future direction, and the project's analysis framework lays the foundation for this.

As LLM penetration in production scenarios increases, inference performance optimization has become an engineering necessity. PD-Disaggregation-Eval provides a reference baseline through rigorous experiments and data analysis, offering valuable insights for technical leaders in architecture selection and researchers in understanding LLM inference characteristics.