Combining TensorRT-LLM and DeepEP V2: A New High-Performance Inference Solution for MoE Models

This project integrates TensorRT-LLM, DeepEP V2, and AWS EFA technologies to provide a high-performance inference solution for Mixture-of-Experts (MoE) large language models. It aims to address key challenges in MoE inference such as communication overhead and load imbalance, significantly improving distributed inference efficiency while achieving a good balance between latency, throughput, and scalability.

Mixture-of-Experts (MoE) models achieve parameter expansion and controlled computational costs by splitting the feedforward network into multiple expert sub-networks and activating only a subset of experts. However, they also present unique inference challenges:

1. Communication Overhead in Expert Parallelism: In distributed deployment, different experts are distributed across different GPUs, requiring frequent cross-device communication for token routing.
2. Load Imbalance: Differences in expert activation frequencies lead to some GPUs being overloaded while others are idle.
3. Memory Bandwidth Bottleneck: MoE models have a huge number of parameters, placing extremely high demands on memory bandwidth.
4. Latency Sensitivity: Additional latency from expert routing affects real-time interaction experiences.

The project innovatively integrates three key technical components:

TensorRT-LLM

An optimization framework designed by NVIDIA specifically for LLM inference, providing capabilities such as operator fusion, INT8/FP8 quantization, paged attention, and multi-GPU parallelism. It is specially optimized for the expert computation and routing logic of MoE.

DeepEP V2

An expert parallel communication library that optimizes All-to-All communication, supports communication-computation overlap, and uses adaptive routing strategies to effectively reduce communication latency in MoE inference.

AWS EFA

An Elastic Fabric Adapter that provides OS bypass, RDMA support, and a high-throughput, low-latency network, offering high-performance infrastructure for cross-node expert communication.

Adopts the "inference cascading" design concept:

1. Local Priority: Process tokens on local GPUs first to reduce cross-node communication.
2. Hierarchical Routing: Call remote experts according to network topology hierarchy when local processing is insufficient.
3. Batch Aggregation: Batch routing requests to improve bandwidth utilization. Optimization directions for Wave30 version: finer-grained expert scheduling, dynamic load balancing, and memory layout optimization to increase cache hit rate.

Performance Advantages

• Latency Optimization: EFA low-latency network + DeepEP communication optimization significantly reduces cross-node expert call latency.
• Throughput Improvement: Communication-computation overlap and batch aggregation strategies efficiently utilize GPU resources.
• Scalability: Supports flexible scaling from single-node multi-GPU to multi-node clusters.

Application Scenarios

• Large-scale MoE model services (deployment of 100-billion/1-trillion parameter models).
• Multi-tenant inference platforms (resource sharing and performance isolation in cloud-native environments).
• Real-time interactive applications (low-latency responses for chatbots, code assistants, etc.).

Deployment Requirements

• Hardware: NVIDIA Ampere or higher GPUs, AWS EFA network cards, high-speed interconnection networks.
• Software: TensorRT-LLM, DeepEP V2, AWS EFA drivers, NCCL.

Technical Challenges

• Expert Placement Strategy: Optimal distribution needs to consider factors such as expert co-occurrence patterns, communication patterns, and load balancing.
• Fault Tolerance and Recovery: Rapid detection and rescheduling when nodes fail to ensure service continuity.
• Dynamic Scaling: Adjust the number of GPUs and expert allocation based on load to achieve efficient resource utilization.

Future Outlook

• Support for finer-grained expert structures (shared experts, hierarchical experts).
• Integration with compiler technology to achieve more aggressive operator optimization.
• Exploration of new network topologies to further reduce communication overhead.

Conclusion

The combination of TensorRT-LLM + DeepEP V2 + AWS EFA provides a powerful technology stack for high-performance inference of MoE models, balancing latency, throughput, and scalability. It is an open-source project worth attention for MoE production deployment, and its technical route is expected to become a standard paradigm for MoE inference.