MnemoCUDA: A Streaming Inference Engine for Running 235B+ Parameter MoE Large Models on Consumer GPUs

MnemoCUDA is a streaming inference engine. Through expert streaming loading and intelligent memory management technologies, it breaks through the memory limitations of consumer GPUs, allowing 235B+ parameter MoE large models to run efficiently locally, providing a key technical path for the democratization of large models.

Mixture of Experts (MoE) is the mainstream architecture for scaling large language models currently. It can increase the number of parameters while maintaining computational efficiency, but during inference, the complete expert weights need to reside in memory. A 235B parameter MoE model may exceed 100GB of memory even after quantization, far beyond the capacity of consumer GPUs (e.g., RTX4090 with 24GB), leading ordinary developers to rely on cloud services and hindering AI democratization.

MnemoCUDA proposes an expert streaming loading scheme. Based on the sparse activation characteristics of MoE, it only loads the experts that are about to be activated from main memory/SSD to GPU memory, and unloads those not in use temporarily. Through pipeline overlapping technology, it parallelizes expert loading with current computation, and uses prefetching strategies to hide IO latency, making memory demand proportional to the number of activated experts.

MnemoCUDA uses a three-level cache: L1 (GPU memory) stores currently/soon-to-be activated experts; L2 (host memory) stores recently inactive experts; L3 (NVMe SSD) stores the complete expert library. The layered design adapts to hardware configurations, maximizing hit rates and minimizing loading overhead through intelligent prefetching and cache replacement.

MnemoCUDA integrates multiple compression technologies: expert-level quantization (allocating precision based on sensitivity), expert sharing and deduplication (reducing redundant parameters), and incremental encoding (storing only weight differences), significantly reducing storage volume and transmission bandwidth requirements.

MnemoCUDA successfully runs 235B parameter MoE models on RTX4090/3090 (24GB memory); during inference, it controls loading overhead through overlapping computation, with increased latency within an acceptable range for interactive applications; the streaming architecture supports model scaling—only additional SSD storage is needed to handle larger models.

The open-sourcing of MnemoCUDA lowers the research threshold for ultra-large MoE models, allowing more developers to participate; it provides the possibility of local deployment for edge AI (offline/privacy scenarios); its ideas such as streaming loading and multi-level caching can be extended to other sparsely activated models, providing references for efficient inference system design and promoting AI inclusiveness.