Practical Guide to Local Large Model Inference Server: Efficient Deployment of MoE Models on RTX 5080

This article introduces llm-server, a local LLM inference server project optimized for consumer-grade GPUs. Based on the llama.cpp framework, it optimizes Mixture of Experts (MoE) model inference, achieving efficient inference of the Qwen3.5-35B-A3B model on NVIDIA RTX5080 16GB GPU with a performance of 75 tokens per second. It provides practical references for local AI deployment and addresses needs in scenarios such as privacy, offline use, and cost control.

With the development of large language model technology, developers and researchers hope to deploy large models locally, but the high hardware resource requirements of large models are daunting. The open-source llm-server project aims to solve this problem by demonstrating how to achieve efficient large model inference on consumer-grade GPUs, providing a solution for local AI deployment.

llama.cpp Basic Architecture

llama.cpp is a lightweight and efficient inference library that supports cross-platform, GGUF quantization format, CUDA and other hardware accelerations, and batch processing capabilities.

Special Optimization for MoE Models

• Intelligent Layered Offloading: Active experts reside on the GPU, inactive ones are offloaded to memory, with predictive preloading;
• Memory Management Optimization: Expert parameter caching, dynamic memory pool, KV cache optimization, supporting 35B-parameter MoE models with 16GB VRAM.

Performance of Qwen3.5-35B-A3B model on RTX5080 16GB configuration:

• Inference speed: ~75 tokens per second;
• VRAM usage: ~14GB (including KV cache);
• Memory usage: ~8GB (inactive experts);
• First token latency: 200-300 ms. Compared to traditional solutions: VRAM usage reduced by 40%, inference speed doubled, supporting larger MoE models.

Hardware Requirements

Minimum: RTX3080 10GB; Recommended: RTX5080 16GB, 32GB DDR5 memory, SSD storage.

Software Dependencies

CUDA Toolkit 12.x, CMake 3.20+, Python 3.10+.

Quick Start

1. Download precompiled binaries or compile from source; 2. Obtain MoE models in GGUF format; 3. Configure parameters; 4. Start the server.

API Interfaces

RESTful interfaces compatible with OpenAI API, supporting conversation completion, text completion, model query, and streaming output.

Privacy-Sensitive Scenarios

Local processing of sensitive data in medical consultation, legal consultation, enterprise internal use, etc., avoiding the risk of cloud upload.

Offline Environments

Providing reliable AI capabilities in network-free or restricted environments such as field research, confidential networks, and emergency backups.

Cost Optimization

Development testing, large-scale content generation, replacing commercial API calls to reduce long-term costs.

VRAM Bottleneck

Challenge: 35B models still require a lot of VRAM after quantization; Solutions: 4-bit quantization, dynamic expert offloading, KV cache layout optimization.

Inference Latency

Challenge: Extra latency from expert switching; Solutions: Expert access prediction, preloading popular experts, asynchronous loading.

Concurrent Processing

Challenge: Multiple requests accessing different experts; Solutions: Request batching, expert sharing, dynamic load balancing.

Future Directions

• Multi-card support: Expert parallelism, tensor parallelism, pipeline parallelism;
• Heterogeneous computing: CPU+GPU hybrid, NPU/DPU support, cloud collaboration;
• Model compression: Aggressive quantization, expert pruning, knowledge distillation.

Conclusion

llm-server proves that consumer-grade GPUs can efficiently run 35B-parameter MoE models. Local AI deployment will become more common, providing more possibilities for privacy protection, cost control, and offline applications.