Design of Production-Grade Large Model Inference System: A Complete Guide from Architecture to Implementation

Fastino Labs' open-source LLM inference system design document details how to build a production-grade inference platform supporting 5000 RPS with P95 latency below 2 seconds. It covers core technologies like multi-tenant isolation, cache-aware routing, and paged attention, and provides a complete architecture design and runnable code skeleton to serve as a reference for building LLM inference services.

As LLMs move from labs to production, inference services need to meet requirements such as high RPS (thousands of requests per second), low latency, multi-tenant isolation, and maximum GPU resource utilization. Fastino Labs' document provides a complete engineering solution, including architecture design and Python skeleton code, to help build production-grade inference services from scratch.

The system's target scale is clear: the model is a 13B-parameter FP16 model (occupying 26GB of VRAM), using NVIDIA H100 80GB hardware; performance targets are 5000 RPS and P95 latency <2 seconds. Capacity estimation: KV cache per request is about 1.1GB, each H100 supports 45 concurrent requests, and 5000 RPS requires approximately 12 H100s.

The system adopts a three-layer architecture:

1. Entry layer: Includes L7 load balancing, tenant-aware routing (hash to reuse KV cache), token cost rate limiting (token-based leaky bucket), and admission queue (backpressure mechanism);
2. Inference layer: Based on vLLM, using paged attention (improves memory utilization to 80-90%), continuous batching (dynamically add/remove sequences), and chunked prefill scheduling (reduces tail latency for long prompts);
3. Streaming layer: Uses SSE protocol to push tokens in real time, reducing TTFT.

Several tradeoffs were made in the design:

• Cache affinity vs load balancing: Gain KV reuse benefits via hash routing;
• Pessimistic reservation vs precise billing: Simplify implementation with maximum token count reservation;
• Chunked prefill vs throughput: Sacrifice a small amount of throughput to improve tail latency.

The project provides a complete implementation skeleton (gateway, inference worker, scheduler, etc.), supports local demos on CPU/Apple Silicon, and allows switching to the vLLM engine. Deployment recommendations: Build a cluster with 12 H100s, orchestrate with Kubernetes or Docker Compose, and refer to docs/sizing.md for capacity calculation.

The document demonstrates a systematic engineering design methodology (goal → estimation → architecture → tradeoff → implementation), providing a reference architecture for LLM inference teams. Designs like multi-tenant isolation and cache routing can be directly applied. Large model inference requires a combination of engineering and algorithms, and such open-source documents are of great value to the industry.