Building a Production-Grade LLM Inference Platform: Full-Stack Practice from API Calls to FinOps

This article introduces the open-source project llm-platform, a production-oriented LLM inference platform that fills the gap in the open-source community for production-grade inference platforms. The platform has core capabilities such as multi-model routing, auto-scaling, observability, and FinOps cost control, aiming to push LLM inference from prototype to industrial deployment and embodying the systematic methodology of AI platform engineering.

By 2025, LLM application development has become simple (just call an API), but enterprises face a huge engineering gap when pushing it from prototype to production. Production environments need to handle complex issues like multi-model routing, load balancing, auto-scaling, performance monitoring, and cost control. However, most open-source projects either focus on model optimization or stay at the demo level, lacking complete platform solutions to support enterprise-level applications.

The llm-platform project is a complete platform engineering product that builds the infrastructure layer required for industrial LLM deployment. The core idea is: serving LLMs reliably, scalably, observably, and cost-effectively is an independent discipline—AI platform engineering. Its standout feature is FinOps capabilities: accurately measuring token consumption, response latency, and estimated cost for each inference request, and supporting cost attribution by model and user. This is extremely rare in open-source inference platforms but is a must-have for production environments.

The project uses a layered architecture with clear and replaceable responsibilities for each layer:

1. API Gateway Layer: Based on FastAPI, responsible for multi-model routing, authentication, rate limiting. Communicates with the backend via HTTP interface contracts, supporting backend replacement.
2. Model Service Layer: Runs on Kubernetes, supports Mock (GPU-free environment testing) and vLLM (high-performance inference) backends, achieving decoupling between infrastructure and models.
3. Observability Layer: Prometheus + Grafana collect and display metrics such as P99 latency and tokens processed per second.
4. FinOps Layer: Automatically calculates and records cost data via middleware.

The project adopts milestone-driven progressive delivery:

• Milestone 0: Repository skeleton setup and toolchain configuration;
• Milestone 1: Local Mock backend implementation;
• Milestone 2: Kubernetes deployment introduction;
• Milestone 3: Multi-model routing gateway construction;
• Milestone 4: Observability system integration;
• Milestone 5: FinOps cost measurement implementation;
• Milestone 6: GitOps and Infrastructure as Code automation completion. This path clearly shows the process of building a production-grade platform from scratch, with clear goals and verifiable results at each stage.

LLM inference costs are directly related to token consumption and business traffic; unoptimized systems easily generate high bills. The project's FinOps layer implements:

• Technical level: Accurately measure token consumption, response latency, and estimated cost;
• Business level: Support cost attribution by model and user. Administrators can optimize based on data: such as prompt optimization (abnormal user request costs), model distillation/quantization (rising model costs), etc.

Technology selection considers maturity and ecosystem:

• Python3.11: Rich AI ecosystem + modern language features;
• FastAPI: Automatic OpenAPI documentation + efficient asynchronous processing;
• Kubernetes: Container scheduling and resource management;
• Terraform + Helm: Infrastructure as Code and configuration standardization;
• kind: Local K8s cluster testing;
• Mock backend: Experience system functions even in GPU-free environments. Emphasizes local development experience and lowers the entry barrier.

Deployment follows the GitOps concept: All changes are version-controlled via Git, and CI/CD (GitHub Actions) applies them automatically; Terraform is responsible for infrastructure creation, and Helm manages K8s application deployment. Applicable scenarios: Enterprises needing self-hosted models to meet data privacy requirements, teams wanting fine-grained control over inference, multi-model combination service scenarios, internal AI capability centers. This project provides a runnable codebase and platform engineering methodology, making it an excellent learning sample for enterprises planning LLM infrastructure.