KAITO Production-Grade Inference Stack: Open-Source Model Serving Practice on Kubernetes

The KAITO (Kubernetes AI Toolchain Operator) project aims to bring native LLM inference capabilities to Kubernetes. It simplifies open-source model deployment and management through declarative configuration, and combines llm-d to implement production-grade features such as auto-scaling and resource optimization, bridging the gap between Kubernetes' native architecture and the special needs of AI workloads.

As large language models move from experimentation to production, enterprises face challenges in running AI workloads efficiently under cloud-native architectures. Traditional machine learning deployment methods (manual GPU node configuration, model weight management, etc.) cannot meet the needs of elasticity, observability, and operational automation. Kubernetes, as the cloud-native orchestration standard, lacks special support for AI workloads (such as GPU scheduling, model caching, inference-specific scaling). Thus, the KAITO project was born with the goal of making open-source large model deployment and management as easy as ordinary microservices.

KAITO follows three design principles: declarative configuration, model-as-a-service, and heterogeneous hardware support. Its core components include: the Operator main controller in the control plane (monitors Workspace CRD changes, coordinates model downloads, Pod scheduling, etc.); Workspace CRD (user interaction interface that defines model name, version, resource requirements, etc.); Model Image Builder (converts raw model weights into container images, supports backends like vLLM/TGI/llama.cpp); GPU Provisioner (integrates with cloud vendor GPU instances, automatically creates/destroys nodes and supports spot instances for cost reduction).

KAITO has three production-grade features: 1. Elastic scaling: Adjusts instances based on the depth of the inference request queue; a preheating mechanism ensures new Pods are ready, and graceful scaling avoids interruptions. 2. Model management and caching: Hierarchical caching (node-local, shared storage, image layer) reduces loading time; version control supports blue-green/canary releases and rollbacks. 3. Multi-tenant isolation: Achieves resource and permission isolation via namespaces, ResourceQuota, and NetworkPolicy.

KAITO and llm-d have complementary architectures: KAITO focuses on large-scale production deployment at the K8s orchestration layer (auto-scaling, declarative CRD management); llm-d, based on Docker Compose, is suitable for development and testing (containerized inference, environment consistency). After integration, a unified pipeline can be achieved, allowing the same configuration to migrate seamlessly between local and cloud environments, and K8s manages llm-d containers to achieve production elasticity.

Deployment requires K8s 1.25+, GPU nodes (NVIDIA A100/A10/T4, etc.), and NVIDIA GPU Operator. KAITO is installed via Helm. The first model deployment is defined via Workspace YAML, and KAITO automatically completes model download, Pod scheduling, and service exposure. Performance optimization: vLLM continuous batching increases throughput by 2-10x, and paged attention optimizes KV cache. Cost optimization: spot instances reduce costs by 60-90%, single GPU runs multiple models, and INT8/INT4 quantization reduces memory requirements.

KAITO provides rich Prometheus metrics (inference latency, throughput, GPU utilization, etc.), structured JSON logs, and OpenTelemetry distributed tracing. Industry applications: Legal tech (multi-client isolation, 60% cost reduction); e-commerce (elastic scaling during traffic peaks); research institutions (fast model switching for comparative experiments).

Current limitations: Expensive GPU resources, long first-time model loading time, and complex multi-region deployment. Future directions: Support for more hardware (AMD/Intel/AWS Inferentia), Serverless inference mode, intelligent cache preheating strategy, and deep integration with Kubeflow/MLflow.