Inference Budget Controller: LLM Inference Resource Budget and Auto-scaling Controller on Kubernetes

Inference Budget Controller is a resource management controller for LLM inference services in Kubernetes environments, designed to address problems such as high resource consumption of LLM inference services, severe idle waste, and inapplicability of traditional scaling solutions. Its core features include memory budget management, automatic scale-to-zero, and OpenAI-compatible admission control, helping enterprises optimize resource utilization, reduce operational costs, and improve service reliability.

With the widespread application of LLMs in production environments, enterprises face resource management challenges for LLM inference services: they require large amounts of GPU memory and computing resources, leading to resource waste during idle periods; traditional Kubernetes auto-scaling solutions struggle to handle the long model loading time, large memory footprint, and highly fluctuating request patterns of LLM inference.

1. Memory Budget Management: Introduces the concept of memory budget, where administrators can set usage limits. The controller continuously monitors consumption and triggers protection mechanisms when approaching the threshold to prevent a single service from occupying excessive resources and affecting other workloads.
2. Automatic Scale-to-Zero: Automatically scales down to zero replicas after the service is idle for a period to release GPU resources, and quickly recovers when new requests arrive; although there is cold start delay, it can significantly reduce costs in non-real-time scenarios.
3. OpenAI-Compatible Admission Control: Implements admission control in OpenAI API format, allowing applications to access without modification, supporting request-level rate limiting, queuing, and routing to ensure system stability under high load.

1. Controller Pattern: Adopts the Kubernetes controller pattern, driving scaling decisions by monitoring state changes of Custom Resource Definitions (CRDs), leveraging the advantages of declarative configuration to simplify resource policy management.
2. Layered Decision Mechanism: Includes a budget layer (decides whether to start new instances based on memory budget), a load layer (scales based on request queue depth and response latency levels), and an idle layer (detects idle time to trigger scale-to-zero).
3. State Persistence: Designs an efficient state persistence mechanism to ensure fast model loading when instances are rebuilt, reducing cold start time.

Deployment Configuration: Released as a Helm Chart, installed via standard Helm commands; users define inference service resource policies (memory budget, idle timeout, scaling thresholds, etc.) through Custom Resources (CRs), supporting independent management of multiple models. Application Scenarios:

• Development and testing environments: scale-to-zero reduces resource consumption, and quick recovery when needed;
• Off-peak optimization: scale down during off-peak hours and up during peak hours to optimize cloud resource costs;
• Multi-tenant isolation: memory budget prevents excessive resource consumption, and admission control ensures service quality.

Ecosystem Integration: Compatible with vLLM inference server; integrates Prometheus metric export, supporting Grafana monitoring; natively supports GitOps workflows, allowing policies to be automatically applied via CI/CD. Performance and Cost: Minimizes cold start delay through model preloading, image optimization, and node affinity (minimum replicas can be configured for latency-sensitive scenarios); typically saves 30%-70% of GPU resource costs, depending on traffic characteristics and policy parameters.

Future Directions: Support more fine-grained resource scheduling, integrate model quantization technology, enhance multi-cluster management capabilities, and explore deep integration with Serverless platforms. Summary: Provides a complete resource management solution for LLM inference services on Kubernetes. Through memory budget, automatic scale-to-zero, and OpenAI-compatible admission control, it helps enterprises optimize resources, reduce costs, and improve reliability, making it a production-ready solution worth considering.