PYXIS3's Kubernetes LLM Inference Architecture Practice: From Runtime Selection to Fair Scheduling

【Introduction】Key Points of PYXIS3's Kubernetes LLM Inference Architecture Practice

Original Author/Maintainer: pyxis3-ai Source Platform: GitHub Original Link: https://github.com/pyxis3-ai/pyxis-arch Publication Date: 2026-06-04

This article provides an in-depth analysis of the PYXIS3 team's architectural design for running large-scale LLM inference workloads on Kubernetes. The core content includes:

• Selection strategies for mainstream LLM inference runtimes (vLLM, TGI, llama.cpp)
• Key technologies for GPU utilization optimization (memory management, model parallelism, warm-up caching)
• Fair sharing scheduling mechanisms in multi-tenant environments
• Observability and fault recovery solutions

It offers practical references for cloud-native deployment of LLM inference.

Background: Real Challenges of LLM Inference on the Cloud

As large language models (LLMs) move from labs to production environments, how to run inference services efficiently and stably on Kubernetes has become a core challenge for many teams. Unlike traditional microservice deployments, LLM inference workloads are resource-intensive, latency-sensitive, and have large throughput fluctuations, placing new demands on infrastructure. The open-source architecture design document from the PYXIS3 team presents a set of battle-tested solutions.

Runtime Selection: Trade-offs Between vLLM, TGI, and llama.cpp

The PYXIS3 team compared three mainstream LLM inference runtimes:

• vLLM: Known for its excellent throughput performance and PagedAttention technology, suitable for high-concurrency scenarios. It achieves higher GPU utilization through fine-grained memory management and is the first choice for large-scale production deployments.
• TGI (Text Generation Inference): Maintained by Hugging Face, it provides enterprise-level features such as streaming output, safety filtering, and multi-model parallelism, lowering integration barriers and suitable for teams needing standardized interfaces.
• llama.cpp: A tool for resource-constrained scenarios, supporting large models on consumer-grade hardware. It reduces memory requirements via quantization technology, ideal for edge deployments and development/test environments.

Practical experience shows that runtime selection should be based on business scenarios: prioritize vLLM for high-throughput production environments, choose TGI for functional integration needs, and use llama.cpp for resource-constrained/edge scenarios.

GPU Utilization Optimization: Memory Management and Request Scheduling Strategies

Efficient use of GPU resources is key to cost optimization. PYXIS3 emphasizes the following strategies:

1. Fine-grained memory management: Dynamic batching and continuous batching technologies reduce GPU idle time.
2. Model parallelism strategy: Ultra-large models require reasonable configuration of tensor parallelism and pipeline parallelism. It is recommended to find the optimal configuration through benchmarking based on model size and request characteristics.
3. Warm-up and caching mechanism: Preloading commonly used models plus efficient weight caching significantly reduces cold-start latency.

Fair Sharing Scheduling: Resource Allocation Mechanisms in Multi-Tenant Environments

When multiple teams share a GPU cluster, fair scheduling ensures service quality:

• Quota management: Set resource usage limits for different teams/applications to prevent a single workload from depleting cluster resources.
• Priority and preemption: Assign higher priority to critical businesses; allow high-priority tasks to preempt resources from low-priority ones when resources are tight.
• Request queue and timeout control: Reasonable queue length plus request timeout mechanisms protect the backend from overload and provide predictable service levels.

Observability and Fault Recovery: Key to Ensuring High Service Availability

LLM services in production environments require comprehensive observability: Monitoring dimensions include:

• Hardware metrics such as GPU utilization, memory usage, and temperature
• Request latency distribution (P50, P95, P99)
• Token generation rate and queue depth
• Error rate and timeout statistics

At the same time, auto-scaling (HPA/VPA) and graceful migration mechanisms for node failures are essential components to ensure high service availability.

Summary and Insights: Core Principles for LLM Inference Deployment

The open-source architecture document from PYXIS3 provides valuable practical experience. Core insights: There is no one-size-fits-all solution; successful deployment requires adjustments based on business characteristics:

• Runtime selection
• Resource optimization (GPU utilization)
• Fine-tuning of scheduling strategies

As LLM application scenarios expand, such production-validated architectural practices will become important assets for the community.