LLM Distributed Inference: A Variant-Optimized Auto-Scaling Solution

This article introduces a variant-optimized auto-scaling system for distributed large language model (LLM) inference workloads, aiming to address resource scheduling and performance optimization challenges in multi-model variant scenarios. Through innovative approaches like variant-aware scheduling, layered decision architecture, and predictive scaling, this solution balances cost-effectiveness and service quality. It is suitable for cost-sensitive applications, scenarios with intense traffic fluctuations, and multi-tenant inference platforms, representing a key direction in the intelligent evolution of LLM inference infrastructure.

LLM inference deployment has evolved from single-machine single-GPU to complex distributed architectures. Production environments need to manage multiple model variants (derived from the same base model with differences in parameter count, precision, etc.). Traditional auto-scaling solutions, designed for stateless web services, cannot adapt to the unique characteristics of LLM inference:

• Compute-intensive: GPU utilization is strongly correlated with token length
• Latency-sensitive: Time-to-First-Token (TTFT) and generation latency impact user experience
• Variant diversity: The same query can choose variants with different cost-quality trade-offs
• Resource heterogeneity: Clusters have GPUs of different generations and memory capacities

Model Variant Definition

Model variants are multiple versions derived from the same base architecture. Common types are as follows:

Variant Type	Description	Typical Scenario
Parameter Count Variant	Different scales such as 7B, 13B, 70B	Choose based on task complexity
Precision Variant	Quantized versions like FP16, INT8, INT4	Performance trade-off when resources are limited
Context Length Variant	Context lengths such as 4K, 32K, 128K	Long document processing vs short queries
Domain Variant	Vertical fine-tuning for code, math, etc.	Professional task optimization

Advantages of Variant-Aware Scheduling

Traditional scheduling treats variants as independent services, while the variant optimization solution leverages substitution relationships:

1. Elastic degradation: Route to low-cost variants when high-cost ones are insufficient
2. Load aggregation: Share GPUs for low-traffic variants to improve utilization
3. Preheating optimization: Preheat high-frequency variants to reduce cold start latency

Layered Decision Model

The scheduling problem is decomposed into three layers:

1. Global Capacity Planning: Calculate the target capacity range for each variant based on historical traffic and SLA
2. Inter-Variant Load Balancing: Real-time performance evaluation to dynamically adjust request routing
3. Instance-Level Scaling: Add/remove instances based on metrics like GPU utilization and KV cache

Key Technical Metrics

System monitoring and optimization metrics:

• GPU utilization: Actual usage of compute cores
• KV cache efficiency: Hit rate and fragmentation level
• Batch processing efficiency: Average batch size and padding efficiency
• Tail latency: P99 latency
• Cost efficiency: Cost per thousand tokens of inference

Predictive Scaling

Adopt a lightweight time-series prediction model to adjust capacity minutes in advance, responding to burst traffic (e.g., product launches).

Variant Cost-Performance Modeling

Maintain a dynamic model for each variant, integrating:

• Quality score: Task accuracy or human preference rating
• Resource consumption: Inference latency and GPU usage
• Monetary cost: Cloud GPU hourly pricing

Adaptive Batching

Implement adaptive continuous batching, dynamically adjusting batch parameters based on queue status and SLO to improve throughput.

K8s Ecosystem Integration

Through Custom Resource Definition (CRD) and Operator pattern, users declaratively define variant groups and scaling policies.

Cold Start Optimization

• Model preloading: Preload weights into memory when nodes start
• Layered initialization: Prioritize initialization of high-frequency model layers
• Instance pool buffer: Maintain hot standby instances to handle burst traffic

Multi-Cluster Federation

Support cross-region cluster scheduling, selecting the optimal execution location based on user location, compliance requirements, and load.

1. Cost-Sensitive Applications: Intelligently switch between high-precision and cost-effective variants to ensure core task quality while reducing edge query costs.
2. Traffic Fluctuation Scenarios: Predictive scaling and fast degradation to ensure service quality during traffic peaks and avoid resource idleness.
3. Multi-Tenant Inference Platforms: Support fine-grained resource isolation and priority management, automatically allocating shared resources.

Future Development Directions

1. Speculative decoding integration: Combine draft models to reduce latency
2. Heterogeneous hardware support: Utilize CPU, NPU, TPU, and other resources
3. Edge-cloud collaboration: Deploy lightweight variants at the edge and handle complex queries in the cloud
4. Reinforcement learning optimization: Use RL to learn optimal scaling strategies

Conclusion

Variant-optimized auto-scaling is a key direction in the intelligent evolution of LLM inference infrastructure. By understanding variant characteristics and load patterns, it achieves a balance between cost and service quality, worthy of team attention and exploration.