AIR Runtime: An Adaptive LLM Inference Engine for Resource-Constrained Environments

AIR Runtime is an adaptive inference runtime system designed for resource-constrained environments (e.g., edge devices, consumer GPUs). It addresses issues like memory limitations, latency sensitivity, throughput requirements, and energy constraints in LLM inference through core technologies such as intelligent routing, speculative decoding, and KV cache compression, enabling performance breakthroughs on limited hardware.

LLM inference needs to run on various hardware from cloud to edge, presenting the following challenges:

• Memory Limitations: Consumer GPUs (e.g., RTX4090 with 24GB memory) struggle to accommodate large models
• Latency Sensitivity: Interactive applications require low-latency responses
• Throughput Requirements: Service scenarios demand high concurrent processing
• Energy Constraints: Mobile/edge devices have strict power consumption requirements Traditional one-size-fits-all solutions fail to fully utilize hardware potential, leading to the birth of AIR Runtime.

Intelligent Routing

Distributes requests by dynamically analyzing input features:

• Input Classification: Classify based on query complexity, domain features, length, etc.
• Model Selection: Intelligently choose among multi-scale models
• Path Optimization: Simple queries use lightweight models; complex queries use large models Benefits: Reduced resource consumption, lower latency, support for heterogeneous deployment

Speculative Decoding

Uses a 'draft-verify' mode to accelerate generation:

1. Draft Phase: Small models quickly generate candidate tokens
2. Verification Phase: Main model verifies candidates in parallel
3. Accept/Reject: Accept if matched; regenerate otherwise Optimization Points: Draft model selection strategy, dynamic adjustment of verification batches, real-time monitoring of acceptance rate.

KV cache is a major memory consumer in Transformer inference. AIR uses multiple compression technologies:

Technology	Principle	Compression Ratio	Quality Impact
Quantization Compression	Quantize FP16/FP32 to INT8/INT4	2-4x	Minor
Sparsification	Remove low-importance KV pairs	1.5-2x	Moderate
Sliding Window	Retain KV of the latest N tokens	Variable	Task-dependent
Dynamic Allocation	Allocate precision based on sequence importance	2-3x	Controllable
Challenges: Compression/decompression overhead, task variation impact, attention mechanism compatibility.

Hardware-Aware Scheduling

Continuously monitors metrics like GPU memory, memory bandwidth, compute utilization, power consumption, and temperature to dynamically adjust:

• Batch size
• Compression level
• Speculative decoding draft length
• Optimization strategy enablement status

Load Adaptation

Optimizes for different loads:

• Short sequences with high concurrency: Prioritize KV cache compression
• Long sequences with low concurrency: Enable speculative decoding
• Mixed loads: Intelligently route to different queues.

Typical Scenarios

1. Edge Device Deployment: Run 7B-scale models on Jetson, Raspberry Pi
2. Consumer GPU Inference: Run models requiring 40GB+ memory on a single 24GB GPU
3. High-Concurrency Services: Serve more requests with fixed hardware
4. Mobile Device Integration: Local LLM assistants on phones/tablets

Performance Improvements

• Throughput: 2-4x (batch processing + speculative decoding)
• Latency: Reduced by 30-50% (routing + parallel verification)
• Memory Usage: Reduced by 40-60% (KV compression)
• Energy Efficiency: Improved by 2-3x.

Implementation Points

• Enhances underlying engines like vLLM/TensorRT-LLM at the upper layer
• Challenges: Low-overhead monitoring, microsecond-level fast decision-making, stability assurance, cross-platform compatibility

Limitations

• Adaptive strategies require hardware tuning
• Some optimizations have limited effect on specific model architectures
• Compression benefits diminish for small models (<3B)

Usage Recommendations

• Conduct sufficient benchmark testing before production
• Adjust adaptive parameters based on load
• Monitor the impact of compression on output quality.

AIR Runtime represents the shift of LLM inference optimization from static configuration to dynamic adaptation. As model scales grow and deployment scenarios diversify, such 'context-aware' systems will become a necessity. In the future, more adaptive technologies will enable large language models to be truly widely adopted across various devices.