Aura: An Intelligent Cloud Resource Auto-scaling System for AI Workloads

Aura is a cloud infrastructure automation project built on AWS EKS, focused on providing intelligent elastic scaling capabilities for large language model (LLM) deployments. Its core value lies in significantly reducing GPU resource idle costs through predictive scheduling, addressing the shortcomings of traditional cloud resource management models in handling AI workloads (such as delayed scaling or waste from over-reservation).

With the widespread application of LLMs across industries, enterprises' demand for GPU computing resources has grown explosively, yet GPUs are costly and in short supply. Traditional resource management models (fixed reserved instances or simple threshold-based scaling) struggle to handle the characteristics of AI workloads such as sudden surges, uncertain duration, and large resource demand fluctuations, easily leading to business disruptions or resource idle waste.

The Aura architecture consists of three modules: the Perception Layer, Decision Layer, and Execution Layer:

• Perception Layer: Collects runtime metrics such as GPU utilization, memory usage, request queue length, and business context information;
• Decision Layer: Analyzes data through machine learning models to predict future resource demands;
• Execution Layer: Manages cloud resource operations (e.g., creating/destroying EKS node groups) via Infrastructure as Code (IaC) methods. Additionally, it adopts a temporary cluster design, reducing node readiness time to tens of seconds using pre-configured images and other technologies; it implements GPU-aware scheduling to allocate appropriate GPU instances based on task requirements.

Aura's prediction capabilities are based on the following technologies:

• Time-series Prediction Model: Uses Transformer architecture to process multi-variable time-series data, combining system metrics with external events (e.g., holidays, marketing campaigns) to predict resource demands for the next 15 minutes to 4 hours;
• Reinforcement Learning Optimization: Continuously evolves strategies through agent decision-making and reward signals (cost + service quality);
• Uncertainty Quantification: Uses Bayesian neural networks to quantify prediction errors and adjust strategies (conservative/aggressive) based on confidence levels.

According to project documents and early feedback, Aura has shown significant performance in LLM inference service scenarios: compared to fixed reserved instance mode, GPU resource costs are reduced by 40%-60% while maintaining P99 latency within an acceptable range. Cost savings come from on-demand scaling to avoid idleness, predictive scheduling to reduce cold start losses, and intelligent scheduling to improve GPU utilization.

Aura offers two deployment methods: Helm Chart and Terraform modules; it supports rich parameter adjustments (prediction sensitivity, scaling response speed, etc.); for compliance requirements, it supports private deployment with data retained in the user's AWS account.

As an open-source project, Aura will support multi-cloud (Google Cloud, Azure) in the future, leveraging price differences across cloud vendors to optimize costs; it will expand support for more AI workloads (training tasks, MLOps pipelines, vector databases, etc.), aiming to become the intelligent brain of cloud-native AI infrastructure.