GAR: A Carbon-Aware Routing Optimization Framework for LLM Inference

Google Research team proposes the GAR (Green-Aware Routing) framework, which incorporates carbon emissions into LLM inference routing decisions. It minimizes the CO₂ emissions per request while meeting the minimum accuracy threshold and p95 latency Service Level Objective (SLO), providing a theoretical foundation and practical solutions for green AI inference.

The deployment scale of Large Language Models (LLMs) is growing rapidly. Existing routing methods mostly balance response quality and computing costs, but rarely take sustainable energy use and CO₂ emissions as optimization objectives—even though grid carbon intensity varies by time and region, and energy consumption differs significantly between models. With the explosive growth of AI inference demand, its carbon footprint is accumulating rapidly, posing severe environmental challenges.

GAR is a constrained multi-objective optimization framework, with the core goal of minimizing carbon emissions while meeting the minimum accuracy threshold and p95 latency SLO. Its key innovations include:

1. Adaptive constraint optimization: Adjust the minimum accuracy threshold for each dataset to dynamically adapt to task requirements;
2. Lightweight estimator: Integrates correctness, tail latency, and carbon emission estimation, supporting real-time routing decisions without additional inference overhead;
3. Online primal-dual algorithm (GAR-PD): Designed specifically for rolling carbon budget scenarios to dynamically and efficiently allocate resources.

GAR models the routing problem as a constrained multi-objective optimization problem, considering three dimensions simultaneously:

1. Carbon emission minimization: Prioritize models and regions with lower carbon intensity;
2. Accuracy guarantee: Ensure response quality does not fall below the preset threshold;
3. Latency constraint: Meet the p95 latency SLO requirements. In addition, the research team developed heuristic variants: Strict Mode (prioritize accuracy and latency), Balanced Mode (balance all three), and Green Mode (prioritize carbon emission minimization), providing flexible options for different scenarios.

The research team evaluated GAR on standard NLP benchmarks using a heterogeneous LLM pool (7B-70B parameter scale). The results show:

1. Carbon reduction: Achieves considerable CO₂ reduction compared to traditional routing strategies;
2. Accuracy: Meets the minimum accuracy threshold, with performance loss controlled within an acceptable range;
3. Latency: Reliably meets the p95 latency SLO;
4. Scalability: Performs well across model pools from 7B to 70B parameters, with strong generalization ability.

The practical deployment value of the GAR framework is reflected in multiple aspects:

• Cloud service providers: Helps meet environmental regulations and ESG requirements, reduces data center carbon footprint and energy costs, and enhances green brand image;
• Enterprise users: Enables sustainable AI deployment without affecting service quality, meets internal carbon neutrality goals, and optimizes inference costs (green energy is usually cheaper);
• AI industry: Drives the industry toward a more sustainable direction, provides references for green AI standard formulation, and promotes the popularization of carbon-aware AI infrastructure.

GAR has the following limitations and future exploration directions:

1. Real-time carbon data dependency: Effectiveness relies on accurate real-time grid carbon intensity data; data quality affects optimization results;
2. Model energy consumption modeling: Currently based on offline measured model energy consumption data; future exploration can include online energy consumption estimation;
3. Multi-tenant scenarios: Fair allocation of carbon budgets in shared infrastructure requires further research;
4. Edge deployment: Extend to edge computing scenarios, considering device-level energy consumption and on-site use of renewable energy.