CR²: Cost-Aware and Risk-Controllable LLM Inference Routing for Mobile Edge Scenarios

CR² is a cost-aware and risk-controllable LLM inference routing framework for mobile edge scenarios. It adopts a two-stage device-edge architecture (device-side edge gating + edge-side utility selector) and integrates a conformal risk control calibration mechanism to achieve flexible trade-offs between latency, energy consumption, and accuracy, reducing deployment costs by 16.9% compared to baseline methods.

The application scenarios of large language models (LLMs) are expanding from cloud data centers to mobile edges, but resource constraints in edge environments pose unique challenges: edge devices have limited computing/memory resources and cannot run large models directly; routing decisions need to balance the quality of local processing with the latency and energy consumption of edge calls; existing solutions are mostly designed for centralized cloud environments and do not consider the dynamic characteristics of wireless edges, leading to poor performance in actual deployments.

CR² uses a two-stage device-edge routing architecture: the first stage is a lightweight edge gate on the device side, which predicts the optimal utility of local execution by combining user cost weights; the second stage is an edge-side utility selector that evaluates the benefits of routing to a stronger model and makes the final decision. This design enables fast processing of most simple queries on the device side, reducing unnecessary network overhead.

CR² achieves explicit risk control through the Conformal Risk Control (CRC) calibration mechanism: before deployment, it uses validation data to select a threshold that meets the target risk level, ensuring that the false acceptance risk (device-side incorrect acceptance of low-quality outputs) is controlled within the preset confidence level; it supports users to adjust risk preferences according to scenarios (e.g., conservative for medical scenarios, lenient for real-time dialogue scenarios).

In real edge deployment scenarios, CR² dominates the accuracy-cost Pareto frontier: at the same accuracy level, the normalized deployment cost is reduced by 16.9% compared to the best baseline; the edge gate can accurately predict whether local execution is sufficient based on device-side signals; the actual false acceptance rate of CRC calibration is highly consistent with the target value, verifying the effectiveness of risk control.

CR² adapts to practical deployment needs: the edge gate is lightweight and can run on various edge devices; CRC calibration only needs to be completed once before deployment, simplifying operation and maintenance; it supports personalized cost weight settings for multiple users to meet different latency-quality preferences; when collaborating with speculative decoding, the small model on the device side can serve as both a gate and a draft model, reducing computational overhead.

CR² currently has limitations: it relies on the distribution consistency between validation data and deployment data; it assumes that there is a clear capability hierarchy between device-side and edge-side models; dynamic network condition estimation remains challenging. Future research can explore online adaptive calibration, support for complex capability structures, and intelligent routing strategies combined with network prediction models.