atomr-infer: A Heterogeneous LLM Inference Runtime Based on the Actor Model

atomr-infer is a Rust-implemented unified abstraction layer for LLM inference. Its core integrates local GPU runtimes (e.g., vLLM, TensorRT-LLM) and remote APIs (e.g., OpenAI, Anthropic) into a single interface via the Actor model, solving system fragmentation issues in heterogeneous inference scenarios. It supports flexible scaling from pure remote deployment with zero GPU dependency to heterogeneous clusters, providing developers with a unified mental model and stable system capabilities.

Modern AI application inference requirements exhibit heterogeneous characteristics: teams may simultaneously use local GPU clusters (e.g., DGX nodes equipped with H100, using vLLM/TRT-LLM) and remote managed APIs (OpenAI, Anthropic), while edge scenarios require lightweight CPU runtimes. This heterogeneity leads to system fragmentation: each runtime has its own SDK, retry/rate-limiting mechanisms, and the application layer needs a lot of glue code to handle differences. Additionally, the lack of unified error handling, backpressure management, and fault recovery poses stability risks.

atomr-infer is built on the Rust Actor runtime atomr, encapsulating each inference backend (local vLLM or remote OpenAI API) into a unified ModelRunner Actor. The core advantage is a unified mental model: developers only need to understand the Deployment value object, routing CRDT, and supervision tree to manage deployments from single machines to clusters. The same actor_ref.tell(msg) call can be routed to a local H100 or a remote API, and the caller does not need to care about underlying differences.

atomr-infer adopts a layered crate design:

• Core Abstraction Layer: Defines the Deployment value object (unified description of local/remote deployments, with automatic runtime inference), ModelRunner trait (inference interface), and InferenceError type system (structured error classification).
• Runtime Layer: Includes Gateway entry Actor, Request Actor session management, DP-Coordinator data parallel coordinator, and Placement/Deployment Manager (model replica placement and lifecycle).
• Remote Runtime Layer: Implements ModelRunner for OpenAI, Anthropic, Gemini, and LiteLLM, sharing infrastructure such as distributed rate limiters, circuit breakers, and retry strategies.
• Local Runtime Layer: Provides backends like vLLM (high throughput), TensorRT (low latency), ORT (cross-platform), and mistral.rs (pure Rust lightweight), interacting with GPUs via atomr-accel.

• Layered Compilation: Uses feature flags to prune dependencies, supporting build modes like remote-only (zero GPU dependency), default-prod (production heterogeneous deployment), and all-runtimes (development and testing), reducing binary size and ensuring zero GPU dependency at compile time.
• Pipeline Orchestration: Offers advanced features such as dynamic batching (improves GPU utilization), inference cascading (small models process first, low confidence automatically escalates), model replica pool and fair scheduling (fair allocation by tenant/priority), and model hot-swapping (version switching without service interruption).

• Observability: Each Actor automatically reports metrics like request latency, token throughput, queue depth, and error rate, supporting integration with Prometheus and Grafana.
• Resilience: Uses the Actor supervision tree mechanism—when a ModelRunner Actor fails, the supervisor restarts, degrades, or isolates it according to policies. Combined with distributed rate limiters and circuit breakers, it provides multi-level fault recovery capabilities.

atomr-infer is suitable for the following scenarios: hybrid deployment (local GPU + remote API), dynamic routing (cost/latency/quality trade-off), phased migration (pure remote to heterogeneous cluster), and high-demand production environments. As a Rust ecosystem project, it leverages Rust's advantages (deterministic resource usage, zero-cost abstractions, concurrent safety) to provide a reliable technical direction for the next-generation LLM service platform.