Concerto: An LLM Inference Multiplexer Written in Rust for Multi-Model GPU Cluster Sharing

Concerto is an LLM inference multiplexer written in Rust. It addresses the pain point of GPU memory waste in self-hosted LLM scenarios by dynamically loading and unloading models. It supports inference engines like vLLM, llama.cpp, and SGLang, enabling multi-model GPU cluster sharing and significantly improving resource utilization.

Traditional LLM deployment methods launch an independent inference process for each model, which permanently occupies memory, leading to significant resource waste from idle models. For example, when deploying 4 models on a server with 2 24GB GPUs, 50-70% of the memory is occupied by idle models for long periods, requiring additional GPU purchases.

Concerto's core innovation is dynamic model lifecycle management: loading models on request and unloading them after a period of idleness to free up memory. The trade-off is cold start latency (about 30-90 seconds for a 7B model on RTX A4000). It is suitable for scenarios like internal tools, batch processing, and multi-tenant fine-tuning, but not for real-time traffic requiring sub-second responses.

Concerto is implemented in Rust with a modular architecture including: concerto-api (HTTP interface), concerto-core (routing core), concerto-backend (inference engine management), and concerto-gpu (GPU telemetry). Key features include pluggable eviction policies, GPU health classification, TOML configuration, OpenAI-compatible API, and Prometheus metrics.

Applicable scenarios: multi-tenant SaaS platforms, enterprise internal AI platforms, research experimental environments, and AI features controlled by feature toggles. Comparison with competitors: it does not replace vLLM/SGLang but manages them; it complements K8s (fine-grained scheduling on single nodes); it differs from model merging solutions (keeps models independent).

Already implemented: routing core, GPU telemetry, multi-backend management, TOML configuration. v0.2 under development: OpenAI API, CLI, Prometheus metrics, etc. Future plans include a warm pool mechanism to reduce cold start time from 60 seconds to 5-10 seconds.

Build commands: basic build cargo build, enable NVML with cargo build --features nvml. It supports dual MIT/Apache 2.0 licenses. Recommendations: prioritize scenarios that are cost-sensitive, have many models, or have large load fluctuations; use cautiously in real-time low-latency scenarios.