AiController: A Modular AI Inference Stack with Dynamic Backend Switching

This article introduces the open-source project AiController, a modular AI inference stack optimized specifically for NVIDIA DGX Spark. Its core features include dynamic backend switching between vLLM (large language model inference) and diffusers (image generation), addressing the challenges of backend adaptation and resource management in diverse inference scenarios. The project is maintained by lioilsources, with source code hosted on GitHub (link: https://github.com/lioilsources/AiController), and the update time is 2026-05-28T11:43:49Z.

With the development of generative AI, the complexity of inference scenarios has increased: LLMs require high-throughput text generation, image generation relies on diffusers; hardware varies greatly from cloud to edge. NVIDIA DGX Spark (formerly Project DIGITS) is a desktop-level high-performance AI device, but its software stack needs optimization to address issues such as multi-model support, dynamic backend selection, and simplified operation and maintenance.

AiController adopts a microservice architecture, decoupling modules for model loading, inference execution, request routing, and resource management. The dynamic backend switching mechanism records backend metadata (supported model types, load, resources, etc.) through a registry; the routing layer selects the optimal backend based on request characteristics and system status, with switching being transparent to the caller (unified RESTful/gRPC interface). Additionally, it implements containerized resource isolation (supports MPS/MIG), adaptive scheduling, and model lifecycle management (lazy loading, automatic unloading).

To address the limited VRAM issue of DGX Spark, AiController uses multi-level caching (active models in GPU VRAM, standby in memory, cold models on SSD) and integrates TensorRT optimization to improve throughput. In terms of quantization, it supports INT8/4 mixed precision, AWQ/GPTQ, and other algorithms; for image generation scenarios, it accelerates inference via LCM and distillation.

The application scenarios of AiController include:

1. Local development workstations: Run multiple models (CodeLlama, Stable Diffusion, etc.) on the same device, with a unified API to simplify development;
2. Edge inference nodes: Run both visual and dialogue models simultaneously in smart retail scenarios, with dynamic resource allocation;
3. Private services: Enterprises deploy DGX clusters to ensure data privacy and reduce costs.

The project provides containerized deployment solutions (Docker Compose/K8s), with declarative YAML configurations defining backends, model repositories, resource limits, etc. It has built-in health checks and Prometheus metric collection; logs support structured output and distributed tracing, facilitating monitoring and troubleshooting.

AiController provides an efficient solution for diverse AI inference scenarios through modularization and dynamic switching, fully leveraging the potential of DGX Spark. In the future, it will support more model backends (audio, video), reinforcement learning scheduling algorithms, and cloud-edge collaboration integration, offering an open-source option for local/edge multimodal AI deployment.