Shardon: A Self-Hosted LLM Routing and Scheduling Platform for Constrained GPU Environments

Shardon is a self-hosted Large Language Model (LLM) routing and scheduling platform designed for constrained GPU environments. It aims to address key challenges enterprises face when deploying LLMs, such as scarce GPU resources, coexistence of multiple models, cost optimization, and API compatibility. Its core features include dynamic model loading, GPU group-aware scheduling, an OpenAI-compatible API layer, and a Linux-first optimization strategy, providing enterprises with deployable, maintainable, and scalable LLM inference infrastructure.

With the popularization of LLMs in enterprises, traditional deployment models (dedicated GPU instances or unlimited cloud scaling) struggle to handle real-world constraints: 1. Scarce GPU resources (most enterprises only have consumer-grade GPUs or even CPUs); 2. Need for multi-model coexistence (different teams require different models, frequent switching); 3. Cost optimization pressure (GPU idle waste requires intelligent lifecycle management); 4. API compatibility requirements (existing toolchains are based on OpenAI API, avoiding refactoring is necessary). Shardon is a Linux-first self-hosted platform designed specifically for these constraints.

Shardon's design philosophy is "seeking optimal solutions within constraints". Its core architecture includes:

1. Dynamic Model Loading: On-demand loading (lazy loading + LRU cache), supports GGUF quantization format, automatically selects precision based on video memory;
2. GPU Group-Aware Scheduling: Divides physical GPUs into logical groups, supports heterogeneous management, load balancing (round-robin/least connections), GPU affinity, and failover;
3. OpenAI-Compatible API Layer: Fully supports core endpoints (e.g., /v1/chat/completions), adds enterprise features (request priority, rate limiting, multi-key management).

Shardon's technical implementation focuses on practicality and optimization:

• Linux-First Optimization: Integrates systemd (auto-start/restart), cgroups (resource isolation), eBPF (fine-grained monitoring), and supports containerized deployment;
• Inference Backend Integration: Defaults to llama.cpp (GGUF format, cross-platform optimization), optional vLLM (high throughput), supports custom backends;
• Management Interface & Tools: Web UI provides model repository management, real-time monitoring dashboard, A/B testing, audit logs, and other features.

Shardon is suitable for various scenarios:

1. Internal AI Platform for SMEs: Teams of 10-100 people, 2x RTX4090 can host 3-5 quantized models, supporting 50-200 concurrent users;
2. Development & Testing Environment: CPU-only mode for running small models, supports Docker/K8s integration and Mock mode;
3. Edge Computing & Hybrid Cloud: Local processing of sensitive data, cloud as overflow backup, unified OpenAI interface;
4. Research & Education Environment: Multi-user GPU sharing, model version management, resource usage reports.

Feature	Shardon	vLLM	TGI (Hugging Face)	Ollama
Dynamic Model Loading	Core Feature	Not Supported	Not Supported	Supported
GPU Group Scheduling	Natively Supported	Basic Support	Basic Support	Not Supported
OpenAI API Compatibility	Full	Partial	Partial	Partial
Management Interface	Built-in	None	Yes	Basic
Consumer-grade GPU Optimization	Yes	No	No	Yes
Enterprise Features	Yes	No	Partial	No
Deployment Complexity	Medium	High	High	Low

Current Limitations: Limited support for Windows/macOS; Performance ceiling (generality sacrifices some extreme performance); Model format support focuses on GGUF, native formats require conversion. Future Roadmap: Multimodal support (VLM inference); Distributed inference (cross-node model/data parallelism); Auto-scaling (K8s HPA integration); Federated learning integration (model fine-tuning under privacy protection).

Shardon represents a pragmatic AI infrastructure design philosophy, providing deployable, maintainable, and scalable solutions under real-world constraints. It lowers the threshold for enterprises to integrate LLMs into existing IT infrastructure, serving as a bridge between cutting-edge AI capabilities and actual business needs. As LLMs move toward production environments, such infrastructure layers will become increasingly important.