Epyc Orchestrator: Engineering Practice of a Hierarchical Orchestration System for Local LLMs

Epyc Orchestrator is a hierarchical multi-model orchestration system for local LLM inference, designed to resolve the conflict between speed and quality under limited hardware resources in local inference. It achieves efficient task scheduling through technologies like intelligent routing, automatic escalation, and speculative decoding. Adopting a four-tier model echelon architecture, it supports both Mock and production deployment modes, suitable for scenarios such as enterprise privatization and real-time interaction, providing a complete engineering reference solution for local LLM deployment.

With the rapid development of open-source LLMs, local deployment is favored by developers due to its advantages in privacy protection and cost control, but it faces core challenges: How to balance response speed and output quality under limited hardware resources? A single-model solution is hard to achieve both—lightweight models are fast but have limited capabilities, while large-parameter models are powerful but slow in inference. Epyc Orchestrator is designed as a hierarchical orchestration system to address this issue.

The system adopts a hierarchical model organization strategy, divided into four capability tiers:

• Tier A (Front Door Layer): Lightweight models handle simple queries (e.g., greetings, basic Q&A) to provide instant feedback;
• Tier B (Expert Layer): Domain-specific professional models (code experts, architects, etc.) handle tasks requiring specific skills;
• Tier C (Worker Layer): General-purpose models balancing capability and speed, responsible for exploratory tasks, math calculations, etc.;
• Tier D (Draft Layer): Draft and embedding models that accelerate upper-layer model inference by generating candidate tokens.

Intelligent Routing and Automatic Escalation

Requests are analyzed for complexity by the routing component and assigned to the appropriate tier. If the model fails to complete the task on time or the output quality is substandard, it automatically escalates to a higher tier, and events are recorded to optimize the routing strategy.

Speculative Decoding Acceleration

Uses Tier D lightweight draft models to generate candidate token sequences, and the main model verifies them in parallel, achieving 2-12x acceleration, suitable for real-time interaction scenarios (e.g., dialogue, code completion).

Contextual Memory and Skill Tracking

FAISS-based contextual memory supports long-term cross-session memory; skill tracking monitors task success rates and dynamically adjusts model allocation strategies.

Tool Execution and MCP Integration

A sandboxed REPL environment supports code execution, network retrieval, etc., with a plug-in design for easy expansion; implements a Model Context Protocol (MCP) server for seamless integration with external tools.

The system supports two operation modes:

• Mock mode: No local models required; enable by setting the environment variable ORCHESTRATOR_MOCK_MODE=1, suitable for development and testing;
• Production mode: Requires configuring a llama.cpp model server, edit the .env file to set model paths, and configure each tier's model roles, acceleration parameters, and timeout policies via model_registry.yaml. Configuration is based on pydantic-settings, supporting full registry mode (including model paths and performance data) or simplified mode (only routing and timeout configurations).

Epyc Orchestrator is particularly suitable for the following scenarios:

1. Enterprise privatization deployment: Run LLMs locally to meet performance requirements for tasks of varying complexity;
2. Multi-model resource management: Maximize hardware utilization of local multi-scale models;
3. Real-time interaction applications: Latency-sensitive scenarios like customer service bots and code assistants;
4. Long-session applications: Complex dialogue systems with cross-session memory and personalized responses.

Epyc Orchestrator demonstrates an engineering solution for local LLM inference. Through hierarchical architecture, intelligent routing, and speculative decoding, it achieves response speed and output quality close to cloud APIs under limited hardware resources. It provides a complete reference implementation for production-level local LLM deployment. As local model capabilities improve, the hierarchical orchestration approach may become a standard practice for local LLM applications.