Agentic Runtime Platform: Architectural Practice of a Production-Grade Multi-Agent Orchestration Platform

Agentic Runtime Platform, an open-source multi-agent orchestration platform, addresses the reliability, observability, and cost optimization challenges of complex AI workflows through innovative designs such as a DAG execution engine, hierarchical model routing, and Rubric evaluation framework. This article will discuss the platform's background, core architecture, evaluation system, practical applications, and conclusions.

With the enhancement of large language model capabilities, AI applications have evolved from single model calls to multi-agent collaborative architectures. Typical complex tasks (such as code review and research report generation) require collaboration among multiple specialized agents (planning, research, coding, review). However, building multi-agent systems faces many challenges: How to define agent dependencies? How to handle mixed parallel and sequential scenarios? How to select models with optimal cost and performance? How to implement cross-vendor failover? These have spurred the demand for specialized orchestration platforms.

Agentic Runtime Platform uses DAG (Directed Acyclic Graph) as the underlying execution model for workflows, implementing topological sorting and parallel scheduling based on Kahn's algorithm. Compared to traditional linear pipelines, DAG can naturally express complex dependencies:

• Fan-out/Fan-in Mode: After a single task is completed, multiple downstream tasks run in parallel, then converge for summary
• Conditional Branching: Dynamically decide step execution based on runtime conditions
• Iterative Loop: Loop with boundary conditions until the quality threshold is met
• Failure Cascade Propagation: Cancel dependent downstream tasks when a key step fails The platform uses asyncio for parallel scheduling, leveraging asyncio.wait(FIRST_COMPLETED) to maximize throughput and improve efficiency.

The platform introduces the concept of "ability stratification", where each agent is assigned to an ability tier rather than a specific model:

• Tier1 (Lightweight Layer): gemini-2.0-flash-lite, gpt-4o-mini, and other fast, low-cost models
• Tier2 (Standard Layer): gemini-2.0-flash, claude-3-haiku, and other balanced models
• Tier3 (Enhanced Layer): gemini-2.5-flash, gpt-4o, and other high-performance models
• Tier4 (Expert Layer): gemini-2.5-pro, claude-3.5-sonnet, and other top-tier models At runtime, the SmartModelRouter selects models based on weighted factors of health, latency, and cost, and has a built-in failover chain (e.g., Tier3 alternative chain: gemini-2.5-flash→github:gpt-4o→openai:gpt-4o→anthropic:claude-sonnet). It also implements adaptive cooling: models with consecutive failures are exponentially backed off, and their weights are restored once they are healthy.

Evaluation Framework: Built-in Rubric-based multi-dimensional scoring, categorized into 5 orthogonal dimensions:

• Coverage: Whether all aspects of the problem are fully addressed
• Source Quality: Whether references are authoritative and reliable
• Consistency: Whether internal logic is self-consistent
• Verifiability: Whether conclusions can be independently verified
• Timeliness: Whether information is up-to-date Each dimension is scored as S/A/B/C/D/F to form a multi-dimensional quality profile.

Observability:

• Real-time DAG Visualization: React19 dashboard with SSE/WebSocket streaming to display execution status
• Token Usage Tracking: Record token count, API call frequency, estimated cost, and support cost attribution
• Historical Execution Replay: Save complete history for review and debugging
• Zero-Credential Development Mode: The AGENTIC_NO_LLM=1 environment variable allows running tests without API keys, simulating LLM responses.

Built-in Templates: The platform preconfigures 6 production-grade workflow templates:

Workflow	Mode	Description
code_review	Fan-out/Fan-in	Parse code → parallel architecture review + quality review → comprehensive report
bug_resolution	Sequential + Verification	Reproduce → root cause analysis → fix → test → verify
fullstack_generation	Parallel Sub-steps	API design → parallel front-end and back-end development → integration
iterative_review	Multi-cycle	Review → feedback → revision until passing the quality gate
conditional_branching	Conditional DAG	Dynamically execute or skip steps based on runtime conditions
test_deterministic	Tier-0	Purely deterministic steps, no LLM calls required

Tech Stack: Developed with Python3.11+, core dependencies include FastAPI (server), LangGraph (state machine compilation), Pydantic v2 (type safety); test coverage exceeds 80%, supporting over 8 mainstream LLM providers.

Application Scenarios: Enterprise code review, research report generation, customer service upgrade, content moderation pipelines, etc.

Agentic Runtime Platform provides reliable orchestration infrastructure for production-grade multi-agent applications through its DAG execution engine, hierarchical model routing, and refined evaluation framework. Declarative workflow definition lowers the usage threshold, comprehensive observability ensures maintainability in production environments, and the zero-credential development mode optimizes the developer experience. As AI application complexity increases, specialized multi-agent orchestration platforms will become key infrastructure for building reliable AI systems, and its open-source release provides reference engineering practices for this field.