Orla: Harvard's Open-Source High-Performance Multi-Agent System Execution Engine

Orla is an open-source project from Harvard's Computer Science Laboratory (Harvard CNS) Professor Minlan Yu's team, providing a unified execution framework for building and running LLM-based multi-agent systems. Its core design principle is separating workflow decision-making from request execution, enabling efficient scheduling and coordination across heterogeneous models. Key features include support for heterogeneous model routing, workflow orchestration with fault tolerance, and cross-stage KV cache management to boost inference efficiency.

As LLM capabilities evolve, multi-agent applications shift from single dialogue to complex multi-step workflows. Manual orchestration of model calls, tool executions, and infrastructure brings challenges:

1. Tight coupling between workflow decision logic and execution, making maintenance/extension hard.
2. Lack of unified abstraction for scheduling across models/backends, requiring custom adapters.
3. Complex state management (e.g., KV cache sharing/reuse) needing custom implementations.

Orla's architecture centers on three components:

• Stage Mapper: Maps workflow stages to suitable models/backends via declarative requirements, optimizing resource use (e.g., GPU for intensive tasks, CPU for simple ones).
• Workflow Orchestrator: Coordinates execution order/dependencies, supports parallel/conditional/loop execution, and has fault tolerance (retry/recovery).
• Memory Manager: Manages KV cache across stages, enabling reuse to reduce redundant computation and improve inference efficiency in multi-step scenarios.

Orla uses Go for its core engine (high performance, low resource use) and provides a Python SDK (pyorla) for easy integration. Installation methods:

• Daemon: brew install --cask harvard-cns/orla/orla
• Python SDK: pip install pyorla This dual-language approach balances performance and accessibility.

Orla is suitable for:

• Complex dialogue systems: KV cache reuse reduces latency in multi-round conversations.
• Tool call workflows: Orchestrates tool order/dependencies and routes to appropriate backends.
• Multi-model collaboration: Unifies orchestration for scenarios like code generation → review → documentation.

Orla is backed by academic research, with a paper Orla: A Library for Serving LLM-Based Multi-Agent Systems on arXiv (authors: Rana Shahout, Hayder Tirmazi, Minlan Yu, Michael Mitzenmacher). It's open-source, with contribution guidelines and GitHub Issues for community interaction.

Orla represents a shift from manual to declarative, high-performance multi-agent execution frameworks. By separating concerns, providing unified abstractions, and optimizing KV cache management, it lays a solid foundation for production-grade multi-agent apps. As LLM applications expand, such engines will become critical for lowering development barriers and ensuring performance/reliability. Teams building multi-agent systems should consider evaluating Orla.