ARBITER OS: Distributed AI Inference Operating System and Topology-Aware Routing Architecture

ARBITER OS is a distributed AI inference operating system inspired by the Arbiter from the Halo series. It breaks single-machine AI boundaries to build a cross-device intelligent network. As a topology-aware routing and context layer above existing tools (Ollama, OpenClaw, cloud APIs), it serves two key roles:

1. Orchestrator: Routes tasks to optimal node-model combinations using rules, hardware capabilities, and fallback chains.
2. Liaison: Maintains context continuity across nodes/models for seamless task handover. It manages a Centralized Inference Network (CIN) for personal multi-machine AI infrastructure.

Background

The name draws from Halo's Arbiter—an elite warrior who broke dogma to form alliances, mirroring ARBITER OS's goal of uniting heterogeneous devices.

Project Overview

ARBITER OS is not a chatbot wrapper or model framework; it unifies existing AI tools. Its CIN includes:

• Nodes: Tailscale/Syncthing-connected machines (e.g., ThinkCentre M70q Gen5 as always-on inference center, GPD Pocket4 as dev workstation).
• Models: Local Ollama models (3B-140B+ params) and cloud models via OpenClaw (e.g., Kimi 2.5).
• Services: Tailscale (network), Syncthing (sync), SSH (remote exec), Proton VPN (privacy).

ARBITER OS’s architecture relies on three core subsystems:

1. Topology Registry: Perception layer maintaining network resources (hardware, models, services, health) in synced TOML files.

2. Context Thread: Memory system bridging model/node gaps via session context, briefing protocols, task history, and handover records.

3. Router Engine: Decision core classifying tasks, selecting optimal node-model pairs, using fallback chains, executing tasks, and logging decisions.

Technical Stack

• Core: Python (async-native), Textual (TUI), TOML (config).
• Network/Sync: Tailscale (secure grid), Syncthing (config/context sync).
• AI Integration: Ollama (local models), OpenClaw (cloud relay).

TUI Design

Phosphorescent/vector aesthetic (retro terminal + sci-fi) via Textual, supporting async operations. Accessible from any CIN node via Tailscale for remote health checks and task execution.

ARBITER OS solves critical multi-device AI workflow pain points:

• Heterogeneous Hardware Use: Routes light tasks to mobile devices, heavy tasks to workstations, creative tasks to cloud models.
• Context Continuity: Seamless device/model switches without context loss.
• Fault Tolerance: Fallback chains ensure task completion if preferred nodes/models fail.
• Privacy-Efficiency Balance: Local processing for sensitive tasks, cloud for resource-intensive ones.

Current Status

Early development (Phase 0):

• Static node configuration.
• Basic TUI interface.
• Simple task routing.

Roadmap

• Phase1: Real-time health polling, dynamic node discovery, performance metrics.
• Future: mDNS auto-discovery, complex task decomposition, multi-agent collaboration.

ARBITER OS shifts AI infrastructure from centralized cloud/single-machine systems to distributed personal device networks. It unifies heterogeneous devices via topology-aware routing and context management.

As edge AI and device computing power grow, systems like ARBITER OS may become standard for AI-native workflows—acting as a distributed OS designed for the AI era.