SAI: An Enterprise-Grade AI Agent Framework Centered on Evaluation, Building Trustworthy Automated Workflows

This article introduces the SAI (Structured AI) framework—an open-source AI Agent framework for enterprise scenarios. Its core is treating evaluation data as a first-class citizen. Through designs like cascaded execution, human-machine separated verification mechanism, and complete audit logs, it solves the trustworthiness issues of AI automation in production environments. The framework aims to balance cost and quality, meeting enterprises' needs for AI systems such as trustworthiness, auditability, and permission management.

Large language models have spawned numerous personal AI tools, but enterprises face four major challenges when adopting AI automation: trustworthiness (needing an evidence chain to prove completion), regression risk (model updates may break functions), audit requirements (traceable operations), and permission management (fine-grained control). SAI originated from a Cornell University course project; initially a RAG-version teaching assistant tool, it evolved into an AI automation framework for production environments after two years of iteration.

SAI's core philosophy is "evaluation data is a first-class citizen". The system collects and uses each user interaction (approval, editing, etc.) as structured feedback. Its cascaded execution architecture establishes hierarchical decision-making among rules → classifiers → local LLM → cloud LLM → humans: simple tasks are resolved at early levels, complex tasks are escalated; during construction, costs are gradually reduced from cloud to local. Additionally, workflows (skills) are defined via skill.yaml manifests, enforcing evaluation requirements; policy gating separates permission decisions from execution to reduce risks.

SAI defines five types of evaluation datasets: CanaryDataset (ensures rules take effect), EdgeCaseDataset (records reasoning cases), WorkflowDataset (captures workflow drift), DisagreementDataset (model disagreements), and TrueNorthDataset (long-term trend benchmarks). For security, it uses multi-layer protection such as per-workflow OAuth scopes, reality-only as truth value, append-only audit logs, hash-verified loading, and reflection suggestions not being applied automatically.

SAI provides two onboarding paths: Wizard Mode (guided configuration via Claude Code/Co-Work, completing the first email tagging in 30 minutes) and Manual Mode (cloning the repository, configuring the environment, etc.). Interaction is mainly through the Slack #sai-eval channel or local HTTP fallback, supporting lightweight feedback: input a rule proposal, apply it after reacting with ✅, and continuously improve the taxonomy.

SAI is currently in the early stage, mainly focused on email classification scenarios, and has crash risks. Future directions include more workflow templates, evaluation visualization, multi-modal support, error recovery, enterprise SSO integration, etc. SAI explores the path of trustworthy AI automation, providing enterprises with a starting point for reliable, predictable, and maintainable AI systems. Its core insights have reference value for AI tool design.