Zing Forum

Reading

RUSH: Building a Strategy Graph System Using Expert Feedback and High-Reasoning Models

RUSH is a strategy graph system that extends the decision quality of Subject Matter Experts (SMEs) to non-expert annotators through explicit, versioned, graph-structured strategies, enabling high-quality AI data annotation.

数据标注SME策略图ObsidianLLM生成式AI图像分类质量控制
Published 2026-05-21 12:43Recent activity 2026-05-21 12:51Estimated read 4 min
RUSH: Building a Strategy Graph System Using Expert Feedback and High-Reasoning Models
1

Section 01

RUSH: Strategy Graph System for Scaling Expert-Level Data Annotation

RUSH (Reinforcement learning Using SME feedback and High-reasoning models) is a strategy graph system designed to solve the problem of scaling expert (SME) decision quality to non-expert annotators for high-quality AI data annotation. Its core solution is to make strategies explicit, versioned, graph-structured, and validated with gold standards.

2

Section 02

Background: The Scaling Dilemma of Annotation Quality

In AI data annotation, a key challenge is enabling non-experts to reach SME-level quality. Traditional solutions have limitations: detailed guides can't cover all edge cases; multi-round audits are costly and slow; model consensus may lead to correlated failures. RUSH addresses this by rethinking strategy representation.

3

Section 03

Core Approach: Strategy as Explicit Graph Product

RUSH's core principles include: 1) Strategy as product (nodes with definition, standards, examples, source anchors); 2) Graph-structured strategy (Obsidian-style Markdown for clarity); 3) Auditable strategy diffs (patches for SME approval); 4) hierarchical metrics with clear denominators. Its project structure includes policy-graph, schemas, validation scripts, and a web demo.

4

Section 04

Pilot: GenAI Image Classification Strategy

RUSH's first pilot is GenAI image classification. Process: collect data → build strategy graph → capture SME-LLM disagreements → propose patches → SME approval. Binary labels: gen_ai (AI-generated) vs not_gen_ai (real, edited, CGI, or insufficient evidence). Subcategories: e.g., impossible hands, garbled text, plastic skin. Boundaries: photo editing, CGI/game renders, low-quality uncertain cases.

5

Section 05

Key Insights: Advantages of Explicit Graph Strategy

Explicit strategy beats implicit knowledge: auditable (trace decisions to nodes), iterable (patch-managed changes), teachable (fast onboarding), measurable (coverage as metric). Graph structure beats linear docs: visual coverage, modularity, linkability, progressive improvement.

6

Section 06

Limitations & Applicable Scenarios

Limitations: seed metrics are simulated; current web flow doesn't call LLMs; needs local image list. Applicable scenarios: high-quality annotation with limited budget, frequent strategy changes, multi-annotator collaboration, strict compliance requirements.

7

Section 07

Conclusion & Future Milestones

RUSH is a 'strategy-first' method to ensure data annotation quality. Future milestones: M1 (graph parsing/cold start), M2 (gold standard registry), M3 (annotation queue), M4 (metrics dashboard), M5 (strategy diff workflow), M6 (LLM integration)