MAVEN: A Multi-Stage Agentic Annotation Pipeline for Video Reasoning Tasks

MAVEN is an automated annotation system for video reasoning tasks, which converts raw videos into high-quality structured training data through a multi-stage agent collaboration pipeline. Its core advantages include support for domain adaptation (adapting to new domains without manual redesign), continuous quality improvement, and an efficient design of "one annotation, multi-task reuse". This system aims to address bottlenecks in video understanding annotation such as high labor costs, poor consistency, and limited scale, and has demonstrated significant results in the traffic video domain.

Training Visual Language Models (VLMs) for video event reasoning requires a large amount of high-quality structured annotations (covering multi-dimensional information such as event time, location, cause, and consequence). Manual annotation has three major bottlenecks: high cost (professional annotators spend hours on a single video), poor consistency (differences in subjective understanding), and limited scale (difficult to support million-level datasets). Existing automated methods are mostly single-stage designs, which cannot effectively capture temporal relationships and causal logic, leading to insufficient annotation quality.

MAVEN (Multi-stage Agentic Video Event aNnotation) is event-centric and introduces Multi-scale Spatio-Temporal Event Description (MSTED) as an explicit intermediate representation. Its core process consists of three stages:

1. Multi-scale Video Understanding: Global (overall scene), local (key frame/region), and temporal (event timeline) agents fuse outputs to generate MSTED;
2. Chain-of-Thought Generation: Reasoning agents generate explicit reasoning paths based on MSTED, enabling training data to include thinking processes;
3. Multi-task Data Synthesis: Generate multiple training samples such as multiple-choice questions, open-ended Q&A, event sorting, and causal reasoning from MSTED, achieving "one annotation, multi-task reuse".

One of MAVEN's core innovations is agent-driven domain adaptation capability:

• Automatic Prompt Engineering: After receiving problem examples from a new domain, it automatically analyzes domain features, rewrites prompt templates, and adjusts output formats to achieve zero-shot domain transfer;
• Hierarchical Refinement Loop: Through a closed loop of error classification → root cause tracking → targeted repair (prompt or process adjustment), it continuously improves data quality.

Large-scale validation results in the traffic video domain:

• Dataset: 5300+ surveillance videos covering urban roads, highways, and intersections, with event types including normal driving, violations, and accidents;
• Model Training: Fine-tuned the Cosmos-Reason2-8B model using MAVEN data;
• Core Results: On the private CCTV test set, the multiple-choice accuracy exceeded Gemini 2.5 Pro/3.1 Flash, improving by 38.8 percentage points over the zero-shot baseline; performance on the AccidentBench benchmark improved by 10.7 percentage points, and cross-domain generalization (warehouse/public safety videos) performed well.

MAVEN's technical contributions include:

1. Structured Intermediate Representation: MSTED achieves the separation of "video → description → multi-task generation", reducing multi-task annotation costs;
2. Agent Collaboration: Specialized division of labor improves the quality of individual links, intermediate outputs are interpretable for easy debugging, and individual agents can be flexibly upgraded;
3. Automatic Domain Transfer: No need to redesign processes, lowering the threshold for data construction in new domains, and accelerating model application and continuous learning.

MAVEN can be widely applied in scenarios such as intelligent transportation (accident/violation annotation), security monitoring (anomaly detection data), sports analysis (game event annotation), industrial quality inspection (defect detection), and educational videos (intelligent Q&A). The research team has open-sourced core components, provided pre-trained models and annotation data in the traffic domain, and is expected to significantly reduce the training data cost of video understanding models and promote the development of the field.