AutoCutAI: Autonomous Video Rough-Cut System Based on Semiotics and Rhythm Perception

AutoCutAI is a research-oriented multimodal video editing engine that generates narratively coherent film sequences from raw footage via visual symbol parsing, emotional trajectory modeling, and rhythm structure induction. This post breaks down its core strategies, perception modules, chaos analysis CI process, and future directions. Note: Current implementation focuses on beat-aligned shot assembly and chaos analysis, while advanced features (symbolic parsing, emotional curve extraction) are outlined in DESIGN.md (not in current code).

AutoCutAI is in the early research stage. Its README clarifies: the current implementation includes a deterministic rough-cut strategy (beat-aligned shot assembly) and a chaos analysis CI workflow. More ambitious goals—visual symbolic parsing, emotional trajectory modeling, generative editing grammar—are in DESIGN.md (not in current code). This transparency helps contributors understand boundaries between existing features and future plans.

The rough_cut_v1 strategy is a frame-precise, deterministic algorithm:

Input:

• VideoStructurePerception (lens boundaries, frame rate, resolution)
• AudioPerception (beat onset frame positions)

Output: A RoughCut object with EditDecision list (source [src_in, src_out] and target timeline position).

Algorithm Steps:

1. Filter short shots (discard <0.5s)
2. Align each retained shot's start to the nearest beat onset after original start
3. Recheck duration post-alignment, discard too-short fragments
4. Keep output frame rate same as input (no conversion)
5. Export EDL via RoughCut.to_csv(path)

This aligns with music video editing practices—syncing cuts to beats for visual rhythm.

Two core modules provide input for the rough-cut strategy:

1. AudioPerception: Extracts beat onset positions from audio tracks (foundation for rhythm alignment)
2. VideoStructurePerception: Detects lens boundaries via frame difference analysis, splitting footage into semantically coherent lens units

These modules together supply all necessary info for the rough-cut process.

AutoCutAI uses a chaos check workflow with three C++ native tools:

1. WTMM: Wavelet Transform Modulus Maxima—analyzes visual content complexity/change rate via multi-scale signal analysis
2. bb-extract: Exports basic block hit matrix from llvm-cov JSON to analyze code execution path complexity
3. jnorm: Computes Jacobian matrix infinity norm on LLVM IR using interval arithmetic for numerical stability

Tools are built via make native-tools and run in chaos-check.yml. Note: This is a "structural smoke test" (not formal verification) as per docs.

AutoCutAI uses modern Python practices:

• Languages: Python 3.12/3.13
• Dependency: Poetry 2.4.1
• Code Quality: Black (formatting), Ruff (linting), mypy (type checking)
• Testing: pytest
• CI/CD: GitHub Actions (two workflows: ci.yml for code quality checks; chaos-check.yml for chaos analysis)

Module structure: src/autocutai/ → editor/ (rough-cut + EDL), perception/ (audio/video), math/ (shared tools) Other dirs: ci/ (chaos pipeline), fixtures/chaos/ (input for chaos pipeline), tests/ (pytest suite)

AutoCutAI's value lies in its research framework. Future directions (per DESIGN.md):

• Visual Symbolic Parsing: Understand semantic layers of screen content
• Emotional Trajectory Modeling: Track audience's emotional response curve
• Generative Editing Grammar: Automatic editing based on narrative rules

Current rough-cut strategy is the first milestone toward these goals.

AutoCutAI is a research-driven open-source project with:

1. A runnable beat-aligned rough-cut strategy
2. Clear separation between perception layers and editing strategies
3. Unique chaos analysis CI for code complexity
4. An open research roadmap (DESIGN.md)

Licensed under Apache 2.0, with detailed contribution guidelines (CONTRIBUTING.md, CODE_OF_CONDUCT.md) to foster community participation. It represents a promising direction in combining multimodal AI with video editing.