Watching Time Flow: New Frontiers in Video Time Perception and Manipulation

The research team treats 'time' as a learnable visual concept and has developed models that can perceive and manipulate the flow of time in videos. Using self-supervised learning to detect speed changes and building the largest slow-motion dataset to date, they have achieved speed-conditional video generation and temporal super-resolution, opening up a new dimension for video understanding.

Video is at the core of modern computer vision research, yet the fundamental dimension of time flow has long been neglected. Existing systems struggle to distinguish speed variations in videos or generate videos at a specified speed; time remains in a state of 'being seen but not understood'. The research raises a fundamental question: Can time be treated as a learnable visual concept?

Using multimodal cues (visual motion, audio rhythm, scene dynamics) and temporal structures (continuity constraints, periodic patterns, causal order), the model learns to detect speed changes and estimate playback speed without manual speed label annotations.

Speed-Conditional Video Generation

Challenges: Physical consistency (motion follows physical laws), detail preservation (slow motion requires fine temporal details), semantic understanding (action adaptability); Possible architectures include temporal embedding, multi-scale modeling, and integration of physical constraints.

Temporal Super-Resolution

Difference from spatial super-resolution: Handles dynamic details (motion trajectories, deformations); Application scenarios: Old video restoration, slow-motion synthesis, action analysis.

Existing slow-motion data is scarce and of varying quality. The team used a temporal reasoning model to filter and organize videos from the internet: candidate clip identification → quality screening → speed normalization → diversity assurance, resulting in the largest slow-motion dataset.

Establishes time as a manipulable perceptual dimension, analogous to breakthroughs in color, depth, and style; Application prospects: Time-controllable video generation, temporal forensics detection, temporal understanding in world models (grasping event rhythms).

Current Limitations

Performance degradation at extreme speeds (very slow/very fast), challenges in complex scenes (multiple independent moving objects), and temporal consistency in long videos needing improvement.

Future Directions

Support variable speeds within videos, causal reasoning to generate physically consistent manipulations, cross-modal temporal alignment (extending to audio/text).