TLG: A Three-Layer System for Video Temporal Logic Reasoning Achieves 71.37% Accuracy Using Real Annotations Instead of Large Models

TLG (Temporal-Logic Grounding) is a three-layer system for video temporal logic reasoning. It achieves 71.37% accuracy on the TimeLogic Challenge benchmark, a 24.5 percentage point improvement over the VLM baseline. Its core insight is that real annotations drive accuracy more effectively than model scale. Through methods such as timeline reconstruction using source annotations, temporal logic program execution, and targeted routing of weak categories, it demonstrates the value of cleverly leveraging existing annotation resources.

Video understanding requires handling action sequences, durations, and temporal relationships in the time dimension. The TimeLogic Challenge is a key benchmark for evaluating this capability:

• Includes 16 temporal operators (before/after/until, etc.)
• Question formats are boolean judgments or four-choice selections

Current end-to-end Video Language Models (VLMs) perform poorly:

• Accuracy is only about 46.9% (close to random)
• Root cause: Treating videos as "bags of frames" and failing to locate action times
• Limitation: Good at understanding "what", but struggling with "when"

The core idea of TLG is real annotations take precedence over model scale. The three-layer architecture is as follows:

1. Annotation Reconstruction and Deterministic Execution:
   • Reconstruct video action timelines from source dataset annotations
   • Parse the problem into a temporal logic program and execute it to get precise results
2. VLM Fallback: Use strong open-source VLMs as a supplement when there are no annotations
3. Targeted Reasoning Routing:
   • Identify the problem categories where VLMs perform the weakest
   • Route only these categories to cutting-edge reasoning models to balance cost and effectiveness

Core Results

Method	Accuracy	Improvement
VLM Baseline	46.9%	-
TLG	71.37%	+24.5%
Top of Leaderboard	~74%	-3%

Validation via Ablation Experiments

• Contribution of Layer 1: Using only annotation reconstruction achieves high performance, proving the value of real annotations
• Contribution of Layer 2: Fills the coverage gap for unannotated videos
• Contribution of Layer3: Targeted resolution of VLM weaknesses, further improving effectiveness

Key Findings

Comparing model-reconstructed timelines (VLM extraction, larger models, specialized temporal models) with real annotations:

• All model-reconstructed variants are weaker than real annotations
• Temporal grounding is the bottleneck, and real annotations are the key to solving it

TLG has made important progress in the field of video temporal reasoning:

• Achieves 71.37% accuracy, a 24.5 percentage point improvement over the baseline
• Core contribution: Proves that real annotations drive accuracy more effectively than model scale, challenging the "bigger is better" trend
• Methodological value: The combination of neural and symbolic approaches (neural network perception + symbolic logic reasoning) provides high interpretability and reliability
• Community insight: Data quality and utilization of existing resources are as important as model scale

Applicable Scenarios

• Scenarios requiring precise temporal understanding, such as video analysis, surveillance analysis, content moderation, and educational applications

Deployment Considerations

• Modular architecture: Offline timeline reconstruction + online logic execution + on-demand VLM services + selective cutting-edge model routing
• Cost optimization: Most queries are handled by the low-cost first layer

Limitations and Future Work

• Limitations: Dependent on source dataset annotations, only tested on TimeLogic Challenge, generalization to be verified
• Future Directions: Automatic annotation generation, multimodal expansion, online learning of routing strategies, open-source implementation