R3-CoVR: An Reasoning-Aware Framework for Zero-Shot Compositional Video Retrieval

This article introduces the R3-CoVR framework, which targets the Compositional Video Retrieval (CoVR) task. It achieves zero-shot retrieval using a frozen foundation model through a three-stage pipeline of "Reasoning-Retrieval-Reranking", and reaches an R@1 accuracy of 91.9% on the test set of the CVPR 2026 VidLLMs Challenge. This framework addresses the complex needs of users to find target videos based on reference videos and text modification instructions.

Traditional video retrieval is based on a single text query, while Compositional Video Retrieval (CoVR) needs to handle scenarios of reference video + text modification instructions (e.g., "A person walking in the park → running"). The core difficulty is understanding the semantics of state transitions, and Reasoning-Aware Compositional Video Retrieval (CoVR-R) further requires explicit reasoning of editing effects instead of simple feature concatenation.

The CoVR-R Challenge at the CVPR 2026 VidLLMs Workshop adopts a zero-shot setting: the system cannot use labeled training data for end-to-end training and only relies on pre-trained foundation models. The rationality of this setting lies in: improving generalization ability, enhancing reproducibility, and fitting real-world scenarios (lack of large amounts of labeled compositional video data).

The R3-CoVR framework is divided into three stages:

1. Reasoning: Use the Qwen3-VL-8B multimodal model, input reference video frames + modification instructions, and generate edited scene descriptions (including state transitions, action phases, etc.);
2. Retrieval: Use the SigLIP-2 contrastive encoder to encode text descriptions and candidate videos, and return Top-K candidates;
3. Reranking: Use the same model as a constraint-aware reranker to determine whether candidates comply with editing constraints and reorder them.

On the test set of the CVPR 2026 VidLLMs Challenge, R3-CoVR achieved excellent results:

Metric	Value	Description
R@1	91.9%	The proportion of cases where the top-ranked candidate is the correct answer
R@10	98.2%	The proportion of cases where the correct answer is among the top 10 candidates
This indicates that the framework performs excellently in both exact matching and recall rate.

The study identified two key decisions:

1. Matching Description Length with Encoder Window: When the description length matches the SigLIP-2 text window, R@1 increases from 67.5% to 72.7%, emphasizing the importance of aligning tasks with model capabilities;
2. Gain from Constraint-Aware Reranker: After adding the reranking stage, R@1 increases from 72.7% to 91.9% (+19.2%), effectively filtering out false positives from the retrieval stage.

• Model Freezing Strategy: Fully rely on frozen foundation models, with advantages of computational efficiency, stability, and scalability;
• Prompt Engineering in Reasoning Stage: Adopt structured prompt templates to guide the model to generate descriptions from dimensions such as action changes and scene environments;
• Scoring Mechanism in Reranking Stage: Output continuous scores (not binary judgments) to improve ranking accuracy.

Insights: 1. Explicit reasoning of intermediate representations improves accuracy; 2. Multi-stage architecture is suitable for complex compositional retrieval tasks; 3. Composing foundation models can achieve good results in zero-shot settings. Limitations: High computational cost, insufficient scalability for large-scale video libraries, and unproven generalization of new editing instructions. Future Directions: Develop efficient reranking strategies, explore the potential of end-to-end fine-tuning, and extend to other compositional retrieval tasks.