ParaVT: A Multi-Agent Parallel Video Tool Calling Framework to Resolve the Tool Prior Paradox

ParaVT is the first end-to-end RL-trained multi-agent parallel video tool calling framework. Its core innovation lies in invoking multiple time window cropping tools simultaneously in a single dialogue turn, addressing the error propagation, context contamination, and inference cost issues of serial calling. The framework proposes the PARA-GRPO algorithm to tackle the tool prior paradox, achieving an average performance improvement of 7.9% across 6 long video understanding benchmarks.

Large multimodal models (LMMs) face context window capacity limitations when processing long videos, requiring tool calls to extend their perceptual capabilities. Existing RL-based tool calling methods mostly use serial mode, which has defects such as uncorrectable propagation of single wrong cropping, context contamination from multi-turn calls, and inference cost growing linearly with the number of turns.

ParaVT adopts a multi-agent architecture where the main model generates multiple cropping instructions, and sub-agents process the corresponding segment features in parallel and aggregate the results, compressing multi-turn tasks into a single turn to reduce latency. The framework uses end-to-end RL training, where the model autonomously learns the optimal cropping strategy through interaction with the environment instead of relying on imitation of manual annotations.

When applying standard RL to ParaVT, the tool prior paradox was discovered: the tool prior formed by LMM pre-training leads to format collapse (unparseable output) and tool-skipping shortcuts (directly guessing answers). Cross-model verification shows that weak-prior models have stable formats but cannot trigger tool calls, confirming that prior is both a necessary condition and a training threat.

PARA-GRPO adds two mechanisms to the standard GRPO: 1. Targeted format reward: Apply rewards only at structured token positions prone to collapse, stabilizing the format while preserving exploration space; 2. Frame budget randomization: Randomly vary the frame budget to force the model to call tools instead of relying on shortcuts.

ParaVT achieves an average improvement of 7.9% across 6 long video understanding benchmarks (including action recognition, temporal localization, and video question answering); PARA-GRPO increases the format compliance rate from 0.13 to 0.64; parallel calling compresses multi-turn interactions into a single turn, reducing inference latency proportionally with the number of turns.

ParaVT reveals that RL training needs to collaborate with pre-trained priors rather than confront them. Limitations include only supporting video cropping tools and high system complexity; future directions are to expand complex tool chains, explore the tool prior paradox in other domains, and develop general prior-aware RL algorithms.