MultiToP: Mitigating Hallucinations in Video Multimodal Large Models via Visual Token Patching Technology

Research teams from Zhejiang University, Sun Yat-sen University, and East China Normal University proposed the MultiToP framework, which effectively mitigates hallucinations in video multimodal large models by finely patching unreliable visual tokens before language generation. This method improved the F1 score of Qwen3-VL-4B-Instruct by 50.60% on the Vript-HAL benchmark while maintaining general video understanding capabilities. The original paper was published on arXiv (June 10, 2026), link: https://arxiv.org/abs/2606.11792.

Video multimodal models (VideoLMMs) such as Qwen3-VL and Video-LLaVA perform strongly in video understanding and reasoning, but they have a hallucination problem: generated answers are inconsistent with video content, and the spatiotemporal dynamics of videos easily lead to combinatorial errors (e.g., incorrect entity associations, misjudgment of time order). Existing methods mostly intervene at the macro level (e.g., data, video), ignoring the reliability of visual tokens—unreliable tokens (background, redundant information) are easily amplified after entering the model, causing hallucinations.

The core of MultiToP (Multimodal-context-aware visual Token Patching) is to patch unreliable visual tokens before language generation. A lightweight visual token patcher is introduced to predict the replacement distribution and generate dynamic global patching tokens, selectively replacing unreliable tokens. Advantages: no need to modify the original model, negligible inference overhead, precise intervention on problematic tokens.

The training strategy uses information-guided ranking calibration: extract answer-conditioned frame-level information clues from the VideoLMM backbone (reflecting the importance of frames to the correct answer), and combine the token replacement distribution to guide patching. Training combines real answer supervision (to ensure that the patched tokens support correct answers) and sparsity regularization (to avoid over-intervention), balancing hallucination mitigation and model capabilities.

Experimental verification: On the Vript-HAL hallucination benchmark, Video-LLaVA-7B's F1 score increased by 9.68%, and Qwen3-VL-4B-Instruct's F1 score increased by 50.60%; in the ActivityNet-QA benchmark, Video-LLaVA-7B's relative accuracy increased by 18.58% while maintaining general capabilities. In terms of computational efficiency, the patcher is lightweight with minimal additional overhead, suitable for real-time deployment.

Technical details: The patcher uses an attention architecture to aggregate multimodal context; training data is based on existing video question-answering datasets, extracting frame-level clues to build signals; hyperparameter adjustment needs to balance the sparsity regularization weight (too strong leads to insufficient patching, too weak leads to over-patching).

Significance: Provides a new idea for token-level intervention, reveals the key impact of visual token reliability on generation quality, and helps understand model weaknesses. Future directions: expand to audio token patching, explore efficient patcher architectures, self-supervised training methods, and combine with existing reasoning intervention methods.