OmniVCHall：诊断视频多模态大模型组合幻觉的综合基准

This is an ICML 2026 accepted study presenting the first systematic benchmark (OmniVCHall) for diagnosing compositional hallucinations in video multimodal large language models (VLLMs). It also introduces the TriCD decoding framework, which can significantly improve model robustness without fine-tuning. Key focus areas include evaluating VLLMs' performance on combined visual evidence reasoning and addressing hallucination issues in complex video scenarios.

Video multimodal LLMs (VLLMs) have made progress in understanding complex video content but suffer from hallucinations (answers without content support). Existing benchmarks focus on single-type errors (e.g., wrong actions, time confusion), but real-world scenarios require joint reasoning over multiple visual evidence (object, action, time, camera motion, etc.)—this is called 'compositional hallucination', a major challenge for current VLLMs.

OmniVCHall is the first benchmark for compositional hallucination. It includes:

• Dataset: 823 videos (real + AI-generated) with 9,027 QA pairs (public on Hugging Face).
• 8 Hallucination Types: Object, Scene, Event, Action, Relation, Attribute, Temporal, Camera (newly introduced).
• Dual Test Structure: Single-type (one evidence) and Compositional (multiple evidences) queries, with Yes/No and Multiple-choice QA formats.

Evaluation of 39 mainstream VLLMs shows:

• Performance drops significantly when shifting from single-type to compositional queries (even top models).
• Camera motion reasoning is particularly hard: models often confuse lens movement (zoom/pan) with object motion, revealing flaws in visual grounding mechanisms.

TriCD (Triple-path Contrastive Decoding) is a no-fine-tuning framework to boost VLLM robustness:

• Three Paths:

1. Original: Standard model logits.
2. Negative: Adaptive perturbation (APC) to expose hallucination paths.
3. Positive: Saliency-guided enhancement (SGE) using DINOv3's spatial/temporal cues to reinforce evidence-supported predictions.

• Calibration Formula: q_t = q_t^o + α₁(q_t^p - q_t^o) + α₂(q_t^o - q_t^n) (encourages evidence-supported answers, suppresses hallucinations).

TriCD shows strong results:

• Improves average accuracy of representative VLLMs by over 10 percentage points (both Yes/No and Multiple-choice).
• Corrects camera motion confusion (lens vs object movement).
• Handles tricky questions (e.g., adversarial options like 'all correct'/'none').

Project code is available with setup steps:

1. Create environment: conda env create -f environment.yml then conda activate videoproject.
2. Smoke test: bash vcd/train/run_smoke_fast5_llavanv.sh.
3. Full training: bash vcd/train/run_fast5_subset1800_llavanv_1epoch.sh.

OmniVCHall and TriCD open new directions for VLLM hallucination research:

• Provides a standardized benchmark for compositional hallucination.
• Offers a cost-effective way (no fine-tuning) to improve model reliability.
• Valuable for video understanding, multimodal learning, and AI safety.
• Future work: Solving compositional hallucination to build trustworthy visual AI systems as video content grows in AI applications.