Video Modality Diagnostics: Diagnose Whether Multimodal Video Models "Truly" Understand Video Content

Video Modality Diagnostics (VMD) is a tool for diagnosing multimodal VideoQA models (visual/audio/subtitle), supporting modality ablation, contribution analysis, and robustness testing. It can be used for offline testing or with the Colab VLM backend. Its core purpose is to help researchers determine whether models truly utilize video information rather than relying on audio or subtitles to "cheat".

Original author/maintainer: mlahozy21, Source platform: GitHub, Project link: https://github.com/mlahozy21/video-modality-diagnostics, Update time: 2026-06-11T14:42:28Z.

In recent years, multimodal VideoQA models have made significant progress, but a core issue has been overlooked: do models actually watch the video, or do they rely on audio/subtitles? If a model mainly depends on non-visual modalities, it will perform poorly on pure visual tasks and mislead capability evaluation. The VMD project aims to solve this problem by providing systematic tools to quantify the degree of model dependence on each modality.

VMD adopts three strategies:

1. Modality Ablation Experiment: Remove a certain modality input and observe performance changes (supports combinations such as pure visual, pure audio, pure subtitle, etc.);
2. Modality Contribution Analysis: Measure output changes by perturbing modality inputs (e.g., adding noise, shuffling time sequence) to quantify contributions and generate visual heatmaps;
3. Robustness Testing: Evaluate the model's performance under adversarial perturbations (invisible noise), time perturbations (deleting/repeating frames), spatial perturbations (cropping/occlusion), and cross-modal inconsistencies (contradictory audio and video).

VMD adopts a modular design:

• Core Engine: The src/vmd/ directory contains ablation.py (ablation), contribution.py (contribution), robustness.py (robustness), and metrics.py (metrics);
• Interactive Tools: The notebooks/ directory provides reproducible workflows, sample data, and visualizations, supporting Colab;
• Batch Processing: The scripts/ directory supports large-scale offline testing;
• Sample Data: The data/sample/ directory contains test samples.

The design is flexible, supporting integration with local/Colab VLM backends.

Application Scenarios:

1. Model development and debugging: Verify multimodal fusion, identify over-reliant or neglected modalities;
2. Model evaluation and comparison: Go beyond accuracy to identify "superficially high-performance" models;
3. Teaching and popular science: Demonstrate model "cheating" cases and explain multimodal concepts.

Usage Workflow:

1. Prepare pre-trained models and datasets;
2. Record baseline test performance;
3. Perform ablation experiments to remove each modality;
4. Analyze contribution of key samples;
5. Apply perturbations for robustness testing;
6. Generate visual reports.

The value of VMD lies in raising a methodological question in AI evaluation: how to correctly assess the capability of multimodal models? Traditional accuracy is prone to misjudgment; VMD advocates "white-box" diagnosis to deeply understand model mechanisms. In practical applications, it can guide model deployment (e.g., avoid using subtitle-dependent models in scenarios with poor subtitle quality).

Summary: VMD is an important self-examination tool for multimodal video understanding research. It reminds us that high accuracy does not equal true understanding, and it is recommended that researchers include it in their toolkits to guide model design directions.