DualVision: A Multimodal Large Model Fusing Infrared and Visible Light for Robust Visual Reasoning in Adverse Weather Conditions

The University of Wisconsin-Madison and Amazon team propose DualVision, which injects infrared image information into multimodal large language models via a lightweight cross-modal fusion module. It achieves a 75% reduction in computational load and significant performance improvements in degraded scenarios like fog, low light, and blurriness, providing solutions for multiple scenarios such as autonomous driving and security monitoring.

Current multimodal large language models (MLLMs) perform well in tasks like image understanding and visual question answering, but these capabilities are mostly based on RGB visible light images. In the real world, cameras often face challenges such as haze, motion blur, and low light at night—under these degraded conditions, even top models like GPT-4V and Claude see a significant drop in recognition accuracy. Infrared imaging, as a mature complementary technology, can maintain stable imaging quality in the above adverse environments. However, how to efficiently integrate infrared information into large models has long been an open problem in academia and industry.

The core innovation of DualVision lies in the localized cross-modal attention mechanism:

1. Spatially Aligned Attention Design: Each RGB image patch only focuses on the infrared region at the corresponding spatial position, compressing the computational load to O(N²) and avoiding the sharp increase in computational complexity caused by simple concatenation;
2. Multi-Scale Progressive Fusion: Capture cross-modal features from small to large regions through a series of local attention layers with different receptive fields;
3. Plug-and-Play Compatibility: Can be directly inserted into existing MLLM architectures (e.g., LLaVA series) without large-scale modifications to the base model.

Supporting Datasets

The team open-sourced two datasets:

• DV-204K: Approximately 25,000 pairs of registered RGB-infrared images + 204,000 modality-aware QA annotations (for instruction fine-tuning);
• DV-500: 500 pairs of test images + 500 QA pairs (for evaluating robustness in degraded scenarios), with data sourced from LLVIP and HDRT datasets and supplemented with QA annotations.

Experimental Results

• Computational Efficiency: Achieves about a 75% reduction in computational load compared to the RGB-IR concatenation scheme;
• Robustness Improvement: Significantly outperforms open-source models like LLaVA 1.5 and Qwen2-VL, as well as closed-source APIs like Claude Sonnet 3.5v2 in degraded scenarios;
• Generalization Ability: Remains competitive under normal lighting conditions without sacrificing performance in regular scenarios.

The technical route of DualVision is applicable to multiple scenarios:

• Autonomous Driving: Perception enhancement in night driving and foggy environments;
• Security Monitoring: Target recognition and behavior analysis for all-weather video surveillance;
• Industrial Inspection: Equipment inspection and defect detection in low-light environments;
• Outdoor Robots: Navigation and obstacle avoidance under adverse weather conditions.

The project has been fully open-sourced, including fusion module code, pre-training and fine-tuning scripts, and the DV-204K and DV-500 datasets (QA annotations will be released within a few weeks). The paper has been accepted by CVPR 2026 Findings, and the project homepage and demo examples can be accessed at: abrarmajeedi.github.io/dualvision.

DualVision validates the technical route of "modal complementarity + efficient fusion". The lightweight solution has more practical value on edge devices with limited computing power, providing a reference paradigm that balances performance and efficiency for the deployment of multimodal large models.