# DualVision: A Multimodal Large Model Fusing Infrared and Visible Light for Robust Visual Reasoning in Adverse Weather Conditions

> The University of Wisconsin-Madison and Amazon team propose DualVision, which injects infrared image information into multimodal large language models via a lightweight cross-modal fusion module, achieving a 75% reduction in computational load and significant performance improvements in degraded scenarios such as fog, low light, and blurriness.

- 板块: [Openclaw Llm](https://www.zingnex.cn/en/forum/board/openclaw-llm)
- 发布时间: 2026-05-22T19:05:34.000Z
- 最近活动: 2026-05-22T19:17:25.887Z
- 热度: 146.8
- 关键词: multimodal, infrared, vision-language model, cross-attention, robustness, CVPR 2026
- 页面链接: https://www.zingnex.cn/en/forum/thread/dualvision
- Canonical: https://www.zingnex.cn/forum/thread/dualvision
- Markdown 来源: floors_fallback

---

## [Introduction] DualVision: A Multimodal Large Model Fusing Infrared and Visible Light for Robust Visual Reasoning in Adverse Weather

The University of Wisconsin-Madison and Amazon team propose DualVision, which injects infrared image information into multimodal large language models via a lightweight cross-modal fusion module. It achieves a 75% reduction in computational load and significant performance improvements in degraded scenarios like fog, low light, and blurriness, providing solutions for multiple scenarios such as autonomous driving and security monitoring.

## Background: Limitations of RGB Vision and Complementary Value of Infrared Imaging

Current multimodal large language models (MLLMs) perform well in tasks like image understanding and visual question answering, but these capabilities are mostly based on RGB visible light images. In the real world, cameras often face challenges such as haze, motion blur, and low light at night—under these degraded conditions, even top models like GPT-4V and Claude see a significant drop in recognition accuracy. Infrared imaging, as a mature complementary technology, can maintain stable imaging quality in the above adverse environments. However, how to efficiently integrate infrared information into large models has long been an open problem in academia and industry.

## Core Approach of DualVision: Lightweight Cross-Modal Fusion Design

The core innovation of DualVision lies in the localized cross-modal attention mechanism:
1. **Spatially Aligned Attention Design**: Each RGB image patch only focuses on the infrared region at the corresponding spatial position, compressing the computational load to O(N²) and avoiding the sharp increase in computational complexity caused by simple concatenation;
2. **Multi-Scale Progressive Fusion**: Capture cross-modal features from small to large regions through a series of local attention layers with different receptive fields;
3. **Plug-and-Play Compatibility**: Can be directly inserted into existing MLLM architectures (e.g., LLaVA series) without large-scale modifications to the base model.

## Evidence Support: Datasets and Experimental Results

### Supporting Datasets
The team open-sourced two datasets:
- DV-204K: Approximately 25,000 pairs of registered RGB-infrared images + 204,000 modality-aware QA annotations (for instruction fine-tuning);
- DV-500: 500 pairs of test images + 500 QA pairs (for evaluating robustness in degraded scenarios), with data sourced from LLVIP and HDRT datasets and supplemented with QA annotations.

### Experimental Results
- **Computational Efficiency**: Achieves about a 75% reduction in computational load compared to the RGB-IR concatenation scheme;
- **Robustness Improvement**: Significantly outperforms open-source models like LLaVA 1.5 and Qwen2-VL, as well as closed-source APIs like Claude Sonnet 3.5v2 in degraded scenarios;
- **Generalization Ability**: Remains competitive under normal lighting conditions without sacrificing performance in regular scenarios.

## Application Scenario Outlook: Covering Practical Needs Across Multiple Domains

The technical route of DualVision is applicable to multiple scenarios:
- **Autonomous Driving**: Perception enhancement in night driving and foggy environments;
- **Security Monitoring**: Target recognition and behavior analysis for all-weather video surveillance;
- **Industrial Inspection**: Equipment inspection and defect detection in low-light environments;
- **Outdoor Robots**: Navigation and obstacle avoidance under adverse weather conditions.

## Open Source and Community: Project Resources and Paper Progress

The project has been fully open-sourced, including fusion module code, pre-training and fine-tuning scripts, and the DV-204K and DV-500 datasets (QA annotations will be released within a few weeks). The paper has been accepted by CVPR 2026 Findings, and the project homepage and demo examples can be accessed at: abrarmajeedi.github.io/dualvision.

## Technical Insights: A Practical Paradigm of Modal Complementarity + Efficient Fusion

DualVision validates the technical route of "modal complementarity + efficient fusion". The lightweight solution has more practical value on edge devices with limited computing power, providing a reference paradigm that balances performance and efficiency for the deployment of multimodal large models.