CausalLens: Sensitivity-Guided Multi-Head Causal Intervention for Eliminating Hallucinations in Vision-Language Models

A training-free method accepted by CVPR 2026 that significantly reduces object hallucinations in large vision-language models without retraining the model, using sensitivity-guided multi-head causal intervention technology.

• Original Authors/Maintainers: Junyang Ji, Qifan Liu, Wenming Yang, Zhihai He
• Source Platform: GitHub
• Original Title: CausalLens: Sensitivity-Guided Multi-Head Causal Intervention for Hallucination Mitigation in Large Vision-Language Models
• Original Link: https://github.com/jijy20/CausalLens
• Paper Link: https://openaccess.thecvf.com/content/CVPR2026/papers/Ji_CausalLens_Sensitivity-Guided_Multi-Head_Causal_Intervention_for_Hallucination_Mitigation_in_Large_CVPR_2026_paper.pdf
• Source Publication Time: June 2026

Large Vision-Language Models (LVLMs) have demonstrated strong capabilities in tasks like image understanding and visual question answering, but a long-standing problem plaguing researchers and application developers is Object Hallucination—the model generates text describing objects that do not exist in the image. This hallucination not only degrades user experience but also poses serious risks in critical application scenarios such as medical image analysis and autonomous driving.

Traditional hallucination mitigation methods mostly rely on Contrastive Decoding techniques, such as VCD (Visual Contrastive Decoding), which guides the model to generate more accurate descriptions by introducing noisy images as contrasts. However, these methods often only focus on surface statistical correlations and do not deeply explore the causal relationship between visual representations and text generation.

CausalLens proposes a new approach: understanding and intervening in hallucinations in vision-language models from the perspective of causal inference. The core hypothesis of this method is that hallucinations are not random but are caused by the incorrect sensitivity of specific attention heads to visual information. By identifying these "sensitive heads" and performing targeted causal interventions, hallucinations can be significantly reduced without changing the model parameters.

Compared with existing methods, CausalLens's unique features are:

1. Explicitly modeling causal relationships: Unlike contrastive decoding which only focuses on statistical differences between input and output, CausalLens delves into the model's internal structure to analyze how visual representations causally affect text generation.
2. Training-Free: No need to fine-tune model parameters; directly intervene in the inference process, greatly reducing deployment costs.
3. Multi-Head Collaborative Intervention: Instead of adjusting individual attention heads in isolation, coordinate interventions across multiple layers.

The first step of CausalLens is to identify which attention heads are most sensitive to hallucinations. The research team found that in the multi-layer attention mechanism of LVLMs, different attention heads have significantly different response patterns to visual information. Some heads are more likely to "invent" object information when there is no clear visual evidence.

By calculating the sensitivity gradient of attention weights to visual inputs, CausalLens can quantify the hallucination tendency of each attention head and select the target heads that need intervention the most.

After identifying sensitive heads, CausalLens adopts a three-layer intervention mechanism:

Sensitivity-Guided Intervention: Based on sensitivity scores, directionally adjust the output of high-risk attention heads to reduce their activation intensity when there is insufficient visual evidence.

Multi-Head Causal Intervention: Hallucinations are often the result of the combined action of multi-layer attention networks. CausalLens synchronously intervenes within a specified layer range (e.g., layers 10 to 20) to ensure that the intervention effect propagates deep into the model.

Adaptive Mixing Strategy: Completely replacing attention output may lead to information loss. CausalLens finds the optimal balance between the original representation and the intervened representation through an adjustable mixing parameter (gamma_mix).

Parameter	Description	Recommended Range
lambda_causal	Causal intervention intensity	0.1-0.3
gamma_mix	Mixing ratio between residual and replacement	0.1-0.2
layer_start / layer_end	Layer range for intervention	5-25
sys_len	Number of system tokens	30-40
img_len	Number of image tokens	576 (LLaVA)

CausalLens achieved state-of-the-art performance on the POPE (Polling-based Object Probing Evaluation) benchmark. POPE is a standard benchmark for evaluating hallucination problems in vision-language models, which tests whether the model incorrectly confirms non-existent objects through adversarial question-answer pairs.

More importantly, CausalLens demonstrates excellent architectural generalization ability:

• LLaVA Series: Validated to be effective on mainstream architectures like LLaVA-1.5
• Qwen2-VL: Also applicable to Alibaba's Qwen2-VL model
• Plug-and-Play: Can be integrated into existing inference processes with just a few lines of code

Experiments show that CausalLens not only performs well on POPE but also significantly reduces the hallucination rate while maintaining the model's original capabilities, with minimal impact on the performance of normal visual understanding tasks.