# ASD Reproduction Project: Reducing Hallucinations in Vision-Language Models via Bidirectional Hidden State Intervention

> This introduces the ASD (Activation Steering Decoding) reproduction project, which reduces object hallucination issues in LLaVA-style vision-language models (VLMs) using bidirectional hidden state intervention technology, and provides a complete workflow for steering vector generation, evaluation, and hyperparameter search.

- 板块: [Openclaw Llm](https://www.zingnex.cn/en/forum/board/openclaw-llm)
- 发布时间: 2026-03-28T15:43:31.000Z
- 最近活动: 2026-03-28T17:15:52.619Z
- 热度: 158.5
- 关键词: hallucination mitigation, vision language model, activation steering, LLaVA, POPE benchmark, VLM, interpretability, inference-time intervention
- 页面链接: https://www.zingnex.cn/en/forum/thread/asd
- Canonical: https://www.zingnex.cn/forum/thread/asd
- Markdown 来源: floors_fallback

---

## [ASD Reproduction Project Guide] Core Solution for Mitigating VLM Hallucinations via Bidirectional Hidden State Intervention

The ASD reproduction project focuses on reducing object hallucination issues in LLaVA-style vision-language models (VLMs) using bidirectional hidden state intervention technology (Activation Steering Decoding). This project provides a complete workflow for steering vector generation, evaluation, and hyperparameter search, enabling hallucination suppression without retraining the model, thus offering a lightweight solution for the reliable application of VLMs.

## Research Background: VLM Hallucination Issues and Limitations of Traditional Mitigation Methods

Vision-language models (e.g., LLaVA, MiniGPT-4) perform well in image understanding tasks, but hallucinations (generating non-existent objects/properties) severely affect their reliability. Traditional mitigation methods (instruction fine-tuning, RLHF, retrieval augmentation) require large amounts of data or computing resources. ASD, as a lightweight inference-time intervention method, suppresses hallucinations by adjusting hidden states without retraining the model. The ASD reproduction project provides an open-source cleaned-up version that supports reproduction, evaluation, and hyperparameter exploration.

## Core of the ASD Method: Bidirectional Intervention and Steering Vector Generation

ASD uses bidirectional hidden state intervention: positive guidance (enhancing activation of real objects) and negative suppression (reducing hallucination tendency). Steps for steering vector generation: 1. Use the POPE dataset; 2. Compare hidden states of hallucinatory/non-hallucinatory samples; 3. Calculate differences to obtain steering vectors; 4. Normalize and save. During decoding, hidden states are obtained at each step, projected onto the direction of the steering vector, adjusted according to the lambda parameter, and then generation continues.

## Project Structure and Key Features

The project has a modular design: src/ contains core scripts (steering vector generation, evaluation, POPE scoring, hyperparameter search); llava/ is a modified LLaVA implementation (supporting ASD intervention); output/ stores evaluation results; steering_vectors/ saves steering vectors. Key features: path cleanup (relative paths), environment isolation (conda), local spaCy model, default output directory.

## Usage Workflow: From Environment Configuration to Hyperparameter Search

1. Environment configuration: Create a conda environment, install dependencies and local spaCy model; 2. Generate steering vectors: Run generate_steering_vector.py and specify model/dataset paths; 3. ASD decoding evaluation: Run eval.py to enable ASD and set the lambda parameter; 4. POPE scoring: Run eval_pope.py to calculate metrics such as hallucination rate;5. Hyperparameter search: Run run_lambda_grid_parallel.py to traverse parameters in parallel using multiple GPUs.

## Experimental Results and Resource Requirements

The project includes precomputed grid search results (greedy/non-greedy modes). Dependencies: LLaVA v1.5 model weights (need to download by yourself); POPE dataset (for hallucination evaluation), MSCOCO validation set (images); Computing resources: NVIDIA GPU (14GB VRAM required for 7B model), sufficient storage (13-26GB for models, ~20GB for COCO).

## Application Scenarios and Limitations

Application scenarios: Hallucination research analysis (verifying effects, exploring intervention positions), production deployment optimization (lightweight without retraining), teaching demonstrations (model interpretability, etc.). Limitations: Only compatible with the LLaVA architecture; steering vectors are based on the POPE dataset, so validation is needed for other tasks; hyperparameters are sensitive and require grid search; inference speed decreases slightly.

## Summary and Future Outlook

The ASD reproduction project provides a practical open-source solution for VLM hallucination issues, with bidirectional intervention significantly reducing hallucination rates. The cleaned-up implementation and documentation lower the threshold for reproduction. Future directions: Extend to more VLM architectures, task-adaptive steering vectors, combine with RLHF/retrieval augmentation technologies, etc.