ASD Reproduction Project: Reducing Hallucinations in Vision-Language Models via Bidirectional Hidden State Intervention

The ASD reproduction project focuses on reducing object hallucination issues in LLaVA-style vision-language models (VLMs) using bidirectional hidden state intervention technology (Activation Steering Decoding). This project provides a complete workflow for steering vector generation, evaluation, and hyperparameter search, enabling hallucination suppression without retraining the model, thus offering a lightweight solution for the reliable application of VLMs.

Vision-language models (e.g., LLaVA, MiniGPT-4) perform well in image understanding tasks, but hallucinations (generating non-existent objects/properties) severely affect their reliability. Traditional mitigation methods (instruction fine-tuning, RLHF, retrieval augmentation) require large amounts of data or computing resources. ASD, as a lightweight inference-time intervention method, suppresses hallucinations by adjusting hidden states without retraining the model. The ASD reproduction project provides an open-source cleaned-up version that supports reproduction, evaluation, and hyperparameter exploration.

ASD uses bidirectional hidden state intervention: positive guidance (enhancing activation of real objects) and negative suppression (reducing hallucination tendency). Steps for steering vector generation: 1. Use the POPE dataset; 2. Compare hidden states of hallucinatory/non-hallucinatory samples; 3. Calculate differences to obtain steering vectors; 4. Normalize and save. During decoding, hidden states are obtained at each step, projected onto the direction of the steering vector, adjusted according to the lambda parameter, and then generation continues.

The project has a modular design: src/ contains core scripts (steering vector generation, evaluation, POPE scoring, hyperparameter search); llava/ is a modified LLaVA implementation (supporting ASD intervention); output/ stores evaluation results; steering_vectors/ saves steering vectors. Key features: path cleanup (relative paths), environment isolation (conda), local spaCy model, default output directory.

1. Environment configuration: Create a conda environment, install dependencies and local spaCy model; 2. Generate steering vectors: Run generate_steering_vector.py and specify model/dataset paths; 3. ASD decoding evaluation: Run eval.py to enable ASD and set the lambda parameter; 4. POPE scoring: Run eval_pope.py to calculate metrics such as hallucination rate;5. Hyperparameter search: Run run_lambda_grid_parallel.py to traverse parameters in parallel using multiple GPUs.

The project includes precomputed grid search results (greedy/non-greedy modes). Dependencies: LLaVA v1.5 model weights (need to download by yourself); POPE dataset (for hallucination evaluation), MSCOCO validation set (images); Computing resources: NVIDIA GPU (14GB VRAM required for 7B model), sufficient storage (13-26GB for models, ~20GB for COCO).

Application scenarios: Hallucination research analysis (verifying effects, exploring intervention positions), production deployment optimization (lightweight without retraining), teaching demonstrations (model interpretability, etc.). Limitations: Only compatible with the LLaVA architecture; steering vectors are based on the POPE dataset, so validation is needed for other tasks; hyperparameters are sensitive and require grid search; inference speed decreases slightly.

The ASD reproduction project provides a practical open-source solution for VLM hallucination issues, with bidirectional intervention significantly reducing hallucination rates. The cleaned-up implementation and documentation lower the threshold for reproduction. Future directions: Extend to more VLM architectures, task-adaptive steering vectors, combine with RLHF/retrieval augmentation technologies, etc.