VCP-Attack: A New Transferable Targeted Attack Method Against Large Vision-Language Models

This article introduces VCP-Attack—a new transferable targeted attack method against large vision-language models (LVLMs) using visual contrastive projection technology—and discusses its technical principles, attack mechanisms, and implications for the security of multimodal AI systems. This method aims to address security challenges faced by LVLMs, such as cross-modal attacks and adversarial sample threats, and features high attack success rate, good transferability, and stealthiness, providing an important reference for multimodal AI security assessment and defense.

Large vision-language models (LVLMs) can understand both images and text simultaneously and perform well in tasks like image captioning and visual question answering, but their security risks are increasingly prominent:

1. Cross-modal attack surface: Manipulating visual inputs to influence text outputs is harder to detect than pure text attacks;
2. Adversarial sample threat: Minor image perturbations can cause the model to produce incorrect outputs;
3. Targeted attack risk: Attackers can precisely control the model to output specific target text, which may be used to generate false or harmful content. VCP-Attack is a new targeted attack method against LVLMs proposed in this context.

VCP-Attack stands for Visual-Contrastive Projection Attack. Its core idea is to use the principle of contrastive learning to construct a projection direction in the visual feature space, so that the model produces a preset target output for the attacked image. The attack flow includes:

1. Target text encoding: Encode the desired target text into a feature vector;
2. Visual feature analysis: Understand the image-feature mapping of the LVLM's visual encoder;
3. Contrastive projection construction: Calculate the difference direction between the target text feature and the original image feature, and construct a projection matrix;
4. Perturbation optimization: Find the minimal image perturbation such that the perturbed image feature meets the attack target;
5. Transferability enhancement: Jointly optimize multiple models/layers to improve cross-architecture transferability.

An important feature of VCP-Attack is transferability—attack samples optimized on one model can successfully attack LVLMs of other architectures. Its sources include:

1. Shared visual representation: Different LVLMs often use similar visual encoders (e.g., CLIP's visual branch);
2. Similarity of alignment mechanisms: Visual-language alignment training objectives enable different models to learn similar cross-modal mappings;
3. Commonality of adversarial samples: Deep learning models have common weaknesses that adversarial samples can exploit. This method further enhances transferability by optimizing the projection direction, allowing attacks without knowing the specific architecture of the target model.

VCP-Attack was tested on mainstream LVLMs such as LLaVA, MiniGPT-4, and InstructBLIP, and the results show:

1. High attack success rate: The success rate in targeted attack scenarios is significantly higher than traditional adversarial attacks;
2. Good transferability: Samples optimized on the source model maintain a high success rate on unseen target models;
3. Stealthiness: Adversarial samples are visually indistinguishable from the original images;
4. Cross-task effectiveness: Effective in tasks like visual question answering, image captioning, and image-text matching.

Defense directions against VCP-Attack include:

1. Input purification: Detect and purify images before input (e.g., transformation, denoising, compression);
2. Feature space monitoring: Add anomaly detection at the output layer of the visual encoder;
3. Adversarial training: Introduce adversarial samples during training to enhance robustness;
4. Multi-model integration: Use multiple visual encoders for cross-validation;
5. Output review: Perform post-processing review on text outputs.

Research Significance:

• Provides a new testing method for LVLM security assessment;
• Promotes research on the robustness of multimodal models;
• Reveals potential weaknesses in the cross-modal alignment mechanism of LVLMs. Ethical Considerations:
• Emphasizes defensive purposes and explains the limitations of the method;
• Provides defense suggestions to protect systems;
• Follows the principle of responsible disclosure, giving time for fixes before public release.