White-box Adversarial Attacks Reveal Social Bias Vulnerabilities in Large Multimodal Models

This article introduces a white-box adversarial attack study targeting social bias issues in Large Multimodal Models (LMMs). The project provides complete code implementations, including targeted PGD attacks, universal adversarial perturbations, defense evaluation, and noise similarity analysis, serving as an important tool for AI safety research.

• Original Author/Maintainer: vicentiu25
• Source Platform: GitHub
• Original Title: Adversarial-Attacks-Social-Bias-LMMs
• Original Link: https://github.com/vicentiu25/Adversarial-Attacks-Social-Bias-LMMs
• Publication Date: July 12, 2026

With the widespread application of Large Multimodal Models (LMMs) in tasks such as visual question answering and image caption generation, the social bias issues carried by these models have received increasing attention. Social bias may lead to unfair and discriminatory outputs from models on issues related to specific groups, posing severe challenges to the credibility and safety of AI systems.

Traditional bias detection methods often rely on black-box testing or manual auditing, making it difficult to deeply understand the internal mechanisms of bias formation in models. The emergence of white-box adversarial attack methods provides researchers with a new perspective—by directly accessing model parameters and gradient information, it is possible to more accurately locate and manipulate bias representations in models.

This project is an open-source research tool library focusing on white-box adversarial attack research targeting social bias in large multimodal models. The project provides complete experimental code, supporting attack testing on two mainstream multimodal models (BLIP-2 and LLaVA), and uses the SB-Bench dataset for systematic evaluation.

The core contribution of the project lies in combining adversarial attack technology with social bias research—it not only provides attack implementations but also includes defense evaluation and noise analysis tools, forming a complete research loop.

The project implements a variant of the Projected Gradient Descent (PGD) attack, specifically optimized for social bias. PGD is an iterative adversarial sample generation method that performs small-step updates along the gradient direction of the loss function in the input space while limiting perturbations within a preset epsilon range.

The project provides two attack modes:

• Suppress Attack: Aims to reduce the bias expression of the model on bias-related samples
• Induce Attack: Used to test the sensitivity of the model's bias expression under specific conditions

In addition to sample-specific attacks, the project also implements training code for Universal Adversarial Perturbations. The characteristic of universal perturbations is that they can produce consistent attack effects across different input samples, which is of great significance for understanding the global bias representations in models.

The project includes evaluation code for various common defense mechanisms, including:

• JPEG compression defense
• Image blurring
• Random cropping

By comparing the success rate of attacks with and without defense, researchers can evaluate the effectiveness of different defense strategies.

The project provides a cross-image perturbation similarity analysis tool to study the correlation between adversarial noises generated on different samples. This analysis helps reveal the shared structure of bias representations in models.