Hybrid Random Smoothing: Providing Joint Adversarial Robustness Certification for Multimodal Models

This paper proposes the Hybrid Random Smoothing Framework, the first random smoothing technique that can uniformly handle discrete-continuous hybrid inputs. Through Neyman-Pearson joint worst-case analysis, it provides model-agnostic joint adversarial robustness certification for multimodal safety filtering. This framework addresses the problem that traditional single-modal robustness methods cannot handle heterogeneous joint perturbations, unifies the classic methods of Gaussian (continuous) and discrete random smoothing, and provides theoretical guarantees for the safe deployment of multimodal AI systems.

With the rapid development of large multimodal models (such as GPT-4V, Claude 3, Gemini, etc.), AI systems can now understand multiple modal contents like text, images, and audio simultaneously, but this also introduces new security risks: adversarial attackers may perturb multiple input modalities at the same time (e.g., modifying image pixels and text tokens in image-text safety filtering). Traditional single-modal robustness certification methods cannot handle such heterogeneous joint perturbations—they only consider continuous inputs (e.g., images under Gaussian noise) or discrete inputs (e.g., text token replacement) and cannot address combined threats.

Random smoothing is a mainstream model-agnostic robustness certification technique, but existing methods face fundamental difficulties when dealing with hybrid modalities: the mathematical properties of continuous and discrete noise are different, making it hard to unify them into the same framework.

The core innovations of the framework include:

Theoretical Framework: Neyman-Pearson Analysis of Joint Worst-Case Scenarios

Modeling robustness certification under heterogeneous perturbations as a joint worst-case problem: the input contains continuous (e.g., image pixels) and discrete (e.g., text tokens) parts; attackers can perturb both simultaneously within budget constraints, and the goal is to prove that the model's prediction remains unchanged within the perturbation range. The researchers used an extended form of the Neyman-Pearson lemma to handle composite hypothesis testing under hybrid distributions. The key insight is: when continuous and discrete noises follow a factorized distribution (independent), the joint likelihood ranking can be decomposed into a combination of likelihoods from each modality, simplifying the multi-dimensional optimization into a one-dimensional problem.

Closed-Form Certification: Unified One-Dimensional Certificate for Two Classic Methods

A closed-form one-dimensional robustness certificate is derived:

• Degenerates to the classic Gaussian random smoothing certificate when only continuous inputs are present
• Degenerates to the classic discrete random smoothing certificate when only discrete inputs are present
• Provides a strict certification lower bound under joint perturbations for hybrid inputs

This unification shows that continuous and discrete smoothing are special cases of the same framework—only the hybrid certificate needs to be implemented to handle any single/multimodal scenario.

The framework's effectiveness was verified on the multimodal safety filtering task (judging whether an image-text combination is non-compliant). The challenges of this task include:

• Modal interaction dependency: Violation judgment depends on semantic association between images and text
• Adversarial vulnerability: Attackers can fine-tune images or rewrite text to evade detection
• Joint perturbation threat: Attacks that perturb both modalities simultaneously are the most dangerous

Experimental results show that the framework can provide model-agnostic Neyman-Pearson certification (a first in the field), specifically:

• Computes an explicit robust radius for image-text inputs
• Any joint perturbation (image pixel changes + text token replacement) within the radius does not change the safety judgment
• The certification is applicable to any base classifier

Theoretical Level: Fills the theoretical gap in robustness certification for heterogeneous inputs, proves that continuous and discrete input certification can be handled uniformly, and opens up new research directions. Practical Level: Provides provable guarantees for the safe deployment of multimodal systems; in high-risk scenarios (content moderation, medical diagnosis, autonomous driving), it can quantify the model's resistance to joint attacks. Method Level: The closed-form certificate is highly efficient with minimal overhead, making it suitable for online applications (superior to numerical optimization or Monte Carlo simulation methods).

Current limitations:

1. Assumes factorized (independent) noise across modalities; actual modalities may have correlations, so extending to handle such cases is an open problem.
2. Experiments focus on binary classification safety filtering; certification boundaries for multi-class scenarios need further research.

Future directions:

• Explore certification under complex modal interaction models such as attention mechanisms;
• Extend to more modalities like audio and video;
• Study the relationship between certification boundaries and model architecture features like Transformers.

The Hybrid Random Smoothing Framework, through Neyman-Pearson joint worst-case analysis, achieves unified robustness certification for discrete-continuous hybrid inputs for the first time, unifies classic Gaussian and discrete random smoothing methods, and provides theoretical guarantees for the safe deployment of multimodal AI systems. As multimodal models are increasingly applied in key fields, such provable safety technologies will become more important.