Rethinking Jailbreak Detection for Large Vision-Language Models: The Representational Contrastive Scoring (RCS) Approach

This article introduces the Representational Contrastive Scoring (RCS) method proposed in an open-source ACL 2026 paper, addressing the problem of jailbreak attack detection for Large Vision-Language Models (LVLMs). It identifies malicious prompts by comparing the differences in model representations between normal inputs and jailbreak inputs. The open-source codebase of this method supports mainstream models such as LLaVA and Qwen-VL, aiming to improve detection accuracy and robustness, and promote multimodal AI security research.

Background

Large Vision-Language Models (e.g., GPT-4V, LLaVA, Qwen-VL) combine language and visual capabilities, but face the risk of jailbreak attacks—attackers use text-image combinations to induce models to generate harmful content, making the attack forms more complex.

Limitations of Existing Methods

• Output-based detection: Post-hoc detection, where harm has already occurred;
• Input pattern-based detection: Struggles to handle new attack techniques;
• Perplexity-based detection: Prone to false positives for normal complex queries;
• Representation-based detection: Lacks systematicity, making it hard to distinguish between normal complex inputs and malicious inputs.

The RCS method captures representation differences through contrastive learning, with core components including:

1. Contrastive Sample Construction: Generate normal versions, perturbed versions, and known jailbreak template samples of the input to be detected;
2. Multi-Layer Representation Extraction: Analyze representations from multiple hidden layers of the model to capture early attack impacts;
3. Contrastive Score Calculation: Compute the jailbreak score based on cosine similarity, cross-layer consistency, and deviation from the normal distribution;
4. Adaptive Threshold: Dynamically adjust the threshold to reduce misjudgments of complex queries.

Supported Models

The codebase supports mainstream LVLMs such as LLaVA-v1.6-Vicuna-7B, Qwen2.5-VL series, InternVL3-8B, and FLAVA.

Dataset

Uses JailbreakV-28k, custom text-image jailbreak samples, and normal queries.

Key Results

• Detection accuracy (AUC) is significantly higher than baselines like HiddenDetect;
• Strong cross-model generalization ability;
• Better robustness against adaptive attacks than rule-based methods;
• Optimizes computational efficiency via layer selection heuristics.

Technical Implementation

The open-source codebase includes:

• Core scripts: kcd.py (RCS implementation), mcd.py (contrastive variant), and baseline reproduction code;
• Auxiliary tools: feature extractor, caching mechanism, performance analysis tools;
• Experiment management: batch run scripts and visualization analysis code.

Deployment Considerations

• Integration strategy: Pre-filter + manual review;
• Performance trade-off: Layer selection heuristics balance accuracy and speed;
• Continuous update: Add new jailbreak samples to optimize the model;
• False positive handling: Combine manual review to reduce impact.

Methodological Contributions

1. Shift from output-based detection to representation-based detection, enabling preventive security;
2. Innovative application of contrastive learning in the security domain;
3. Systematic method for multi-level representation analysis;
4. Improve detection interpretability (visualized representation comparison).

Limitations and Future Directions

• Limitations: High computational cost, need to verify effectiveness against adaptive attacks, insufficient multimodal coverage;
• Future directions: Efficient representation extraction, integration of adversarial training, exploration of cross-modal contrastive learning, real-time threshold adjustment.

The RCS method represents an important progress in LVLMs security research. It redefines the jailbreak detection problem through representation-level analysis, and its open-source implementation provides a foundation for academia and industry. With the popularization of multimodal AI, this research is expected to promote further development of the field and is worthy of attention from security researchers and engineers.