SARSteer: Protecting Large Audio Language Models via Safe Ablation and Refusal Steering

SARSteer Core Information

• Source: ICML 2026 accepted paper, published on arXiv in October 2025
• Position: First inference-time defense method for large audio language models (LALMs)
• Technologies: Text-derived refusal steering + safe subspace ablation
• Effectiveness: Effectively blocks harmful audio queries while avoiding over-refusal of normal queries
• Keywords: Audio language models, AI security, jailbreak attack defense, representation engineering

Original Authors and Sources

• Authors: Weilin Lin, Jianze Li, Hui Xiong, Li Liu
• Code link: https://github.com/linweiii/SARSteer
• Paper link: https://arxiv.org/abs/2510.17633

New Security Challenges of Audio Input

Large audio language models (LALMs) have become core components of multimodal AI, but audio input is more likely to induce harmful responses than pure text:

• Audio Jailbreak Attacks: Attackers bypass security protections using specific intonations, background noise, or acoustically processed speech, with a higher success rate than text jailbreaks
• Modality Uniqueness: The high dimensionality and continuity of audio signals provide more room for adversarial manipulation
• Limitations of Existing Technologies: Traditional safety alignment techniques do not fully address the unique challenges of the audio modality

Users expect safe and reliable voice interactions, but existing protection mechanisms struggle to address new threats in audio scenarios

Problems with Transferring Text/Visual Security Technologies

1. Activation Steering Failure:

   • In text models, refusal vectors are constructed by calculating activation differences between harmful queries and refusal responses
   • There are distribution differences between audio and text activations, so direct application results in technical failure

2. Over-refusal in Prompt-based Defense:

   • Explicitly refusing harmful questions via system prompts is effective in text models
   • Audio queries have high ambiguity (e.g., variations of the same content in different contexts), leading to many benign queries being incorrectly rejected

Existing methods fail to balance security and usability in audio scenarios

Technology 1: Text-derived Refusal Steering

• Core Insight: The model's high-level semantic processing mechanism is shared (similar representations of the "refusal" concept in audio and text)
• Steps:
  1. Calculate refusal vectors in text mode (by comparing activation differences between normal queries and those injected with refusal instructions)
  2. Overlay refusal vectors onto hidden states via forward hooks during audio inference

Technology 2: Decomposed Safe Space Ablation

• Core Idea: Refusal vectors only affect harmful queries and do not interfere with benign responses
• Steps:
  1. Collect benign audio queries and extract the safe subspace (principal components of benign activations) using SVD
  2. Ablate the projection component of the refusal vector in the safe subspace
  3. Hyperparameter control (lambda_: ablation coefficient; k_: subspace dimension)

The two technologies achieve a balance between security and usability

Experimental Setup

• Models: Qwen2-Audio, Kimi-Audio, Qwen-Audio, GPT-4o-audio
• Datasets: FigStep, AdvBench, SorryBench, AJailBench (security evaluation); AIR-Bench (benign evaluation)

Defense Effectiveness

• Harmful Query Blocking: Significantly reduces the attack success rate (ASR) and blocks most malicious audio inputs
• Benign Query Preservation: Normal task performance is roughly equivalent to the original model, without sacrificing core capabilities

Comparative Advantages

• Higher harmful query blocking rate compared to baseline methods
• Lower false positive rate for benign queries (safe subspace ablation mitigates over-refusal)

Theoretical Contributions

1. Cross-modal Representation Alignment: Proves that high-level semantic spaces can be leveraged across modalities, offering new insights for multimodal security research
2. Security-Usability Quantification: The concept of safe subspace provides an interpretable and quantifiable trade-off approach

Practical Value

1. Plug-and-Play: Lightweight inference-time method that requires no retraining and enables fast deployment
2. Strong Generalization: Applicable to LALMs of different architectures (Qwen/Kimi) and scales (7B parameters)
3. Enterprise-level Applications: Provides security guarantees for audio AI applications like voice assistants and intelligent customer service

SARSteer provides practical protection for current audio AI systems and lays the foundation for multimodal security research

Key Insights

1. Modality-specific Solutions: Directly transferring text technologies is not feasible; defenses must be designed for modality-specific characteristics
2. Value of Representation Engineering: Manipulating internal representations can achieve fine-grained behavior control; activation steering has great potential in multimodal scenarios
3. Dynamic Balance: Security and usability are eternal contradictions that require systematic solutions

Future Directions

• Extend to more modalities like video and haptics
• Automatically determine optimal hyperparameters
• Defend against adaptive attackers
• Application in distributed scenarios (e.g., federated learning)

SARSteer advances progress in the field of audio language model security and supports the safe deployment of AI technologies