Panoramic Research on Trustworthy Audio Large Language Models: A Systematic Literature Review

This article is a systematic literature review that introduces the research literature on trustworthiness in the field of audio large language models compiled by the Awesome-Trustworthy-AudioLLMs project. It covers five core dimensions: safety, robustness, fairness, interpretability, and privacy protection, providing resource navigation for researchers and developers, and emphasizing the importance and research value of trustworthiness for Audio LLMs.

Audio LLMs have permeated daily life scenarios (smart assistants, medical diagnosis, autonomous driving, etc.) and face more complex trust challenges:

1. Broad multimodal attack surface: Audio can be tampered with in ways imperceptible to humans (e.g., adversarial examples), leading to model misjudgments;
2. Real-time nature compresses safety buffers: Low-latency streaming interactions cause erroneous outputs to be conveyed instantly;
3. Physical world anchoring: Errors may lead to physical harm or legal risks. The Awesome-Trustworthy-AudioLLMs project provides a knowledge infrastructure for this purpose.

1. Safety

Focuses on harmful outputs under malicious inducement, including adversarial attacks, jailbreak attacks, data poisoning, and needs to address the impact of real acoustic environments (reverberation, noise, etc.) on attacks.

2. Robustness

Measures stability when input distribution shifts, needing to handle variations such as accents/dialects, acoustic environments, device differences, age and health conditions.

3. Fairness

Examines systemic biases, such as dialect discrimination, gender bias, cultural differences, which stem from unbalanced data or stereotype associations.

4. Interpretability

Answers the reasons for decisions, including attention visualization, concept activation vectors, counterfactual explanations, to meet regulatory compliance and troubleshooting needs.

5. Privacy Protection

Protects sensitive audio data, involving membership inference, attribute inference, model inversion attacks, and defense methods like federated learning and differential privacy.

• Adversarial attack defense: Input transformation (audio compression, time-domain smoothing), adversarial training, certification-based defense;
• Fairness improvement: Data rebalancing, adversarial debiasing, post-hoc calibration;
• Privacy protection: Differential Privacy Stochastic Gradient Descent (DP-SGD), which needs to address the problem of maintaining performance under the high-dimensional characteristics of audio.

1. From single-task to multi-task: Shifting from single-task trustworthiness to overall trustworthiness of multimodal large models;
2. From offline to online: Static evaluation to real-time protection of streaming audio;
3. From general to specific domains: Increased dedicated research in high-risk scenarios such as medical, judicial, and automotive;
4. From technical indicators to social context: Focus on the social construct of trustworthiness (e.g., definitions of safety and fairness vary by culture).

1. Prioritize threat modeling: Identify attack surfaces and failure modes during the design phase;
2. Normalize red team testing: Conduct continuous adversarial testing;
3. Monitoring and rollback mechanisms: Monitor outputs after deployment and establish rapid rollback capabilities;
4. Transparent reporting: Explain capability boundaries, limitations, and safety recommendations to users.

The Awesome-Trustworthy-AudioLLMs project not only compiles literature but also establishes a common discourse framework to promote dialogue among researchers. The improvement of Audio LLM capabilities needs to be balanced by trustworthiness; building trustworthy AI requires collective and sustained investment to avoid sacrificing safety, fairness, and privacy.