Listen, Look, Learn: SAM-Audio Empowers Audio-Visual Incremental Learning to Solve the Catastrophic Forgetting Problem

Core Insights

The study integrates the multimodal prior knowledge of SAM-Audio into audio-visual class-incremental learning, effectively solving the catastrophic forgetting problem and achieving state-of-the-art performance on multiple benchmarks through guided attention strategies and dual-layer distillation objectives.

Basic Information

• Original author team: arXiv paper author team
• Source platform: arXiv
• Original title: Listen, Look, and Learn: Learning Without Forgetting through SAM-Audio
• Publication time: June 9, 2026
• Original link: http://arxiv.org/abs/2606.10887v1

The core challenge of class-incremental learning (CIL) is to enable models to learn new categories without forgetting old knowledge. This has been extensively studied in single-modal scenarios, but the audio-visual multimodal scene remains relatively unexplored.

Unique characteristics of audio-visual incremental learning:

1. Dual-modal coupling: Need to maintain single-modal memory and cross-modal associations
2. Temporal dynamics: Audio and visual content have temporal characteristics, requiring consideration of knowledge retention in the time dimension
3. Scene complexity: Real-world scenarios (e.g., video understanding) are more complex than static image classification, leading to more severe forgetting issues

Guided Attention Strategy

Working Principle:

1. Extract audio features to capture sound events and temporal dynamics
2. Use audio features as queries to guide the attention allocation of visual features
3. Audio context determines the regions of focus in visual representations

Effectiveness: Modal complementarity, dynamic focusing, cross-modal reinforcement

Dual-Layer Distillation Objectives

Feature-level distillation: Maintain the similarity of feature spaces between old and new models, protect cross-modal association patterns, and prevent representation drift Logit-level distillation: Maintain consistency in output distribution and protect decision boundaries of learned categories Synergistic effect: The dual-layer design resists forgetting at multiple levels from internal representations to final outputs, outperforming single-layer distillation

Benchmark Datasets

• VGGSound (large-scale audio-visual dataset)
• FSD-Mix (incremental learning benchmark)
• Other audio-visual CIL benchmarks

Core Results

• Consistently outperforms existing SOTA
• Significantly improved retention rate of old categories
• No impact on learning speed of new categories
• Flatter overall performance curve (less forgetting)

Ablation Experiments

• SAM-Audio pre-training brings significant gains
• Performance drops明显 when audio guidance is removed
• Dual-layer distillation is better than single-layer

Qualitative Analysis

The guided attention mechanism can accurately locate visual regions based on audio, and this localization ability is preserved during incremental learning

1. Incremental adaptation of pre-trained models: Powerful pre-trained models (e.g., SAM-Audio) need to be transferred to incremental scenarios through adaptation strategies (guided attention + distillation)
2. Value of cross-modal attention: Audio-guided visual attention improves performance while enhancing robustness
3. Multi-level forgetting protection: Combating forgetting requires multi-faceted strategies; a single method is difficult to address complex challenges

Application scenarios of audio-visual incremental learning:

• Intelligent monitoring: Learn new abnormal sound-visual patterns while maintaining recognition of known threats
• Multimedia content management: Video platforms update classifiers to support new content
• Robot interaction: Learn new instruction-action associations without forgetting learned skills
• Assistive technology: Visual prompt systems for the hearing-impaired adapt to changes in users' personalized needs

Limitations

• Computational overhead: SAM-Audio's dense representations and attention mechanisms increase computational costs
• Long-term increment: Performs well in medium-length incremental sequences; verification needed for extremely long (hundreds of stages) sequences
• Modal imbalance: Handling scenarios with missing audio/visual information remains to be solved
• Generalization: Whether it applies to other pre-trained models (e.g., ImageBind) needs to be studied

Future Directions

• Develop lightweight variants of guided attention
• Explore combination of self-supervised pre-training and incremental learning
• Study dynamic network architectures to adapt to incremental scenarios
• Extend to more modalities (text, depth, etc.)

This study provides a strong baseline for audio-visual incremental learning, demonstrating the effectiveness of combining pre-trained models (SAM-Audio) with incremental technologies (guided attention, dual-layer distillation) to enable continuous learning while retaining knowledge.

Its significance lies not only in performance improvement but also in opening up new directions for multimodal incremental learning. As multimodal AI penetrates various fields, continuous learning without forgetting will become a key issue. The successful transfer of SAM-Audio also indicates that the 'transfer + adaptation' paradigm, where general representations of pre-trained models are adapted to new scenarios, may become the mainstream path for future AI development.