ProtoAda: Prototype-Guided Adaptive Adapter Framework for Multimodal Continual Instruction Tuning

Core Insights: ProtoAda addresses the mismatch between task assignment and response format in multimodal large models during continual learning by introducing format-aware task prototypes and geometry-aware parameter integration mechanisms, significantly improving model performance in multi-task scenarios.

This framework targets the "format blindness" flaw of existing methods, routes tasks by considering both task semantics and output structure, and integrates parameters through geometric analysis to effectively mitigate catastrophic forgetting and inter-task interference.

Core Dilemmas of Continual Learning

Multimodal Large Language Models (MLLMs) need to continuously learn new capabilities, but traditional fine-tuning faces:

1. Catastrophic Forgetting: Learning new tasks leads to forgetting old knowledge
2. Inter-task Interference: Gradient conflicts between different tasks cause performance degradation

Flaws of Existing Methods

Hybrid models based on LoRA experts rely on image-text similarity for routing but ignore response format differences—tasks with similar semantics but different output formats (e.g., localization tasks and VQA) are incorrectly assigned to the same expert, leading to performance degradation.

Two core innovations of ProtoAda:

1. Format-aware Task Prototypes: When routing, consider both task semantics (image-text similarity) and output structure (response format features) to avoid sharing experts between tasks with conflicting formats.
2. Geometry-aware Parameter Integration: Identify updates compatible with existing parameters through geometric analysis to reuse existing knowledge and refine incrementally, avoiding problems caused by simple averaging or overwriting.

Prototype Learning Mechanism

Learn prototype vectors encoding semantic and format features for each task; during inference, calculate similarity between input samples and prototypes to dynamically route to the appropriate subset of experts.

Adaptive Adapter Expansion

When encountering new tasks, evaluate whether existing experts are applicable; if not, expand new expert modules.

Geometric Analysis for Parameter Updates

Analyze the geometric relationship between new task gradients and existing parameters, integrate only compatible updates, and isolate conflicting updates into a new parameter space.

Key Experimental Findings

1. Overall Performance Improvement: Outperforms existing methods on multiple MCIT benchmarks
2. Improvement on Format-sensitive Tasks: Significant improvement for tasks where output format is easily disrupted
3. Expert Collaboration Efficiency: Reduces negative transfer and enables more effective collaboration

Ablation Experiments

• Baseline performance drops significantly when only using semantic similarity, verifying the necessity of format-aware prototypes
• Simple parameter averaging strategy cannot achieve ProtoAda's performance, verifying the effectiveness of geometric integration

Theoretical Contributions

Reveals the problem of response format heterogeneity in multimodal continual learning, providing a new perspective for future research (need to consider both content differences and output structure diversity).

Practical Value

Provides a practical framework for developers of multimodal AI systems to improve multi-task performance with low computational overhead, suitable for scenarios like intelligent customer service and educational assistance.

Current Limitations

1. Prototype learning requires additional computational resources
2. Assumes clear task boundaries, but tasks in real scenarios may be ambiguous
3. Increasing number of tasks leads to growth in expert count, causing storage and inference overhead

Future Directions

1. Dynamic prototype compression to control the number of experts
2. Extend to online learning scenarios to handle streaming tasks
3. Explore applications across more modalities (audio, video)