GaMMA: A Large Multimodal Model for Joint Global-Temporal Music Understanding

GaMMA is a state-of-the-art large multimodal model for music content understanding, which uses a Mixture-of-Experts (MoE) audio encoder to unify temporal and non-temporal music understanding tasks. Through a progressive training process and evaluation on the MusicBench benchmark, it achieves new highs with accuracy rates of 79.1% on MuchoMusic, 79.3% on MusicBench-Temporal, and 81.3% on MusicBench-Global, establishing a new SOTA in music understanding.

Music is a unique temporal art form that has both global structural features (style, genre, emotion, etc.) and fine-grained temporal features (melody, harmony, rhythm, etc.). Existing multimodal models struggle to balance these two aspects, while music AI has broad application prospects (recommendation, education, emotion computing, etc.). The emergence of GaMMA is precisely to address this challenge.

Architecture Design

• Inherits the LLaVA encoder-decoder architecture and extends it to the music-language domain
• Core innovation: Mixture-of-Experts (MoE) audio encoder that dynamically selects experts to handle temporal/non-temporal tasks

Training Process

1. Large-scale pre-training: Learn basic mappings on massive music-text data
2. Supervised Fine-tuning (SFT): Adapt to specific tasks on high-quality datasets
3. Reinforcement Learning (RL): Optimize output quality

MusicBench Benchmark

• 3739 manually created multiple-choice questions covering dimensions such as instrument recognition, style and emotion, harmony and melody
• Separates temporal and global evaluations

Experimental Results

Benchmark	Accuracy
MuchoMusic	79.1%
MusicBench-Temporal	79.3%
MusicBench-Global	81.3%

• Balances temporal and global capabilities, with significant scale effects (larger models perform better)

• Unified architecture is feasible: A single model can handle both temporal and non-temporal tasks simultaneously
• Data strategy: Large-scale pre-training + high-quality fine-tuning balances efficiency and capability
• Evaluation benchmarks are important: MusicBench fills the gap in comprehensive evaluation

• Music education: Intelligent teaching assistant (theoretical understanding, work analysis)
• Music recommendation: Content feature-based intelligent recommendation
• Creation assistance: Harmony suggestions, style analysis
• Accessibility: Provide music descriptions for visually impaired users

Current Limitations

1. Insufficient robustness to low-quality/noisy audio
2. Training data is biased towards Western music
3. Limited generation capabilities

Future Directions

• Multimodal expansion (integrating sheet music, lyrics, etc.)
• Real-time processing optimization
• Cross-cultural adaptation
• Joint modeling of generation and understanding