AuRA: Internalize Audio Understanding Capabilities into LoRA, Enabling Large Language Models to Truly Understand Speech

Source Information

• Original Author Team: Paper author team (arXiv:2606.11033v1)
• Source Platform: arXiv
• Publication Date: June 9, 2026
• Original Link: http://arxiv.org/abs/2606.11033v1

Core Insights

AuRA transfers the audio understanding capabilities of ASR encoders to LoRA-adapted Large Language Models (LLMs) via knowledge distillation, enabling end-to-end speech understanding. This method significantly improves multimodal performance while maintaining efficient inference, and has advantages such as parameter efficiency and reuse of pre-trained assets.

Enabling LLMs to understand speech is key to natural interaction, but existing solutions have limitations:

1. Cascaded ASR-LLM Architecture: Speech is first transcribed into text before input to LLMs, leading to high latency and loss of paralinguistic information like prosody and emotion;
2. End-to-End Speech-Language Models: Require large-scale multimodal training, which is costly and makes it hard to reuse pre-trained achievements;
3. Bridging/Distillation Methods: Mostly serially coupled, limiting model expressive power.

Core Question: Can deep speech-language joint modeling be achieved with lightweight adaptation?

Core Innovations

Internalize audio encoding capabilities inside LLMs instead of external connection, using a teacher-student architecture:

• Teacher Network: Mature ASR encoder;
• Student Network: LoRA-adapted LLM (only a small number of parameters are trained);
• Lightweight Audio Embedding Layer: Maps speech features to the LLM input space. During training, layer-wise distillation aligns the hidden states of the teacher and student networks, enabling LLMs to learn to understand speech information.

Technical Advantages

1. End-to-End Parallel Inference: No need to wait for ASR transcription, significantly reducing latency;
2. Parameter Efficiency: Only trains less than 1% of the original model's parameters;
3. Reuse of Pre-trained Assets: Fully leverages pre-trained achievements of ASR and LLMs;
4. Deep Joint Modeling: Understands fine-grained speech features (e.g., prosody, emotion).

The paper validates the effect in multiple speech-language benchmark tests:

• Compared to cascaded systems: Outperforms in both effect and efficiency;
• Compared to speech-to-LLM adaptation baselines: Stronger representation learning ability;
• Compared to large-scale dedicated multimodal models: Still maintains competitiveness.

The results show that AuRA successfully balances efficiency and performance, providing a new paradigm for speech-enhanced LLMs.

Technical Implications

1. New Dimension of Knowledge Distillation: Can be used for cross-modal capability transfer (e.g., audio to language);
2. Expansion of LoRA Boundaries: From efficient fine-tuning to cross-modal internalization;
3. Importance of Representation Learning: Learning deep representations improves generalization ability.

Future Outlook

It is expected to expand to more modalities such as vision and touch, promoting the formation of multi-sensory unified agents.

Key Takeaways Summary

AuRA internalizes ASR capabilities via distillation, enabling end-to-end efficient inference, reusing pre-trained assets, leading across benchmark tests, and having broad application prospects.