Fact Recall Mechanisms in Speech Language Models: A Study on Differences Between Text and Speech Modalities

This study focuses on the fact recall mechanisms in Speech Language Models (SLMs), using causal mediation analysis to explore differences between text and speech modalities. The results show that there are significant differences in fact recall mechanisms between the two modalities, with only some mechanisms transferable from text to speech, providing theoretical guidance for the improvement of SLMs.

In recent years, multimodal language models (SLMs) such as SpiritLM have made progress, enabling cross-modal understanding and generation. However, a key question remains: Are knowledge representation and reasoning mechanisms consistent when switching between text and speech modalities? This relates to the model's interpretability, reliability, and safety.

Fact recall is a core capability of language models; in pure text models, 'knowledge neurons' have been discovered using causal mediation analysis. This study extends this method to SLMs, using SpiritLM as the target, comparing fact recall performance under text-to-text and speech-to-text settings to explore whether the mechanism applies to speech input.

Experimental results show: 1. The neuron activation patterns during speech input are significantly different from those of text, not a simple reuse of paths; 2. Some high-level semantic components are shared between the two modalities, reflecting modality-independent knowledge in cross-modal unified representations; 3. Inconsistent mechanisms are part of the reason for the decline in fact recall accuracy with speech input.

Reasons for the differences include: possible information loss or distortion when speech encoders convert to discrete tokens; longer speech token sequences affect the attention mechanism's capture of key knowledge signals. This suggests the need to optimize speech encoder quality, tokenization strategies, and modality alignment mechanisms.

1. Strengthen modality alignment learning to narrow the mechanism gap; 2. Improve the information retention and semantic alignment quality of speech encoders; 3. Develop fact consistency evaluation methods for the speech modality.

1. Explore cross-modal knowledge transfer training strategies (multi-stage, mixed training, alignment loss); 2. Design speech-specific knowledge injection mechanisms (using paralinguistic information such as prosody); 3. Extend interpretability tools to multimodal scenarios.