MSP Multimodal Speech Recognition: Fusing Audio and Lip Reading to Solve ASR Challenges in Noisy Environments

This article introduces the MSP (Multimodal Speech Perception) project, which builds a multimodal speech recognition system fusing audio and visual lip reading. It combines the Wav2Vec2 audio encoder and visual lip-reading encoder through a cross-attention mechanism, supporting three modes: audio-only, visual-only, and audio-visual fusion. The project aims to solve the problem of decreased automatic speech recognition (ASR) accuracy in noisy environments. Implemented based on Python 3.10+ and PyTorch 2.9, it has completed training and evaluation on the LRS2 dataset.

Although speech recognition technology has made significant progress, noise (such as street traffic sounds, café background noise, and multi-person conversations in meeting rooms) severely reduces ASR accuracy. Traditional solutions rely on microphone arrays or noise reduction algorithms, but their effectiveness is limited under extreme conditions. The human ability to understand conversations in noisy environments by listening to sounds and observing lip movements inspired the MSP project's idea of fusing audio and visual signals.

MSP includes three models: audio-only (MSP Audio, based on Wav2Vec2 + CTC loss), visual-only lip reading (MSP Visual, extracting lip region features), and multimodal fusion (MSP Model). The core innovation is the cross-attention design: audio embeddings serve as Query, while visual embeddings serve as Key/Value. Visual preprocessing flow: extract lip regions → crop to 96×96 RGB frames → normalization and augmentation; audio preprocessing: resample to 16kHz. The model uses a hybrid CNN+Transformer architecture and is trained end-to-end.

The project was trained and evaluated on the LRS2 dataset (45,814 training samples, 1,243 test samples, and 8 SNR noise variants). Technical advantages: modal complementarity (audio is sensitive to phonemes but susceptible to noise; visual is noise-resistant but struggles with some phonemes), flexible fusion strategy (dynamically adjust the weight of visual information), pre-training transfer (Wav2Vec2 pre-training reduces data requirements), and multilingual document support (English + Arabic).

Applicable scenarios: noisy environments (streets/factories), long-distance sound pickup, hearing assistance, video conferences, security monitoring. Limitations: reliance on front-facing perspective, occlusion (masks/beards) affects visual performance, high computational cost for video processing, scarcity of audio-visual annotated datasets, and privacy concerns related to the visual modality.

Future directions: larger-scale pre-training, lightweight architecture (mobile deployment), multilingual support, real-time streaming processing, attention visualization. Summary: MSP effectively fuses audio and video via cross-attention, providing a practical solution for noisy ASR. Open-source implementation and pre-trained models lower the entry barrier, making a positive contribution to the multimodal ASR ecosystem.