SarcEmotiq: A Multimodal Audio Sarcasm Detection Deep Learning Tool

SarcEmotiq is a deep learning-based English audio sarcasm detection tool that integrates four modalities of information: acoustic, text, sentiment, and emotion. It achieves high-precision sarcasm recognition through a carefully designed attention mechanism. This article will introduce its background, technical methods, performance, usage, and application prospects.

Sarcasm is a subtle and hard-to-capture phenomenon in human language, where the literal meaning often deviates from the actual intent. It needs to be conveyed through multiple cues such as intonation, context, and emotional contrast. For AI systems to recognize sarcasm, they not only need to understand text content but also capture changes in sound prosody, emotional color, and subtle contradictions between modalities. SarcEmotiq is a multimodal deep learning tool developed specifically to address this challenge.

Four-Modality Fusion

SarcEmotiq integrates four complementary modalities:

• Acoustic modality: Uses openSMILE to extract ComParE_2016 features (prosodic information such as pitch, energy, and speech rate);
• Text modality: OpenAI Whisper transcription + BERT-base-uncased model to obtain text representations;
• Emotion modality: wav2vec2-large-xlsr model for speech emotion classification;
• Sentiment modality: RoBERTa (sentiment-roberta-large-english) for text sentiment analysis.

Attention Fusion Mechanism

• Contrastive attention: Uses emotion as the query and sentiment as key-value pairs to align and capture inconsistencies between emotion and sentiment;
• Cross attention: Uses text content as the query and acoustic features as key-value pairs to align and capture mismatches between semantics and prosody;
• Subsequently, masked average pooling is used to process variable-length sequences, and after concatenating all modality outputs, an MLP is used for classification.

SarcEmotiq is trained on the MUStARD++ open-source dataset (a multimodal sarcasm detection benchmark), focusing on extracting relevant information from the audio modality. The paper reports an F1 score of 74% on the benchmark data. Considering that sarcasm detection is an extremely challenging task in the NLP field (even human annotation consistency is not high), this performance is quite excellent.

Inference and Retraining

• Inference: A pre-trained model is provided. Command: python src/predict.py --input path/to/audio.wav --model path/to/model.pth. It automatically transcribes using Whisper, and the input must be in WAV format (1-20 seconds, 16kHz);
• Retraining: Requires an audio folder + CSV file (containing KEY and SENTENCE columns). Steps: Generate embeddings → Normalize → Train.

Gradio Demo

Launch command: python -m demo.app. It provides a user-friendly web interface where you can upload audio to view detection results, suitable for demonstration and quick testing.

SarcEmotiq has the following limitations:

1. Mainly trained for English; performance may be poor for other languages;
2. Training data comes from video dialogue scenarios; additional adaptation is needed for different domains (e.g., podcasts, customer service);
3. Sarcasm detection is affected by cultural background, personal style, and context dependence; some types of sarcasm may be poorly recognized.

SarcEmotiq provides a reference for multimodal emotion computing research, and its attention fusion architecture can be extended to other multimodal understanding tasks. At the application level, it can be integrated into scenarios such as customer service systems, social media monitoring, and content moderation to help AI understand users' true intentions and avoid inappropriate responses caused by misunderstanding sarcasm.

Conclusion

SarcEmotiq represents a solid contribution to the field of multimodal sarcasm detection. By integrating four modalities and attention mechanisms, it demonstrates the potential of AI to understand the subtleties of human language. With the development of multimodal large language models, such specialized tools will continue to play an important role.