Innovative Practice of Multimodal Deep Learning in Deepfake Detection: A Fusion Scheme of CNN and FFT Frequency Domain Features

This project introduces an innovative multi-modal Deepfake detection system combining CNN spatial features and FFT frequency domain features. It compares a baseline CNN model with an improved fusion model to demonstrate the value of frequency domain analysis in identifying forged images. The project is open-source, provides a clear experimental framework, and includes an interactive Streamlit demo for easy use.

With the rapid development of generative AI, Deepfake content has become a major challenge in the digital age, misused for misinformation, fraud, and privacy violations. Traditional image detection methods struggle with increasingly sophisticated forgeries that are visually close to real photos. Thus, researchers are exploring multi-modal approaches that extract features from multiple dimensions (like frequency domain) to capture subtle forgery traces.

Developed by Anindya1006 and hosted on GitHub, this open-source project's core innovation is a hybrid model fusing CNN spatial features and FFT frequency domain features. It implements two detection schemes: a baseline traditional CNN model and an improved multi-modal fusion model, allowing direct performance comparison on the same dataset to quantify the gain from frequency domain features.

Baseline CNN Model: Uses classic CNN architecture to extract spatial features via convolution and pooling. However, it may struggle with some Deepfakes as generated images are highly realistic in the spatial domain.

Multi-modal Fusion Model: Introduces FFT frequency domain features. FFT reveals periodic patterns and frequency distribution differences between real and fake images (e.g., abnormal high-frequency energy from upsampling/compression). The workflow: 1) Dual-branch feature extraction (CNN for spatial, FFT for frequency); 2) Feature fusion (concatenation, weighted sum, or attention); 3) Classification via fully connected layers.

The project uses a standard binary classification dataset (train/test sets with real/fake categories). Key metrics:

• Accuracy: Proportion of correctly classified images.
• F1 Score: Harmonic mean of precision and recall, balancing false positives and negatives, critical for Deepfake detection where both errors have severe consequences.

The project uses Python tools: PyTorch (deep learning framework), OpenCV (image preprocessing), NumPy (numerical computing), Scikit-learn (metrics), Matplotlib (visualization), and Streamlit (interactive web app). The Streamlit frontend allows users to upload images and compare real-time predictions from both models, making it easy to demonstrate and understand model behavior.

Significance: This framework is extensible to video/audio forgeries. It provides reproducible benchmarks, modular design for independent experiments, and a user-friendly demo.

Limitations & Future Work: 1) Small dataset size (needs larger, diverse datasets); 2) Adversarial robustness (evaluate against attacks);3) Real-time performance (optimize inference speed);4) Interpretability (improve frequency feature explainability).

Summary: This multi-modal solution shows promise in Deepfake detection, combining spatial and frequency features to enhance accuracy and robustness, playing a key role in maintaining digital content authenticity.