Multimodal Fake News Detection: A Deep Learning Approach Integrating Text and Images

This project focuses on multimodal fake news detection, exploring the application of deep learning models such as BERT+ResNet, BERT+ViT, and CLIP based on the Fakeddit dataset. The core goal is to integrate text and image information to enhance detection robustness. Among them, the CLIPv2 variant achieved the best accuracy of 83.22% through data augmentation and phased fine-tuning. The project also provides a Streamlit demo application and discusses technical limitations and future directions.

Multimodal Challenges of Fake News

In the social media era, fake news is often spread in a multimodal form of text + images, making single-modal detection difficult to handle.

Fakeddit Dataset

This project uses the Fakeddit dataset, which collects posts with text titles and images from Reddit, labeled with 6 categories (true, satire, misleading, fabricated, etc.), covering the spectrum of misinformation and providing sufficient data support for model training.

The project explores three fusion strategies:

1. BERT+ResNet-50: A classic combination where BERT handles text semantics, ResNet extracts image features, and late fusion is applied; its advantages are stability and interpretability, but it may lack early modal interaction.
2. BERT+ViT: Replaces ResNet with Vision Transformer, using Transformer homogeneity to improve modal alignment; ViT is better at global semantic understanding but requires a large amount of data.
3. CLIP-based Approach: Uses OpenAI's CLIP model, which is based on cross-modal representations pre-trained via contrastive learning. CLIPv2 adapts to task requirements through targeted fine-tuning, making it the most innovative attempt.

Two key factors for CLIPv2's best performance:

• Data Augmentation: For text, synonym replacement and back-translation are used; for images, random cropping and color jitter are applied to enhance data diversity and model robustness.
• Phased Fine-Tuning: In the first phase, pre-trained parameters are frozen and only the classification head is trained; in the second phase, the underlying parameters are unfrozen and fine-tuned with a low learning rate to avoid catastrophic forgetting and retain general knowledge.

The performance gradient is obvious: CLIPv2 (83.22%) > BERT+ViT > BERT+ResNet.

• Value of Cross-Modal Pre-Training: CLIP's image-text association knowledge is crucial for understanding text-image relationships.
• Category Differences: Fabricated news has a high recall rate, while satire/misleading content is difficult to detect, reflecting the continuous spectrum nature of misinformation.

The project provides an interactive demo based on Streamlit, supporting users to input titles, upload images, and view prediction results and confidence in real time. This application facilitates research verification and result promotion, and non-technical users can also intuitively experience the detection effect.

Current methods have limitations:

1. Vulnerability to Adversarial Attacks: Easily deceived by maliciously constructed content.
2. Time Sensitivity: The form of fake news evolves over time, so models need continuous updates.
3. Multi-Dimensional Solutions: Technical detection is only part of the solution; it needs to be combined with media literacy education and platform governance policies for collaborative response.

This project demonstrates the technical level and limitations of multimodal fake news detection. CLIP's transfer ability indicates the potential of large-scale pre-trained cross-modal representations. An accuracy of 83.22% is a milestone, but combating misinformation requires joint efforts from technology, education, and policy. In the future, we look forward to more powerful multimodal models contributing to purifying the information environment.