Multimodal Disinformation Detection: From Benchmark Models to Transfer Learning Practices in the African Context

This project focuses on the challenges of transferring multimodal disinformation detection models from Western benchmark datasets to the African context, and significantly improves the model's recognition ability on African media content through localized data adaptation. The study adopts a technical approach of CLIP bimodal encoding + lightweight classifier to explore cross-domain generalization issues, and covers aspects such as data ethics and open-source contributions, providing practical references for AI fairness and inclusivity.

Disinformation detection is a hot direction in AI, but existing models are mostly built based on Western datasets such as Fakeddit and Twitter. Studies have found that these models may have biases in content from different regional and cultural backgrounds, especially when facing multimodal deception methods like 'old images with new narratives' (real images paired with distorted text), which require image-text consistency understanding capabilities. The Carnegie Mellon University team has noticed the performance bias of existing models on African media content.

The project models multimodal disinformation detection as an image-text semantic consistency problem:

1. CLIP Bimodal Encoding: Use CLIP ViT-B/32 to convert images and text into 512-dimensional semantic vectors;
2. Feature Engineering: Construct 1537-dimensional features (1 dimension for cosine similarity + 512 dimensions for absolute difference + 1024 dimensions for concatenation);
3. Lightweight Classifier: Adopt logistic regression, which has advantages of strong interpretability, low training cost, and deployment-friendliness.

The team built a localized dataset for the African context:

• Data Overview: 178 image-text pairs (81 fake, 97 real; 142 for training, 36 for testing);
• Collection Principles: Scene priority (public scenes), privacy protection, fact anchoring;
• Crowdsourced Annotation: Three annotators label independently, labels are determined by majority vote, and ambiguous samples are discussed collaboratively.

Four groups of comparative experiments show: The unadapted Fakeddit model has a recall rate of only 39.51% for disinformation on the African test set; after adding African training data, the recall rate increases to 66.67%, and the F1 score rises from 52.03% to 66.67%. Moreover, the accuracy of the adapted model on the Fakeddit test set increases from 84.73% to 90.78%, indicating that African data helps the model learn more robust cross-domain features.

The project provides a complete open-source implementation, including the main notebook (full process), Streamlit interactive application (supports interpretability such as prediction labels and risk probabilities), and pre-trained models. The Streamlit Community Cloud deployment file has been configured for one-click deployment.

The study has limitations: the African dataset is small (178 entries, 36 for testing), limiting statistical significance; CLIP as a fixed encoder is not fine-tuned for the task; the system outputs 'risk estimation' rather than 'fact-checking' and cannot replace manual review.

This study reveals an important issue in AI fairness: benchmark dataset performance ≠ real-world generalization ability. The success of African context adaptation shows that targeted localization efforts can improve cross-domain generalization. When AI technology penetrates the information ecosystem, fairness and inclusivity are mandatory.