Multimodal Automatic Annotation of E-commerce Products: Robustness Practice of CLIP Model in Product Attribute Prediction

This article introduces a CLIP-based multimodal deep learning project for automatically predicting attributes like category, color, gender, and season from product images and titles. Through a multi-task learning architecture and title-missing augmentation training, the project addresses the robustness issue when title information is incomplete in real e-commerce scenarios, achieving high prediction accuracy.

In e-commerce operations, manual product annotation has high costs and is prone to errors; automated annotation is key to efficiency improvement. However, real e-commerce data often has issues like missing titles or incomplete descriptions, requiring the system to maintain stable prediction capabilities even when information is missing—this is the core robustness requirement addressed by this project.

Based on the Kaggle Fashion Product Images dataset (about 44,000 products), each sample includes an image and a title, and needs to predict 4 attributes: category (20 classes), color (15 types), gender (5 classes), and season (4 seasons). Multi-task prediction aligns with real needs and reduces deployment and maintenance costs.

1. CLIP multi-task model: Use CLIP (openai/clip-vit-base-patch32) as the feature extractor, share the backbone network to extract joint image-text representations, and set an independent linear classification head for each attribute; the training strategy is freezing CLIP + training classification heads (end-to-end fine-tuning is possible). 2. Fusion model and ablation experiments: Implemented a DistilBERT + ResNet-50 fusion model, and verified the value of multimodal fusion through ablation experiments: the accuracy of the text-only model dropped from 97.5% to 2.8% when titles were missing, while the fusion model still maintained 88.6%, proving the key role of fusion in robustness.

Adopted the 'title dropout augmentation' training strategy—during training, empty the title with a certain probability to force the model to rely on image information. CLIP model evaluation shows: the average accuracy dropped from 92.2% to 81.9% when titles were missing, a loss of only about 10 percentage points, which meets real scenario requirements.

Provides a complete deployment solution: 1. Online demo: Gradio application on Hugging Face Spaces, allowing users to upload images to get prediction results; 2. Result display page: Visualizes model performance and examples; 3. Local running support: requirements.txt and scripts, supporting operation in Kaggle or local environments.

Core insights: 1. Multimodal pre-trained models (like CLIP) provide a strong feature foundation and reduce training costs; 2. Robustness training (like title dropout) is key to dealing with missing real data; 3. Ablation experiments quantify the value of multimodal fusion; 4. Multi-task learning improves efficiency. The project provides a reproducible and deployable technical solution for the intelligent transformation of e-commerce.