Multimodal Image Retrieval: Comparative Study and Optimization of CLIP and BLIP on Flickr30K

This project focuses on the Flickr30K dataset, systematically compares the image-text retrieval performance of two representative multimodal models—CLIP and BLIP, conducts in-depth analysis of model failure cases and interpretability, and optimizes performance through fine-tuning strategies. The study covers dataset characteristics, model architecture differences, experimental design, key findings, and practical application value, providing reproducible benchmarks and insights for the multimodal retrieval field.

Project Background

Multimodal learning aims to break down the barriers between vision and language, and image-text retrieval is a core task: finding matching images given text, or finding appropriate descriptions given images. This project focuses on retrieval tasks on Flickr30K, compares the performance of CLIP and BLIP, and explores failure cases, interpretability, and fine-tuning optimization methods.

Flickr30K Dataset

• Overview: Contains 31,783 daily scene images, each paired with 5 manual English descriptions (158,000 in total), with rich language diversity.
• Characteristics: Diverse scenes (sports, social interactions, etc.), multi-angle descriptions (actions/scenes/character relationships), and high annotation quality.
• Task Settings: Image retrieval (text-to-image) and text retrieval (image-to-text).

CLIP (Contrastive Language-Image Pre-training)

• Architecture: Two-tower structure (image encoder + text encoder), mapping images and text to the same semantic space.
• Training Objective: Contrastive loss, maximizing the similarity of matched image-text pairs and minimizing that of mismatched pairs.
• Advantages & Disadvantages: Strong cross-modal alignment and good zero-shot transfer; however, limited understanding of fine-grained spatial relationships.

BLIP (Bootstrapping Language-Image Pre-training)

• Architecture: Multi-task framework (image encoder + text encoder + text decoder), supporting image-text matching and description generation.
• Training Objective: Image-text contrastive loss + image-text matching loss + language modeling loss.
• Advantages & Disadvantages: Capable of retrieval and generation, robust to noise; however, complex model structure and high training/inference costs.

Evaluation Metrics

Uses standard retrieval metrics: Recall@K (R@1/R@5/R@10), Median Rank, Mean Rank, R-Precision.

Failure Case Analysis

• Fine-grained understanding failure: Ignoring key details (actions/object relationships).
• Confusion of quantity and attributes: Inaccurate understanding of quantifiers (e.g., two) and attributes (e.g., red).
• Difficulty in coreference resolution: Confusing relationships between multiple objects.
• Abstract concept understanding: Limited handling of abstract content such as emotions/atmosphere.

Fine-tuning Methods

• Full fine-tuning: Updates all parameters, adapts to target distribution but has high cost and is prone to overfitting.
• LoRA fine-tuning: Trains only low-rank matrices, reducing the number of parameters.
• Prompt learning: Adds learnable prompt vectors to guide the model to adapt to tasks.
• Contrastive learning enhancement: Continues to use contrastive loss during fine-tuning to strengthen image-text alignment.

Performance-Cost Trade-offs

• Model scale: Compares the parameter count and performance relationship of different ViT variants (B/32, B/16, L/14).
• Training optimization: Early stopping strategy and learning rate scheduling to shorten training time.
• Inference efficiency: Evaluates model inference speed and memory usage to provide references for deployment.

Model Capability Comparison

• Retrieval performance: CLIP shows outstanding zero-shot performance, while BLIP is better after fine-tuning.
• Generation capability: BLIP generates more fluent and rich text descriptions.
• Robustness: BLIP is more robust to noisy data and distribution shifts.

Interpretability Analysis

• Attention visualization: Observes the image regions the model focuses on.
• Feature space analysis: Understands the distribution of image-text features in the joint space.
• Error clustering: Identifies systematic weaknesses of the model.

Practical Applications

• Search engines: Finding images via natural language descriptions.
• Recommendation systems: Precise personalized recommendations.
• Auxiliary tools: Image description for the visually impaired, semantic search for designers.
• Content moderation: Identifying inconsistent image-text content or harmful content.

Current Limitations

• Dataset size: Flickr30K is relatively small, limiting the model's capability.
• Language singularity: Only supports English, restricting application scenarios.
• Scene limitations: Mainly focuses on daily scenes; transferability to professional fields (medicine/satellite images) needs verification.

Future Directions

• Larger-scale data: Pre-training with large-scale web-crawled image-text pairs.
• Multilingual support: Exploring multilingual pre-trained models.
• Fine-grained understanding: Introducing object detection and scene graph generation to improve spatial relationship understanding.
• Efficient inference: Model quantization and knowledge distillation to reduce deployment costs.