Visual Question Answering for Smartphone Photo Albums: New Challenges for Multimodal AI in Understanding Daily Photos

This article introduces the AI challenge problem of the DACON 2025 Samsung Collegiate Programming Contest, which aims to develop multimodal AI models capable of understanding daily photos in smartphone users' albums and explore the application of Visual Question Answering (VQA) in real-world scenarios. This task combines computer vision and natural language processing, facing unique challenges brought by real users' photos, and has wide practical application value.

DACON is a well-known data science competition platform in South Korea, similar to Kaggle; Samsung Electronics hosts technical challenges through this platform to discover talents. The Samsung Collegiate Programming Challenge (SCPC) is a series of programming contests for college students organized by Samsung. The 2025 AI Challenge focuses on the task of visual question answering for smartphone photo albums, aiming to enable AI to understand real users' daily photos.

Unlike traditional VQA datasets (e.g., VQA v2), album VQA faces the following challenges:

1. Diverse image quality: Uneven quality due to light, angle, device differences, post-processing, and compression loss;
2. Wide range of content types: Covers various types such as people, scenes, objects, activities, screenshots, etc.;
3. Diverse question types: Includes existence, counting, attributes, relationships, reasoning, time, etc., requiring common sense reasoning;
4. Privacy and ethical considerations: Involves privacy issues such as face recognition, location information, and sensitive content, requiring desensitization processing.

For the album VQA task, possible technical routes include:

• CLIP-style alignment models: Use CLIP image and text encoders to extract features, select answers through similarity matching; strong zero-shot capability but limited fine-grained spatial understanding;
• Transformer fusion architectures: ViT/CNN extract visual features, BERT-like models encode questions, cross-attention for fusion; representative models include ViLT, VL-BERT;
• Large-scale pre-trained models: Fine-tune open-source models such as GPT-4V and Gemini, combined with prompt engineering or retrieval augmentation. Key technical challenges include fine-grained localization (object detection + referring expression understanding), multi-image reasoning, and OCR integration (processing photos with text).

Data composition speculation: The training set contains tens of thousands of (image, question, answer) triples; the validation set is used for parameter tuning; the test set answers are not public; images are from simulated or desensitized real albums. Evaluation metrics:

1. Accuracy: The proportion of predictions that exactly match the standard answers;
2. Tolerant matching: Word-level matching or semantic similarity measurement;
3. WUPS: A semantic similarity metric based on WordNet;
4. Analysis by question type: Report accuracy for yes/no, numerical, and open-ended questions separately.

Application scenarios of album VQA technology include:

1. Smart album search: Natural language search for photos (e.g., "Find photos of the beach last year");
2. Automatic photo organization: Classify important moments, detect duplicate or blurry photos, generate title descriptions;
3. Assisting visually impaired users: Describe photo content or answer specific questions;
4. Content moderation: Detect sensitive content (ID cards, bank cards) and prompt privacy risks.

Cutting-edge technical trends:

1. Explosion of multimodal large models: Models like GPT-4V, Gemini, and Qwen-VL show strong zero-shot VQA capabilities;
2. Edge deployment requirements: Mobile terminals need model compression (quantization, pruning), efficient inference (mobile NPU), and privacy protection (local inference);
3. Personalization and context: Future AI will understand user relationships, timelines, and emotions to achieve more personalized album understanding.

Contestants can refer to the following strategies:

1. Data exploration: Analyze question distribution, answer distribution, and error cases;
2. Model selection: Use pre-trained models, multi-model integration, and answer post-processing;
3. Iterative optimization: Monitor the validation set to avoid overfitting, conduct ablation experiments to understand component contributions, and improve based on error cases.