FusionVLM: A Multimodal RAG-Based Visual-Language Model for Image Captioning

FusionVLM is an image captioning visual language model that combines multi-modal retrieval with a custom bidirectional fusion block architecture. Its core goal is to improve description quality, reduce hallucination, and enhance generalization by retrieving visually or semantically similar images and descriptions from datasets.

Basic Info:

• Author/Maintainer: Mahan-M47
• Source: GitHub (link: https://github.com/Mahan-M47/FusionVLM-Retrieval-Augmented-Image-Captioning)
• Release Time: 2026-05-28

Image captioning is a cross-field task between computer vision and natural language processing, aiming to generate natural language descriptions for images. Traditional visual language models (VLMs) rely solely on learned parameters for generation, leading to hallucinations (descriptions inconsistent with image content) and poor generalization to rare scenes or out-of-domain data.

FusionVLM addresses these issues by introducing the Retrieval-Augmented Generation (RAG) paradigm. Unlike pure parametric methods, it retrieves similar images and descriptions from datasets to support more accurate and reliable caption generation.

FusionVLM's architecture consists of 7 key components:

1. Retriever: Uses CLIP embedding cosine similarity to retrieve similar images/descriptions, with two FAISS databases (image embedding DB and description embedding DB).
2. CLIP Image Encoder: Uses openai/clip-vit-base-patch32 to encode query and retrieved images into visual embeddings.
3. T5 Text Encoder: Uses t5-base to encode top-k retrieved descriptions into context strings.
4. Input Projection Layer: Projects visual/text embeddings to 768-dimensional (fusion block dimension).
5. Fusion Blocks: 4 bidirectional cross-attention blocks (8 heads each) for iterative multi-modal fusion, including cross-attention (text→visual and visual→text), self-attention, and feed-forward networks with residual connections.
6. Text Encoder Projection: Projects fused text embeddings to decoder input space.
7. T5 Text Decoder: Uses t5-base with LoRA (low-rank adaptation) for parameter-efficient fine-tuning and autoregressive generation.

Datasets: Uses Flickr30k (31,783 images, 5 descriptions per image) for training/evaluation. Data splits: ~31k (train), 1k (val), 1k (test).

Retrieval Example: For an input image of "a person playing in the park", the retriever returns similar outdoor activity images and their descriptions to provide context.

Input/Output Flow: Each sample input includes query image + 1 retrieved image + top3 retrieved descriptions. Retrieval results are cached during training for efficiency and dynamically retrieved during inference.

Technical Features:

• RAG: Reduces hallucination by retrieving similar data for generation support.
• Bidirectional Cross-Attention: Enables symmetric information flow between visual and text modalities.
• Parameter-Efficient Fine-Tuning: Uses LoRA on T5 decoder (encoder frozen) to reduce trainable parameters while maintaining performance.
• Multi-Modal Retrieval: Uses CLIP embeddings for cross-modal similarity search.
• Comprehensive Evaluation: Supports BLEU-1/2/3/4, METEOR, ROUGE-L, CIDEr metrics.

Application Scenarios:

• Assistive vision for visually impaired users.
• Content moderation: Auto-generate descriptions to aid content review.
• Image retrieval enhancement: Improve text-based image search via accurate descriptions.
• Multi-modal research: Serves as a RAG-augmented benchmark architecture.

Implementation Details: Built with Python3.14+, PyTorch2.9+, and HuggingFace Transformers. Code structure includes config management, dataset processing, CLIP embedding wrapper, custom VLM implementation, FAISS retrieval module, training/evaluation functions, metrics, and inference tools.

Summary: FusionVLM demonstrates the application of RAG to visual language tasks. By combining CLIP's multi-modal embedding and T5's text generation capabilities with innovative bidirectional fusion blocks, it improves image caption quality and reliability while remaining parameter-efficient. It is a valuable open-source project for researchers and developers in multi-modal AI and RAG technologies.