RNN-based Image Caption Generation: Complete Implementation from CNN Feature Extraction to Recurrent Neural Network Decoding

This project is an image caption generation system implemented using PyTorch, combining ResNet50 feature extraction and RNN decoder, demonstrating a classic application of multimodal deep learning in the intersection of computer vision and natural language processing. The project originates from the practical assessment of the COMP5625M Deep Learning course at the University of Leeds, aiming to deeply understand the core technologies of multimodal data training through complete system construction.

Image Captioning is an important task in the intersection of computer vision and natural language processing. The core challenge is to enable machines to understand image content and describe it in natural language. Traditional image recognition only outputs category labels, while image captioning requires identifying objects, understanding relationships, actions, and scene contexts, and generating fluent sentences—requiring both visual feature extraction and language modeling capabilities. This project aims to master multimodal training techniques through practice.

The project uses a subset of the COCO dataset, containing about 5070 images, each with more than 5 descriptive texts. COCO is a benchmark dataset in the field of image captioning, covering 80 object categories and daily scenes. The descriptive texts are manually annotated, including prominent entities, activities, and scene information. The multi-annotation design provides rich supervision signals and also increases the generalization requirements for the model.

Encoder: ResNet50 Feature Extraction

A pre-trained ResNet50 is used as the encoder. It solves the gradient vanishing problem in deep networks through skip connections, extracts the output of the last fully connected layer as the image feature vector, and captures high-level semantic information (object categories, spatial layout, scene features).

Decoder: RNN Sequence Generation

The decoder receives image features, which are reduced in dimension via a linear layer and batch-normalized, then input into the RNN along with reference texts. The RNN models temporal dependencies through its recurrent structure and autoregressively generates grammatically correct and semantically coherent descriptive sentences.

Multimodal Feature Fusion

An early fusion strategy is adopted: image features are used as the initial hidden state of the RNN, and word embeddings are combined with image features at each step to achieve deep interaction between visual and language information.

Vocabulary Construction and Embedding Learning

Vocabulary is constructed by extracting words from training data. Each word is mapped to a dense vector of fixed dimension, and the embedding vectors are optimized along with model parameters to learn semantic relationships between words.

Loss Function and Optimization

Cross-entropy loss is used to measure the difference between predicted and real words. Padding and masking are used to handle sequences of different lengths, and the Adam optimizer (momentum + adaptive learning rate) is employed.

Utilization of Pre-trained Weights

ResNet50 is initialized with ImageNet pre-trained weights. Transfer learning accelerates convergence and improves generalization ability on small-scale datasets. The pre-trained model has already learned low-level (edges, textures) and high-level (object parts, structures) features.

Gradient Clipping and Regularization

Gradient clipping is implemented to prevent gradient explosion in RNN, and Dropout regularization is used in the decoder's linear layer to prevent overfitting.

Learning Rate Scheduling

A learning rate decay strategy is adopted: an initial high learning rate is used for fast convergence, and it is reduced later to find a better solution.

Application Scenarios

• Assistive Vision: Provide voice descriptions of images for visually impaired people;
• Content Management: Automatically generate image tags to improve retrieval capabilities;
• Social Media: Automatically generate caption suggestions for photos.

Expansion Directions

The encoder can be replaced with ViT/Swin Transformer; the decoder can be upgraded to LSTM/GRU/Transformer decoder; introducing attention mechanisms to improve visual-language alignment.

Summary

The project fully demonstrates the image caption generation process, covering data preprocessing, model construction, and training optimization, helping developers understand the collaborative principles of CNN and RNN and core technologies. Understanding the basic architecture is a necessary path to mastering cutting-edge multimodal large models (such as CLIP, GPT-4V), providing a solid starting point for learners.