Multimodal Book Recommendation Chatbot: Practice of Hybrid Architecture Fusing CNN and RNN

This project builds a multimodal book recommendation chatbot that innovatively integrates computer vision (CNN) and natural language processing (RNN) technologies. It uses CNN models like ResNet50 to process book cover images and RNN models such as BiLSTM to handle text descriptions, achieving more accurate and intelligent book recommendation services. The core lies in the effective fusion of multimodal information, solving the limitations of traditional single-modal recommendations.

Traditional book recommendation systems often rely on single-modal data (text or user ratings), while books contain rich multimodal information: cover images convey visual style, theme hints, and emotional tone; text such as book titles and introductions carries specific content descriptions. A single modality is insufficient to fully understand a book, hence the need for a multimodal fusion solution.

The image processing end uses three CNN models to extract features in parallel:

• ResNet50: Solves deep gradient vanishing through skip connections, learning complex visual patterns of covers (color, composition, texture);
• MobileNetV2: Lightweight design with depthwise separable convolution to reduce parameters and lower inference latency;
• EfficientNetB0: Compound scaling strategy balances efficiency and performance. Features from the three models are fused to form a comprehensive visual representation.

Text processing uses three bidirectional RNN variants:

• BiLSTM: Captures forward and backward dependencies in text, effectively understanding long-distance semantic associations;
• BiGRU: A simplified version of LSTM, merging states to reduce parameters and speed up training;
• BiLSTM+Attention: Introduces an attention mechanism to automatically focus on key parts of the text (keywords, emotional tendencies).

Multimodal fusion methods include:

• Early fusion: Concatenates image and text vectors at the feature level to form a joint representation;
• Late fusion: Makes predictions based on the two modalities separately and then integrates the decisions;
• Attention fusion: Cross-modal attention dynamically adjusts the weights of modalities. Compared to single-modal systems, this system supports scenarios such as image-based book search and text semantic recommendation.

1. Intelligent Customer Service: Deployed in e-commerce platforms, libraries, or reading apps, providing 24/7 intelligent consultation and supporting book search via photo upload or dialogue;
2. Cross-modal Retrieval: Supports "image-based book search", similar to song recognition by humming;
3. Personalized Recommendation: Analyzes users' historical behaviors (browsing covers, reading introductions) to achieve personalized recommendations (one size does not fit all), improving user stickiness and conversion rates.

1. Model Ensemble: Combining multiple heterogeneous models improves robustness and accuracy;
2. Balance Between Lightweight and High Performance: MobileNetV2 considers deployment scenarios, balancing precision and efficiency;
3. Attention Mechanism: BiLSTM+Attention enhances interpretability by pointing out text segments that influence recommendations;
4. End-to-End Architecture: Forms a closed loop from raw input to recommendation output, facilitating maintenance and iteration.

This open-source project demonstrates the practical application of multimodal deep learning in recommendation systems. It integrates the advantages of CNN and RNN to fully understand book content and provide a natural and intelligent interactive experience. It is a good reference case for developers learning about multimodal fusion and recommendation system architecture design.