LiRiS Chatbot: A Multimodal E-commerce Intelligent Customer Service System Integrating RAG and Computer Vision

LiRiS Chatbot is an open-source AI e-commerce customer service solution developed by V-Sreya, built with Python, FastAPI, and Streamlit. Its core features include integrating Retrieval-Augmented Generation (RAG) technology, large language models (LLaMA architecture), and computer vision capabilities to handle text and image multimodal queries, addressing issues like complex consultations, outdated knowledge, and multimodal needs in traditional customer service. The project is open-sourced on GitHub and was released on May 25, 2026.

The booming development of e-commerce has brought massive customer service demands. Traditional rule-based chatbots struggle to handle complex personalized consultations, while pure LLM solutions face issues like hallucinations and outdated knowledge. In e-commerce scenarios, users often ask questions based on product images (e.g., color, compatibility), requiring multimodal understanding capabilities. RAG technology combines external knowledge bases with LLMs to ensure accurate and timely responses, and integrating computer vision enables multimodal interaction—providing an effective way to address these problems.

Backend: FastAPI + Python

Built on FastAPI, it includes a document processing pipeline (converting documents into vector storage), a retrieval engine (semantic search with vector databases like FAISS/Chroma/Pinecone), an LLM service layer (encapsulating LLaMA model calls), and multimodal processing (integrating Hugging Face vision models to extract image features).

Frontend: Streamlit

Implements a dialogue interface, image upload preview, real-time response streaming display, and session history management—high development efficiency makes it suitable for rapid prototyping.

Vector Database Integration

Supports local (FAISS, Chroma) and managed (Pinecone, Weaviate) solutions, using Hugging Face pre-trained models to generate vector embeddings to ensure accurate semantic similarity calculation.

Intelligent Product Consultation

Answers product information through RAG semantic understanding, e.g., "How's the battery life of this phone?" retrieves battery capacity/reviews; "Which headphones are suitable for running?" recommends based on features.

Visual Question Answering

Upload product images to ask about compatibility, color, model, etc., e.g., "Can this accessory be used with [image]?"

Technical Support and Troubleshooting

Answers electronic product questions based on user manuals and FAQs, reducing manual workload.

Personalized Recommendations

Provides product recommendations based on historical conversations and current consultations to improve conversion rates.

Multimodal Fusion

Extends RAG to multimodal scenarios, aligns image features with text vectors, and achieves key associations for visual question answering.

Open-Source Ecosystem Integration

Uses mature components like Hugging Face Transformers, LangChain, FastAPI, and Streamlit to reduce development and maintenance costs.

Customizability

Clear structure with independent modules; allows replacing embedding models/LLMs, connecting to different vector databases, customizing prompt templates, and extending multimodal capabilities.

The project provides complete deployment documents and installation guides, supporting multiple modes:

• Local development: Functional testing and model tuning
• Docker containerization: Standardized deployment and environment isolation
• Cloud server deployment: Mainstream platforms like AWS, Azure, GCP

The modular architecture allows independent expansion of each component—e.g., scaling vector retrieval or LLM inference services without system reconstruction.

LiRiS Chatbot demonstrates a complete implementation of a RAG-based e-commerce customer service system, combining LLMs, vector retrieval, and computer vision to address complex customer service needs. It provides an excellent starting point for developers and enterprises, with a clear architecture and open-source features facilitating secondary development. In the future, multimodal large model technology will enhance visual understanding, knowledge retrieval, and dialogue experience, and the technical route explored by LiRiS aligns with the intelligent trend of e-commerce customer service.