Google Gemini Embedding 2 Multimodal RAG Framework: A Retrieval-Augmented Generation Solution for Unified Processing of Text, Images, Videos, and Audio

This article introduces an open-source multimodal RAG framework based on Google Gemini Embedding 2, which can uniformly handle embedding and retrieval of four media types: text, images, videos, and audio. Combined with Supabase pgvector vector database and OpenRouter large language models, it provides a complete production-level retrieval-augmented generation pipeline.

As enterprise data types become increasingly diverse, text-only RAG systems can no longer meet practical needs. A complete knowledge base often includes:

• Text documents: PDF reports, Word documents, Markdown notes
• Image materials: Product photos, design drawings, screenshots
• Video content: Meeting recordings, training videos, product demos
• Audio records: Meeting recordings, podcasts, voice memos

Traditional solutions require maintaining separate processing pipelines and storage systems for each media type, which not only increases architectural complexity but also severs the connections between different pieces of knowledge. Google's Gemini Embedding 2 is the first truly full-modal embedding model that can represent text, images, videos, and audio in a unified vector space, laying the technical foundation for multimodal RAG.

The framework adopts a clear layered architecture design with well-defined responsibilities for each component:

The core of the framework uses the gemini-embedding-2-preview model, which generates 3072-dimensional dense vectors via REST API calls. This model supports Matryoshka representation learning, allowing flexible selection of dimensions between 128 and 3072 to balance storage costs and retrieval accuracy.

All embedding vectors are stored in Supabase's PostgreSQL database, using the pgvector extension for efficient similarity search. The database uses the HNSW (Hierarchical Navigable Small World) index structure and supports cosine similarity calculation, ensuring millisecond-level retrieval response.

The framework accesses various large language models through the OpenRouter unified interface, with Claude Sonnet 4 as the default. This design allows users to flexibly switch models according to task requirements without modifying business code.

The asynchronous server built on FastAPI and uvicorn supports high-concurrency request processing. The integrated file monitoring function is implemented using the watchfiles library, which can automatically detect file changes in the data/ directory and trigger incremental indexing.

The framework has designed specialized processing strategies for different types of media content: