NVIDIA NIM Multimodal Agent: A New Paradigm of RAG Integrating Vision and Text

This article introduces the open-source nim-multimodal-agent project by Karthik Venugopal, which is built on LangGraph and the NVIDIA NIM platform to implement a multimodal Agentic RAG architecture. Its core innovation lies in intelligently routing retrieved charts to vision-language models and ensuring answer accuracy through the LLM-as-Judge mechanism, achieving 100% accuracy in benchmark tests. The project source code is available on GitHub (https://github.com/Karthikvenugopal/nim-multimodal-agent) and was released on June 11, 2026.

Traditional RAG systems mainly handle pure text corpora, but in real-world scenarios, a large amount of key information exists in visual forms such as images and charts (e.g., performance benchmark graphs, revenue pie charts, architecture diagrams). How to effectively integrate these visual information into the RAG process is an important challenge in current AI application development. This project is designed to address this problem.

Core Process: Based on the LangGraph state diagram, if the retrieval phase returns image chunks, after a relevance gating check (top-ranked or similarity meets the standard), they are routed to the visual analysis module; pure text is directly used to generate answers.

Tech Stack: Deeply integrated with the NVIDIA NIM ecosystem:

• Vision-Language Model: nvidia/nemotron-nano-12b-v2-vl (analyzes charts and converts them into structured descriptions)
• Text Generation and Judgment Model: nvidia/llama-3.3-nemotron-super-49b-v1.5 (generates answers + LLM-as-Judge)
• Embedding Model: nvidia/llama-nemotron-embed-1b-v2 (generates retrieval vectors for text/image descriptions)

All models are accessed via the OpenAI-compatible API (https://integrate.api.nvidia.com/v1), and versions can be switched via environment variables.

Corpus Design: The mixed corpus contains 3 pure text documents (corpus/docs/) and 5 PNG charts (corpus/images/, including latency benchmarks, revenue pie charts, etc.), and the chart data only exists at the pixel level and cannot be inferred from text.

Benchmark Tests: 11-question test set (5 answerable by text, 5 exclusive to charts, 1 unanswerable). Results:

• Total accuracy: 100%, average fidelity:1.0
• 100% of chart questions trigger visual analysis
• Unanswerable questions are correctly rejected, no hallucination generation.

Automated evaluation using LLM-as-Judge:

1. Correctness: Compare model answers with the gold standard, requiring accurate answers for answerable questions and clear rejection for unanswerable ones;
2. Fidelity: Evaluate whether the answer claims are supported by retrieval context/visual analysis results to measure anti-hallucination ability.

Dual verification ensures reliable system output, suitable for production deployment.

Application Scenarios: Enterprise knowledge bases (architecture diagrams/performance graphs in technical documents), scientific literature analysis (experimental result graphs), financial report interpretation (financial statement charts), operation and maintenance monitoring (dashboards/error rate graphs).

Usage: Provides a CLI interface:

• Single question query: python main.py "question"
• Full benchmark test: python main.py --benchmark

Extensibility: Customize models via the .env file, expand the corpus/ directory to adapt to business scenarios, and scripts/make_images.py can generate custom test charts.