Multimodal RAGOps Platform: Building a Measurable and Iterative Retrieval-Augmented Generation (RAG) Engineering System

This article introduces a neutral vendor's multimodal RAGOps platform, which aims to build a measurable and iterative retrieval-augmented generation (RAG) engineering system. The platform supports unified evaluation, fine-tuning, and routing across OpenAI, Anthropic, Google, and open-source models, with multimodal input processing capabilities. It addresses the engineering challenges of RAG systems and promotes the transition of RAG from prototype to production-level infrastructure.

Most current RAG systems are built as one-off projects, lacking continuous optimization mechanisms. Processes like model selection and prompt engineering have become black-box decisions. Modern RAG needs to handle complex scenarios: supporting multiple LLMs (closed-source like GPT-4, open-source like Llama), multimodal inputs (text, PDF, Excel, voice, charts/images), and solving problems such as intelligent routing, optimal strategy selection, and quantitative effect evaluation.

The multimodal-ragops-platform is a systematic engineering platform that treats model selection, prompt variants, etc., as measurable and versioned experimental variables. Its key feature is neutrality—it is not tied to a single model provider and supports OpenAI, Anthropic, Google Vertex AI, and local open-source models (Ollama). Core components include: ingestion-service (multimodal input normalization), routing-service (model adaptation and cost scheduling), eval-service (RAGAS evaluation + MLflow tracking), vision-service (chart extraction pipeline), and finetune (SFT + DPO scripts).

The platform supports multiple input types: text and documents (PDF parsing, structured extraction), voice input (OpenAI Whisper transcription), tabular data (text-to-SQL processing for xlsx), and visual charts (three-stage processing: classification → extraction → verification). Chart processing has three stages: 1. Fast classification with GPT-4o-mini; 2. Local DePlot for standard charts, GPT-4o Vision for complex ones; 3. XBRL cross-validation for numerical accuracy—deviations exceeding 5% trigger an upgrade. This strategy balances accuracy and cost.

The platform's core innovation is a cost-aware intelligent scheduling mechanism: 1. A/B testing capability: compare model configurations, prompt variants, and retrieval parameters, with experiment records stored in MLflow; 2. Adaptive routing: select the most cost-effective model based on task characteristics and quality thresholds (e.g., Llama3.2 for simple Q&A, GPT-4o/Claude for complex reasoning); 3. Plug-and-play architecture: extend to CV or recommendation systems via the ModelAdapter interface.

The platform supports multiple fine-tuning technologies with unified RAGAS benchmark evaluation: Supervised Fine-Tuning (SFT, supporting OpenAI Fine-tuning API, Google Vertex AI), Direct Preference Optimization (DPO, QLoRA implemented on Llama3.2 3B using HuggingFace TRL library). The tech stack is cloud-native: FastAPI (API framework), Docker/docker-compose (containerization), RAGAS (evaluation), MLflow (experiment tracking), Whisper (voice), DePlot/GPT-4o Vision/img2table (vision).

The platform is a supporting engineering platform for graph-rag-finance-assistant, providing it with evaluation, multi-model routing, and multimodal support. Practical insights: 1. RAG requires an engineering mindset—treat decisions as experimental variables and establish evaluation and tracking mechanisms; 2. Multimodal is a trend—need to handle multiple data formats uniformly; 3. Cost awareness is important—intelligent routing controls costs; 4. Maintain vendor neutrality to avoid lock-in and leverage the advantages of various providers.

The multimodal-ragops-platform represents the engineering evolution direction of LLM applications, pursuing 'running better': through systematic evaluation, flexible routing, multimodal support, and continuous fine-tuning, it becomes an iterative production-level infrastructure. For enterprises that want to push RAG from prototype to production, it is an open-source project worth studying.