Brick-SR1: Multimodal Semantic Routing Gateway for Unified Handling of Text, Image, and Audio Inputs

This article introduces the Brick-SR1 multimodal semantic routing gateway, which unifies the handling of text, image, and audio inputs through a single OpenAI-compatible interface. It automatically detects the modality type and routes requests to the appropriate backend model without requiring any changes to the client. Its core purpose is to solve the fragmentation problem in multimodal AI deployment by migrating modality processing logic to the gateway layer, providing application developers with a transparent and unified dialogue interface.

Modern large language model deployment faces fragmentation issues: different input modalities (text, image, audio) require different backend models and API endpoints, forcing clients to implement modality detection and model selection logic on their own. This coupling makes the system fragile—when backend APIs change, clients need to modify their code, leading to continuously increasing operational overhead. Brick was created to migrate modality processing logic to the gateway layer and provide a truly unified interface.

Brick's architecture is built around the concept of transparent proxy and includes the following components:

1. Modality detection and classification: Classifies requests into five combinations—image+text, image-only, audio-only, audio+image, text-only;
2. Concurrent preprocessing: Uses Go goroutines to execute OCR and speech-to-text in parallel;
3. Intelligent routing decision: For image inputs, first attempts OCR; if the result is too short, falls back to a visual model;
4. Semantic routing integration: Passes preprocessed text to the semantic pipeline, which evaluates 11 signals to select the backend.

Routing Strategy Table:

Input Modality	Processing Method	Destination
Image + Text	Keep original multimodal request	Visual Model
Image Only	OCR processing → if text length ≥ threshold then semantic pipeline, else visual model	Semantic Pipeline / Visual Model
Audio Only	Whisper-compatible STT transcription	Semantic Pipeline
Audio + Image	Parallel OCR + STT	Semantic Pipeline
Text Only	No preprocessing	Semantic Pipeline

Brick is implemented using Go 1.24 as part of an HTTP proxy server:

• Concurrent processing: For audio+image inputs, uses sync.WaitGroup to coordinate goroutines for parallel execution of transcription and OCR;
• Request body rewriting: Rewrites preprocessed text content into OpenAI dialogue format, making it transparent to downstream components;
• Pass-through mode: Clients can specify the backend via the x-selected-model header to skip preprocessing.

Brick is configured via config.yaml (requires enabled:true), which includes visual/OCR/STT models and endpoints, as well as the minimum text length threshold for OCR. Deployment steps:

1. Clone the repository: git clone https://github.com/massaindustries/semantic-routing.git
2. Build the Docker image: docker build -t mymodel:latest .
3. Start: docker compose -f deploy/docker-compose/docker-compose.yml up -d
4. Verify: curl http://localhost:8000/health & curl http://localhost:8000/v1/models

Brick is suitable for the following scenarios:

• Simplify client development: Unified interaction in OpenAI format, no need to care about backend differences;
• Cost optimization: Intelligently route to the appropriate model (simple text → low cost, complex multimodal → high capability);
• Flexible backend switching: Modify gateway configuration only, no client code changes required;
• Unified monitoring: Track model usage and cost attribution via the x-selected-model header.

Brick is based on the vLLM semantic router (Apache 2.0 license), with code hosted on GitHub. Current features include an independent HTTP proxy (no Envoy dependency), Brick virtual model, Regolo API integration, and OpenAI Responses format conversion. As part of the MyModel LLM gateway, it supports a complete semantic routing pipeline and plugin chain, and is continuously evolving.