LatentRouter: An Intelligent Routing System for Multimodal Large Models

This article introduces LatentRouter—an intelligent routing system based on counterfactual multimodal utility prediction, designed to solve the selection challenges brought by the heterogeneity of multimodal large models. Its core idea is to dynamically select the optimal model by matching model capability representation and query demand in the latent space, achieving a balance between performance and cost. This article will elaborate on aspects such as background, methods, experiments, and applications.

With the rapid development of Multimodal Large Language Models (MLLMs), different models exhibit significant heterogeneity in task performance (e.g., OCR, chart understanding, spatial reasoning, etc.), inference latency, and API costs. The traditional approach of fixed use of a single model has drawbacks: using expensive large models for simple queries wastes resources, while using lightweight models for complex queries results in insufficient performance. Therefore, it is necessary to dynamically select the most suitable model for specific image-text queries.

The core innovation of LatentRouter is to transform the routing problem into counterfactual multimodal utility prediction. Given an image-query input, the system needs to predict the output quality of each candidate model, rather than just estimating the query difficulty. This requires understanding both the multimodal needs of the query and the capability characteristics of the model to make informed decisions.

LatentRouter includes three key components: 1. Multimodal Routing Capsule: Extracts visual features, text semantics, and interaction patterns of image-query to form a compact representation; 2. Model Capability Token: Each candidate model is represented as a latent space vector, capturing the distribution of its capability dimensions; 3. Latent Communication Mechanism: Calculates the matching degree between query demand and model capability through interaction methods such as attention, achieving fine-grained semantic matching.

LatentRouter uses distributed output to predict the counterfactual quality distribution of each model, capturing uncertainty and providing rich decision-making information. For ambiguous cases, a bounded capsule correction mechanism is introduced to avoid overconfidence. The system supports flexible utility strategies: performance priority (selecting the model with the highest quality) or performance-cost balance (selecting the model with the lowest cost under the quality threshold).

In actual deployment, the model pool may change dynamically (new models added, old models unavailable). LatentRouter handles this situation through shared per-model scores combined with an availability mask: the model capability representation is fixed, and its score is masked when unavailable, allowing adaptation to new model combinations without retraining.

On the MMR-Bench and VL-RouterBench benchmarks, LatentRouter consistently outperforms fixed model baselines, feature-level routing, and learning routing baselines. The gains are most significant in task groups that are visually dependent, layout-sensitive, or inference-oriented. Ablation experiments verify that the latent communication mechanism is the main contributor to performance improvement.

Application Value: The prediction phase is lightweight with no additional latency; supports flexible strategy adjustment (cost priority during peak periods, performance priority in scenarios with strict quality requirements); modular design facilitates the integration of new models (only need to generate capability tokens). Future Directions: Expand to more modalities (audio, video); explore online learning to adapt to model performance changes; study the interpretability of routing decisions.