Introduction: XTC-Benchmark—A New Framework for Cross-Task Consistency Evaluation of Unified Multimodal Models

This article introduces the XTC-Benchmark evaluation framework, which systematically measures the ability of unified multimodal models to maintain consistency across different tasks, providing a new perspective for the reliability evaluation of multimodal AI. The core problem it solves is: when a model faces different tasks for the same input, does its output remain consistent? This issue directly affects the practical value and user trust of the model.

Background: Cross-Task Consistency—A Reliability Challenge for Multimodal AI

In recent years, unified multimodal large models (such as GPT-4V, Gemini, Qwen-VL, etc.) can handle multiple tasks simultaneously, including image understanding, visual question answering (VQA), OCR, object detection, etc. However, the problem of cross-task consistency has gradually become prominent: if a model says "there is an orange cat in the image" in image description but answers "there is no cat in the image" in VQA, it will seriously affect user experience and trust.

Cross-task consistency is a key dimension to measure model reliability. Its absence may expose three major defects:

1. Unstable representation: The encoding of the same input varies greatly across different task paths, indicating problems with the vision-language alignment mechanism;
2. Fragmented knowledge: Knowledge is scattered across different task heads/adapters, lacking unified semantic understanding;
3. Unreliable reasoning: Guessing answers in some tasks leads to conflicts with other tasks.

Evaluation Methodology of XTC-Benchmark

XTC-Benchmark uses a rigorous process to quantify cross-task consistency:

1. Task pair design: Select semantically related task pairs (e.g., image description and VQA, OCR and visual reasoning, etc.) that share the same visual input but have different output forms;
2. Consistency measurement: Evaluate the logical consistency of outputs through natural language inference (NLI) models and semantic similarity calculation (e.g., the description "a dog is on the grass" and the answer "no animals" are judged as inconsistent);
3. Fine-grained analysis: Provide overall scores and error type analysis to identify weak task combinations of the model;
4. Cross-model comparison: Support horizontal comparison of mainstream multimodal models to reveal the impact of architecture and training strategies on consistency.

Technical Implementation and Dataset Construction

The technical architecture of XTC-Benchmark includes four components:

1. Multi-task data alignment: Build a multi-task annotated dataset for the same image to ensure strict alignment of annotations;
2. Semantic equivalence judgment module: Fine-tune pre-trained NLI models (such as RoBERTa-NLI) to adapt to the expression characteristics of multimodal tasks;
3. Dynamic task generation: Automatically generate task variants based on templates (e.g., convert descriptions into different Q&A forms) to expand the evaluation scope;
4. Evaluation metric system: Define metrics such as strict consistency (complete equivalence), loose consistency (entailment relationship), and contradiction detection (direct conflict).

Research Findings: Model Performance and Influencing Factors

Evaluations based on XTC-Benchmark reveal the following findings:

1. Non-linear relationship between scale and consistency: Larger models perform better in some task pairs but may be worse in others, requiring specialized optimization;
2. Role of instruction tuning: Models with multi-task instruction tuning have better consistency, and joint training helps with unified understanding;
3. Task difficulty differences: Task pairs involving counting, spatial relationships, and attribute reasoning are prone to inconsistency, while existence judgment is more stable;
4. Impact of architecture design: Unified encoder-decoder architectures have better consistency than modularly spliced models, supporting the advantages of end-to-end training.

Implications for Model Developers

XTC-Benchmark provides the following guidance for developers:

1. Training strategy optimization: Introduce cross-task consistency loss functions in pre-training/fine-tuning stages to constrain compatible outputs;
2. Data augmentation: Build more multi-task annotated training data to learn the corresponding relationships between task expressions;
3. Architecture improvement: Explore multi-task architectures that share more parameters to reduce representation divergence of task-specific modules;
4. Evaluation integration: Treat cross-task consistency as a standard evaluation dimension, alongside accuracy and robustness.

Application Scenarios and Future Directions

Application Scenarios:

1. Model selection reference: Enterprise users use XTC scores to evaluate the reliability of candidate models;
2. Quality monitoring: Continuously monitor consistency in production environments to timely detect degradation or edge cases;
3. User trust building: Display consistency metrics to enhance user trust;
4. Academic research: Provide a standardized benchmark for research on multimodal understanding mechanisms.

Future Directions:

1. Expand task coverage: Include emerging tasks such as video understanding and 3D scene analysis;
2. Multilingual support: Evaluate consistency of non-English content;
3. Dynamic consistency: Study cross-turn consistency in multi-round dialogues;
4. Causal analysis: Explore the root causes of inconsistency (representation/knowledge/reasoning issues).

Conclusion: Towards More Reliable Multimodal AI

XTC-Benchmark fills an important gap in the evaluation of multimodal AI. While pursuing accuracy, we cannot ignore the internal consistency and reliability of outputs. Only when unified multimodal models provide coordinated and reasonable responses across all task scenarios can they become trustworthy intelligent assistants. The promotion of this framework will drive the industry towards more mature and reliable multimodal AI systems.