MINOS: A Multimodal Evaluation Model for Bidirectional Image-Text Generation

MINOS (Multimodal Evaluation Model for Bidirectional Generation) is a multimodal evaluation model specifically designed for bidirectional image-text generation tasks, aiming to address the limitations of traditional evaluation methods in handling bidirectional tasks (such as semantic gap, alignment challenges, and lack of bidirectional consistency). It adopts the design principles of semantics first, bidirectional alignment, and human perception. Through a dual-tower architecture (vision tower + language tower), cross-modal alignment module, and multi-evaluation heads, it provides unified, reliable, and fine-grained evaluation. It supports the assessment of quality, faithfulness, and consistency for tasks like image captioning and text-to-image generation, facilitating scenarios such as model development and content quality control.

Current multimodal AI systems (e.g., DALL-E, GPT-4V) can realize bidirectional conversion between images and text, but there are key issues in evaluation: traditional methods only handle unidirectional tasks (BLEU/CIDEr for image captioning, FID for image generation); semantic gap (pixel-level metrics ignore high-level semantics); text-image alignment challenges (lexical similarity fails to reflect content accuracy); lack of bidirectional consistency (absence of cyclic consistency verification).

MINOS follows three core design principles: semantics first (focus on content rather than surface features), bidirectional alignment (verify the faithfulness between generation and input), and human perception (consistent with human judgment). Its technical architecture uses an innovative dual-tower design: vision tower (optimized vision Transformer to extract semantic representations like objects, attributes, and relationships); language tower (fine-tuned pre-trained language model to parse semantics, resolve anaphora, etc.); cross-modal alignment module (contrastive learning maps images and text to a shared semantic space); multi-evaluation heads (quality, faithfulness, consistency, and fine-grained diagnosis).

MINOS training consists of three stages: 1. Pre-training: learn basic cross-modal alignment on large-scale image-text paired data (COCO, VQA, etc.); 2. Contrastive learning: train with hard negative samples, partially matched samples, and perturbed samples to distinguish subtle semantic differences; 3. Human preference alignment: fine-tune using RLHF (Reinforcement Learning from Human Feedback) technique with human evaluation data (quality scores, accuracy judgments, etc.) to calibrate evaluation criteria.

MINOS performs excellently in multiple benchmark tests: image captioning evaluation (correlation with human judgment >0.85 on COCO Captioning, outperforming CIDEr/SPICE); text-to-image evaluation (accuracy of detecting misalignment >90%); bidirectional consistency evaluation (correlation between cyclic consistency score and manual evaluation is 0.88); fine-grained diagnosis (identifies issues such as omissions, misrecognition, and inaccurate counts).

MINOS can be applied in: model development iteration (quickly test variants and accelerate improvements); content review and quality control (automatically filter low-quality results); benchmark standardization (unify evaluation framework to improve comparability); education and explanation (fine-grained feedback helps understand system behavior).

MINOS has limitations: high computational overhead (large model inference cost); domain specificity (performs well in general scenarios but needs adaptation for specific domains); subjectivity challenges (difficult to capture all differences in dimensions like creativity). Future directions: expand to video/audio modalities; develop real-time evaluation; serve as a reward model to optimize the training of generation systems.