MLLM-HSGG: A Multimodal Large Language Model-Enhanced Dataset for High-Information Scene Graph Generation

This article introduces the MLLM-HSGG dataset, which uses multimodal large language models (MLLMs) to enhance the scene graph generation task, aiming to provide richer structured information representation for visual understanding. Its core features include multimodal fusion, high information density, and quality improvement, driving the development of the scene graph generation field through innovative technical methods.

Scene Graph Generation (SGG) is a core task in computer vision, converting images into structured graphs (nodes as objects, edges as relationships). Traditional SGG is limited by the quality and diversity of training data, making it difficult to capture fine-grained relationships in complex scenes. In recent years, MLLMs have demonstrated strong visual-language understanding capabilities, bringing new opportunities to SGG. The MLLM-HSGG project explores the use of MLLMs to enhance dataset quality and information density.

MLLM-HSGG is a dataset project focused on high-information scene graph generation. It enhances existing datasets through MLLMs to produce training data with richer relationship annotations and precise attribute descriptions. Core features:

• Multimodal Fusion: Combines visual features and language understanding to generate more accurate annotations
• High Information Density: Contains more fine-grained object relationships and attributes
• Quality Improvement: Uses MLLM reasoning to filter low-quality annotations and improve reliability

The project adopts innovative technologies to improve results:

1. Visual-Language Alignment: Achieves deep alignment between images and text through MLLMs to capture complex semantic relationships
2. Data Enhancement Strategies: Generates diverse relationship descriptions, verifies and corrects existing annotations, and supplements missing attributes and relationships
3. Quality Control Mechanism: An MLLM-based quality assessment module automatically identifies and filters incorrect annotations

The dataset can be applied to:

• Image Understanding: Improve the performance of visual question answering and image caption generation
• Visual Reasoning: Support complex scene understanding and logical reasoning
• Multimodal Learning: Provide high-quality data for visual-language pre-training
• Robot Navigation: Help robots understand environmental layouts and object relationships

The value of the project lies in exploring the application potential of MLLMs in structured visual data generation, shifting SGG from relying on visual features to visual-language joint modeling, and promoting the development of the field. It also reveals that large language models are not only used for generation tasks but also can serve as guardians and enhancers of data quality, playing a key role in scenarios where data is scarce or annotation is challenging.

MLLM-HSGG represents an important direction in the scene graph generation field—using MLLMs to improve data quality and model performance. With the advancement of multimodal technology, we look forward to more innovative methods emerging to drive visual understanding to a deeper level.