MLLMsent: A Visual Emotion Understanding Framework for Multimodal Large Language Models

MLLMsent is an open-source framework focused on the emotional reasoning capabilities of Multimodal Large Language Models (MLLMs). It provides end-to-end tools from image emotion classification to visual reasoning, exploring the mechanisms by which images convey emotions through complex scene semantics. The framework supports combined evaluation of various mainstream MLLMs and text models, offering a standardized benchmark for multimodal emotion analysis research, and has dual value in promoting academic research and practical applications.

Emotion analysis has long focused on the text domain, but human emotional expression includes rich visual information. With the rise of MLLMs, machines have the potential to understand image emotions, but visual emotion analysis faces three major challenges:

• Scene-level semantic complexity: Image emotions depend on subtle factors such as overall atmosphere, composition, and color
• Subjectivity and cultural differences: Emotional responses to the same image may vary across different cultural backgrounds
• Lack of interpretability: The reasoning process behind the model's emotional judgment is difficult to trace The MLLMsent framework is designed for the systematic study of these issues.

Dual-task Evaluation System

1. Direct Image Classification: Let MLLMs directly classify the emotional polarity of images (positive/negative/neutral) to test end-to-end understanding ability
2. Visual Reasoning Path: First generate a textual description of the image, then use a text LLM for classification, compare the effects of direct and indirect paths, and test the impact of description quality

Supported Model Matrix

• Multimodal models: GPT-4V series, DeepSeek-VL, Phi-4-multimodal, Gemma-4
• Text models: BART, mBERT, LLaMA series (comparison between pre-trained and fine-tuned versions)

End-to-end Pipeline

Covers dataset preprocessing and augmentation, batch inference for multiple models, result aggregation and analysis, and visualization report generation

Tech Stack

Based on PyTorch and Hugging Face Transformers library, with a unified model interface

Evaluation Metrics

In addition to accuracy and F1 score, it includes inter-model consistency analysis, error case clustering and visualization, and comparison of emotional intensity distribution

Academic Value

Provides a standardized evaluation benchmark, supports horizontal comparison of different MLLMs, longitudinal tracking of model iterations, and identification of model blind spots

Practical Application Scenarios

• Social media content moderation: Identify negative emotion images
• Advertising marketing optimization: Evaluate the emotional impact of visual materials
• Mental health assistance: Analyze the emotional tendency of images shared by users
• Art design research: Quantify the correlation between visual elements and emotions

• Comparison between direct and indirect reasoning: Revealing whether MLLMs directly "perceive" image emotions or make indirect judgments through the "visual → language → emotion" path is crucial for understanding the internal mechanisms of the model
• Mediating role of description quality: If the image description quality is not high, subsequent emotion classification will be affected, suggesting the need to optimize the visual-language conversion link

MLLMsent fills the tool gap in the field of multimodal emotion analysis, serving both as an evaluation framework and an experimental platform for exploring the cognitive capabilities of MLLMs. As the visual capabilities of models like GPT-4V evolve, systematic evaluation of their "visual emotional intelligence" will become more important, and this framework lays the foundation for this direction.