MultiPun: Can Large Vision-Language Models Understand Multimodal Puns?

This article introduces MultiPun, a paper accepted at the ACL 2026 main conference, which explores the ability of large vision-language models (LVLMs) to understand image-text combined puns, revealing the limitations and challenges of current models in capturing cross-modal humor and ambiguity.

Puns are exquisite rhetorical devices in human language that use polysemy or similar pronunciation to create humor. When puns are combined with images, it requires simultaneous processing of visual and textual information and establishing connections, which increases the difficulty of understanding. The MultiPun project addresses this challenge by systematically evaluating the ability of mainstream LVLMs to understand multimodal puns.

Multimodal puns rely on image-text interaction to generate humor, with main types including:

• Visual-text puns: The image provides the literal meaning, while the text offers another layer of meaning
• Context-dependent: Requires specific cultural background knowledge
• Ambiguity resolution: The same content may have different interpretations

For example, a picture of a fish playing the piano paired with the text 'piano tuner'—the joke lies in the similar pronunciation of 'tuna' (a type of fish) and 'tuner' (someone who tunes pianos). This cross-modal semantic leap is a huge challenge for AI.

Dataset Construction

The dataset built by the research team has the following characteristics:

1. Diversity: Covers different pun types such as homophones and polysemy
2. Difficulty levels: From simple cases to complex ones requiring deep cultural knowledge
3. Manual verification: Ensures samples have clear humorous intent and reasonable explanations

Evaluation Dimensions

Multi-dimensional evaluation is designed:

• Understanding ability: Can it identify the existence of a pun?
• Explanation ability: Can it explain the joke?
• Generation ability: Can it create pun text given an image?

Key Findings

Tests on GPT-4V, Gemini, Claude, etc., revealed:

1. Limited recognition rate: The best model's recognition accuracy is far lower than that of humans
2. Weak explanation ability: Guessing the answer but misunderstanding the humor mechanism
3. Obvious cultural dependence: Worse performance on puns involving specific cultures

Failure Modes

Typical failures:

• Over-literalization: Missing metaphorical hints
• Modal separation: Difficulty in establishing deep image-text connections
• Lack of common sense: Missing world knowledge and cultural background

Semantic Leap Challenge

Understanding puns requires complex semantic reasoning: identifying multiple meanings, switching interpretations, and evaluating contextual rationality—this is a weak point of current large models.

Complexity of Cross-Modal Alignment

Multimodal puns require non-trivial image-text connections: the image object is literally related to the text, while another meaning of the text contrasts with or supplements the image. Humor comes from the 'unexpected yet reasonable' connection. Such connections are scarce in training data, making it difficult for models to grasp the patterns.

Implications for AI Research

1. Evaluation benchmark: Provides a new dimension to assess the deep understanding ability of models
2. Ability boundary: Reveals the limitations of models in fine-grained semantic reasoning tasks
3. Improvement direction: Points out the importance of enhancing cross-modal reasoning and common sense understanding

Potential Applications

• Creative assistance: Helping advertising and marketing generate and evaluate puns
• Content moderation: Identifying content that may cause misunderstandings due to cultural differences
• Educational applications: Developing tools for language learners to understand humor and idioms

MultiPun uses humor as an entry point to reveal the challenges of LVLMs in deep semantic understanding. The paper's title metaphorically means 'I get your meaning', but current models are still quite far from truly understanding puns. This research points the way for future improvements.