MM-CreativityBench: Testing AI's Creative Physical Intelligence

This post introduces MM-CreativityBench, a benchmark designed to evaluate AI's creative physical intelligence—specifically, the ability to find non-obvious but physically feasible uses of objects in scenes. The core finding from the study is that current multimodal models' failures in such tasks are not due to poor generation ability but a lack of sustained visually grounded exploration. This benchmark and its insights point to key directions for improving AI's ability to solve real-world creative problems.

Creative physical intelligence refers to the ability to discover visually grounded solutions in open environments—like using a book to prop up a monitor or a credit card to open a box. While humans do this intuitively, current AI systems struggle with such 'on-the-fly' creative thinking. Large multimodal models excel at pattern recognition and standard tasks but fall short in reusing objects in non-traditional ways, which is a critical gap for real-world applications.

MM-CreativityBench is an affordance-grounded benchmark for creative tool use. Unlike traditional benchmarks that test standard object uses, it focuses on non-obvious uses (e.g., using a spoon as a screwdriver). Each test instance provides a scene image and structured views of candidate entities and their parts, allowing detailed observation of how models iterate through scene checks, identify relevant affordances, and form solutions based on visual and physical evidence.

Testing mainstream multimodal models revealed three main failure modes:

1. Ignoring relevant entities: Failing to notice objects critical to solving the problem.
2. Insufficient checking: Identifying relevant objects but not examining their key parts or properties.
3. Attribute hallucination: Imagining non-existent properties/functions of objects,脱离 visual content. The root cause? Models can't sustain visually grounded exploration—they rely on internal knowledge instead of continuous observation during multi-step reasoning, leading to either conservative (standard uses) or unrealistic (hallucinated) solutions.

To address these issues, the study proposes an 'affordance grounded alignment' method. It frames creative tool use as a preference learning problem, using Direct Preference Optimization (DPO) to encourage models to choose reasoning paths grounded in visual evidence over hallucinated ones. Additionally, it integrates supervision from an affordance knowledge base to guide broader entity exploration and multi-round planning, helping models distinguish grounded from ungrounded reasoning.

The trained models showed consistent improvements in selecting correct entities/parts and reducing hallucinations, boosting success rates in creative tasks. However, gaps remain:

• Performance in complex scenes is still far below human levels, especially when multiple solution paths exist.
• Cross-domain generalization is poor—models struggle with new object types/scenes, indicating over-reliance on memory/pattern matching rather than core creative reasoning principles.

The findings have key implications for embodied intelligence and robotics. Real-world robots need to 'improvise' with available objects instead of waiting for specific tools. The study suggests that scaling model size or data isn't enough—models need to develop sustained grounded exploration abilities (observe, verify hypotheses, adjust strategies based on evidence). This may require new architectural designs or training paradigms rather than just scaling up existing systems.