IDEAFix: An Evaluation Framework for Creative De-Fixation Prompts in Large Language Models

IDEAFix is a professional evaluation framework for the creative de-fixation ability of large language models (LLMs). Using 14,350 carefully designed prompts and 81 creative tasks, this framework systematically assesses the performance of mainstream models such as GPT-4o, Claude, and Gemini in breaking thinking stereotypes. Traditional creative evaluations often focus on fluency and diversity, while IDEAFix fills the evaluation gap in the key dimension of "de-fixation ability", providing a scientific basis for understanding the creative potential of LLMs.

Research Background

Creativity is a core feature of human intelligence. The performance of LLMs in creative tasks has attracted much attention, but traditional evaluations have ignored the key dimension of "de-fixation ability".

Problem Definition

De-fixation refers to the ability to break thinking stereotypes and break out of existing frameworks. Humans tend to fall into fixed thinking, and LLMs may also exhibit "creative fixation" due to training data biases or pattern limitations. The IDEAFix framework aims to systematically evaluate and quantify this ability of LLMs.

Framework Design Philosophy

IDEAFix integrates perspectives from psychology, design, and artificial intelligence, and builds an evaluation system based on three core dimensions:

• Typicality: Measures the degree of deviation of generated content from conventional patterns;
• Novelty: Evaluates the originality and uniqueness of creativity;
• Fluency and Diversity: Focuses on the quantity of creativity and the distribution of quality. This multi-dimensional approach provides a more detailed portrait of creative ability.

Dataset Construction

The dataset contains 14,350 prompts, covering 81 creative briefs (6 categories, including product design, service innovation, etc.). Prompt engineering adopts strategies such as attribute guidance, chain-of-thought, and emotional polarity control to test the performance differences of models under different conditions.

Innovation in Evaluation Methods

The framework introduces best practice methodologies from the human creativity field as evaluation benchmarks, including SCAMPER (Substitute, Combine, Adapt, etc.), TRIZ (Theory of Inventive Problem Solving), and C-K Theory (Concept-Knowledge Theory), to intuitively compare the gap between AI and human creative methods.

Evaluation Process

A systematic process is adopted: first, classify generated content through attribute labels, then combine expert manual annotations with automatic metric scoring, balancing evaluation depth and scalability.

Model Comparison Experiments

Tests were conducted on mainstream models such as GPT-4o, Claude series, Gemini-2.5-Flash, and Llama-3.1-70B. It was found that model size is not the only determinant of creative ability—some medium-sized models perform well in specific de-fixation tasks, while ultra-large models are sometimes overly conservative.

Key Findings

• Prompt engineering has a significant impact on the quality of creative output; well-designed de-fixation prompts can greatly improve novelty;
• Different models show obvious performance differences in different creative categories (e.g., product design vs. conceptual art);
• There is a tension between safety training and creative ability: some models suppress the boundaries of creative exploration due to conservative safety strategies.

Application Value

• Academic Field: Provides a standardized creative evaluation benchmark, facilitating model/algorithm performance comparison;
• Industry: Helps developers select models suitable for specific creative tasks and guides model improvement directions;
• Open Source Contribution: Open-source datasets and tools lower the threshold for creative AI research and promote field development.

Limitations and Future Directions

• Limitations: The evaluation is based on English corpus, with insufficient cross-language ability assessment; manual annotation costs limit dataset expansion; subjective creative dimensions are difficult to fully quantify.
• Future: Expand multi-language support, develop efficient automatic evaluation metrics, explore multi-modal creative evaluation, and establish dynamic evaluation mechanisms.