DesignDeathmatch: A Benchmark for Evaluating Creative Design Capabilities of Large Language Models

DesignDeathmatch is an innovative open-source benchmark framework aimed at systematically evaluating the comprehensive capabilities of large language models (LLMs) in end-to-end creative design tasks. This test requires models to independently build a complete brand identity system for the fictional brand VEKTRA (Berlin Generative Audio-Visual Instrument Studio), covering processes such as design token definition, logo design and animation, visual identity system construction, and runnable brand website development. It provides a standardized and reproducible testing platform for AI creative capability evaluation, solving the dilemma of subjectivity in existing assessments.

With the improvement of LLM capabilities, they have entered creative fields such as brand design and visual system development. However, how to objectively and systematically evaluate these capabilities remains a major challenge. Existing benchmarks mostly focus on quantifiable tasks like mathematical reasoning and code generation, while creative design evaluation often stays at the subjective level, lacking a standardized framework. The emergence of DesignDeathmatch fills this gap and provides a rigorous testing platform for AI creative capabilities.

What is DesignDeathmatch

DesignDeathmatch is an open-source benchmark project that evaluates the performance of LLMs in end-to-end creative design tasks. The core challenge is to let models independently build a complete brand identity system for the fictional brand VEKTRA, covering tasks such as design token definition, logo design and animation, visual system construction, and brand website development.

Reasons for Choosing the VEKTRA Case

• Domain Complexity: Involves the intersection of music, visual arts, and technology, requiring integration of multi-disciplinary knowledge;
• Cultural Context: The unique atmosphere of Berlin's creative industry hub tests the model's ability to capture regional characteristics;
• Technical Challenge: The 'generative' requirement reflects dynamic algorithmic traits, testing technical understanding;
• Rich Evaluation Dimensions: Covers multi-level assessments including static visuals, dynamic animations, and interactive experiences.

DesignDeathmatch evaluates the creative performance of models from six core dimensions:

1. Design Taste: Aesthetic quality, including color usage, font selection, visual hierarchy, and overall beauty;
2. Brand Consistency: Unified design language, coherent brand tone, cross-media adaptation;
3. Creative Ambition: Concept depth, innovation level, storytelling;
4. Technical Expressiveness: Animation quality, interactive design, code quality, responsive adaptation;
5. Independent Execution Capability: Task completion rate, error handling, process management;
6. Execution Efficiency: Number of API calls, time cost, resource utilization rate.

Preparation Phase

• Environment Initialization: Run setup_run.bat to create an isolated workspace and a dedicated directory for the model;
• File Preparation: Provide BRIEF.md (creative brief), DESIGN.md (style reference), TASKS.md (delivery checklist), and RULES.md (execution constraints) to the model. SCORING.md (manual scoring criteria) and README.md are not provided.

Execution Phase

• Initial Design: The model reads documents in the order of prompts, makes independent decisions on unclear content, updates the progress in TASKS.md, and finally creates RUNLOG.md to record the process;
• Iterative Optimization: The model needs to upgrade the initial version to an excellent level, create a v2/ directory to save the optimized version (without overwriting original files). Optimization content includes logo upgrade, animation interaction enhancement, design aesthetics refinement, and code refactoring.

Evaluation Phase

• Automated Check: Verify file integrity, code syntax, link validity, etc.;
• Manual Review: A double reviewer mechanism scores according to SCORING.md;
• Indicator Recording: Extract efficiency data such as execution time and number of API calls from RUNLOG.md.

Model Capability Evaluation

• Horizontally compare the creative design performance of different models;
• Vertically track the capability evolution of the same model version;
• Diagnose the strengths and weaknesses of models in creative design.

Product Development Guidance

• Identify the current capability boundaries of models;
• Define product function scope based on test results;
• Compare the feasibility of different technical solutions.

Education and Research

• Serve as a teaching case for AI creative design;
• Provide a standardized evaluation benchmark for related research;
• Help designers understand the possibilities and limitations of AI creativity.

Current Limitations

• Subjectivity: Creative evaluation still contains subjective factors, and reviewer evaluations may vary;
• Technical Threshold: Requires models to have front-end development capabilities, which is not applicable to pure text models;
• Cultural Dependence: The VEKTRA case is based on Western context, leading to potential biases in evaluating models for other cultural markets.

Future Directions

• Develop multi-cultural test suites (Asia, Africa, Latin America, etc.);
• Introduce dynamic difficulty adjustment mechanisms;
• Expand to multi-modal creative tasks such as audio, video, and 3D design;
• Establish a community-driven design case library.

DesignDeathmatch elevates AI creative design evaluation from subjective judgment to a systematic benchmark testing level. It tests the model's comprehensive capabilities in aesthetic judgment, creative expression, and independent execution through end-to-end tasks. This framework provides an objective and reproducible evaluation tool for the application of AI in the creative industry. As LLM capabilities improve, such creative benchmark tests will become more important, helping to understand and guide the development direction of AI creative capabilities.