DesignDeathmatch: A New Benchmark for Evaluating the Creative Capabilities of Large Language Models

DesignDeathmatch is a specialized benchmark for evaluating the creative capabilities of large language models (LLMs). By having models independently complete full brand design tasks—from design tokens to animated logos and functional websites—it comprehensively assesses multi-dimensional creative abilities such as design taste, brand consistency, technical expressiveness, and autonomous execution. This benchmark simulates real design project workflows and combines an automated checking and manual review hybrid scoring system, driving the evaluation of AI creative capabilities from purely technical metrics to comprehensive creative quality.

As LLMs excel in code generation, text understanding, and reasoning tasks, researchers are focusing on whether they possess true creative capabilities—including complex cognitive activities like aesthetic judgment, brand consistency, and design system construction. Traditional code capability benchmarks cannot fully measure potential in these creative domains, so DesignDeathmatch was developed to focus on creative quality rather than just technical implementation.

The core test scenario of DesignDeathmatch is to build a complete brand identity system for VEKTRA, a fictional generative audio-visual studio in Berlin, covering the end-to-end process from design tokens to animated logos and websites. Evaluation dimensions include: design taste (aesthetic judgment), brand consistency (coherence across multiple outputs), creative ambition (proactive interpretation and depth), technical expressiveness (dynamic interactive outputs), autonomous execution ability (completing projects without human intervention), and execution efficiency (efficiency in tool usage).

The test is divided into two phases: 1. Initial design execution: After reading four documents such as BRIEF.md and DESIGN.md, the model independently completes the entire process from design token definition and logo design to website construction; 2. Iterative optimization: The model receives upgrade instructions to elevate the baseline version to an excellent level, creates a v2 directory to save the iterative version, retains the original version for comparison, and tests self-criticism and creative upgrade capabilities.

The hybrid scoring system has a total of 157.5 points: automated scoring accounts for 102.5 points (verifying task completion and technical specifications), manual reviews account for 30 points (brand consistency, design taste, creative ambition—scored independently by at least two reviewers and averaged), and creative bonus items account for 25 points (rewarding stunning designs in the iterative optimization phase).

DesignDeathmatch provides a complete testing infrastructure: Windows batch scripts to create isolated test workspaces and detailed scoring guidelines; test results are collected into a VEKTRA dark-themed showcase website; the project is open-source under the MIT license, allowing free use to help establish a standardized creative capability evaluation system.

This benchmark marks the expansion of AI capability evaluation from code generation to complex creative tasks. It provides model developers with improvement directions (enhancing aesthetic perception, brand understanding, etc.), opens up new fields for researchers to quantify machine creativity, demonstrates the possibility of AI-assisted creative work, and lays the foundation for future human-AI collaborative creative workflows.

DesignDeathmatch represents an important direction in the transformation of AI capability evaluation from single technical metrics to comprehensive creative quality. It emphasizes that a truly powerful AI needs to understand beauty, create beauty, and maintain consistency. This benchmark provides a common measurement standard for the industry and promotes the development of AI systems toward more creative capabilities.