LLM Colosseum: An Arena for Evaluating Large Language Models' Reasoning Ability Through Mutual Questioning

LLM Colosseum is an experimental framework that innovatively adopts an adversarial evaluation paradigm between models—allowing large language models to design reasoning challenge questions for each other to evaluate their reasoning abilities, breaking through the limitations of traditional static evaluation and opening up a new direction for LLM reasoning ability assessment.

Current mainstream LLM evaluations rely on static benchmark datasets (such as MMLU, HumanEval, GSM8K, etc.) and have obvious limitations: they are prone to targeted optimization by model developers, leading to a disconnect between scores and actual abilities; fixed questions cannot cover the performance of open-ended reasoning tasks; the one-way "humans design questions, models answer" mode does not fully utilize models' creativity and critical thinking.

Its core is an adversarial cycle:

1. Challenge Design: Model A designs reasoning challenges such as logical puzzles and mathematical problems;
2. Challenge Solving: Model B attempts to solve them, testing reasoning and comprehension abilities;
3. Result Evaluation: The system or a third party evaluates the correctness of the solutions and provides feedback;
4. Role Rotation: A and B swap roles and repeat the process, comprehensively evaluating both question-designing and problem-solving abilities.

This framework can evaluate dimensions that are difficult to cover with traditional tests:

• Creativity and Question Design: Whether it can design interesting and unambiguous challenging questions;
• Reasoning Depth: Whether it can perform multi-step reasoning to reach correct conclusions when facing complex problems from similar models;
• Metacognitive Ability: Whether it can predict error patterns of other models and design "trap" questions;
• Self-Assessment Ability: Whether it can accurately judge the difficulty of questions and predict the performance of other models.

Key components of the project include:

• Frontend Interface: index.html + js/ provides a visual arena interface to display the adversarial process and results;
• Prompt Engineering: Carefully designed templates under prompts/ guide models to generate high-quality challenges;
• Automation Scripts: scripts/ handle model calls, result collection, and scoring logic;
• Static Resources: assets/ provide interface styles and image resources.

The value of this paradigm:

• Dynamic Evaluation: Questions are dynamically generated, making targeted optimization difficult and better reflecting real abilities;
• Complementary Ability Assessment: Evaluates models' two-way abilities of "attack" (question design) and "defense" (problem solving);
• Emergent Ability Discovery: Models may exhibit new abilities not observed in standard evaluations when designing challenges;
• Multi-dimensional Comparison: Reveals models' relative strengths and weaknesses in different domains, rather than a single score ranking.

Potential Applications:

• Model ability benchmarking as a supplement to traditional evaluations;
• Training feedback to identify models' weak points and guide fine-tuning;
• Safety research to evaluate models' safety alignment levels;
• Research platform for multi-agent collaboration and competition dynamics.

Limitations and Future:

• Current limitations: Subjectivity of evaluation criteria, fairness of adversarial processes (preventing loopholes/no solution), and scale expansion (multi-model tournaments);
• Future directions: Refining scoring mechanisms, supporting multi-turn dialogue challenges, and integrating human expert evaluations as benchmarks.