Gordian-X: An Adversarial Cognitive Stress Test Generation Engine for Large Language Models

Gordian-X is an open-source adversarial benchmark generator specifically designed to expose the reasoning flaws and cognitive blind spots of large language models (LLMs). Its core features include:

• Generates high-complexity test cases via 24 attack vectors (divided into 6 major categories)
• Covers 10 target domains including mathematics, computer science, physics, etc.
• Uses a two-stage architecture with separate generation and scoring to ensure test fairness
• Offers enterprise-grade features like batch suite mode and session tracking
• Minimalist tech stack, supports offline operation (except for API calls)
• Compatible with 10 mainstream LLM API providers, with a focus on accessibility design and privacy security

This article will cover its background, design methodology, technical implementation, application scenarios, and future directions.

Existing LLM benchmarks (such as GLUE, SuperGLUE, MMLU, HumanEval) have driven model capability improvements, but have obvious limitations:

• Severe model "score hacking" phenomenon: As training data expands and architectures are optimized, models perform close to or exceed humans on standard test sets, but this does not mean they have robust reasoning abilities
• Adversarial samples expose flaws: Many models that perform well on standard tests will make math errors, logical paradoxes, semantic biases, etc., when facing well-designed adversarial samples

Gordian-X emerged as a response—it is not a static benchmark set, but a dynamic "benchmark factory" aimed at actively mining the cognitive blind spots of LLMs.

The core design concept of Gordian-X is adversarial synthesis, i.e., generating cognitive traps targeting known weaknesses of LLMs via algorithms, rather than extracting questions from fixed question banks.

Attack Vectors and Target Domains

• 24 attack vectors: Divided into 6 categories: logical traps (recursive negation, implicit negation, etc.), constraints and forms (high-dimensional constraint satisfaction, numerical precision traps, etc.), cognitive bias exploitation (anchoring bias, survivor bias, etc.), semantics and language (semantic camouflage, polysemy traps, etc.), reasoning and theory (counterfactual logic, N-order theory of mind, etc.), advanced attacks (causal reversal, modal logic exploitation, etc.)
• 10 target domains: Covers mathematics, computer science, physics, philosophy and logic, economics and game theory, biology and medicine, law and ethics, history and social sciences, linguistics, general/abstract domains

Two-Stage Architecture

• Generation phase: Only outputs scenario prompts, no answers or metadata, ensuring test fairness
• Scoring phase: Independently computes correct answers and scores to avoid answer leakage

Enterprise-Grade Features

Supports batch suite mode, session tracking, problem history storage, structured export (JSON/Markdown/CSV), intelligent deduplication, chat command interaction, etc.

Gordian-X uses a minimalist tech stack with small code volume and clear structure:

• Only includes index.html (361 lines), app.js (2355 lines), style.css (2418 lines), and gordiux.png
• Zero dependencies, no build steps, supports fully offline operation (except for API calls)

Accessibility and Security

• Accessibility: Meets WCAG AA contrast requirements, supports keyboard navigation, ARIA labels, high-contrast mode, etc.
• Security and Privacy: API keys are stored only in browser localStorage, no telemetry, no server-side components, all operations are done on the client side

Supported LLM Providers

Compatible with OpenAI, OpenRouter, Anthropic, Google Gemini, Groq, Together AI, xAI, OpenCode Zen/Go, and custom API endpoints; supports streaming output.

Gordian-X has a wide range of application scenarios:

• Model developers: Identify model weaknesses and improve training data or architecture in a targeted manner
• Enterprise selection: Provide evaluation dimensions beyond standard benchmarks to help select more robust models
• Security research: Demonstrate the importance of adversarial testing in AI evaluation and expose LLM reasoning vulnerabilities
• Educational demonstration: Intuitively show the limitations of LLMs and explain that they do not yet have general intelligence

It provides an important tool for AI reliability research and practice.

Gordian-X has the following limitations:

• Requires manual verification of the rationality of test cases
• Insufficient depth in some highly specialized fields (e.g., cutting-edge mathematics)
• Current attack vectors are statically defined and cannot dynamically adapt to model evolution

Future directions include:

• Adaptive attacks: Adjust attack strategies based on real-time model performance
• Multimodal expansion: Extend tests to multimodal scenarios such as images and audio
• Collaborative evaluation: Support evaluation of multi-model collaboration in solving complex problems

As the project documentation states: 'If your model can solve Gordian-X tests, congratulations. We'll design a harder one.'