CauliBench: Testing Large Language Models' Instruction Following and Reasoning Stability with 'Cauliflower'

CauliBench is an open-source benchmark tool developed and maintained by CookieShualon (Source: GitHub, Link: https://github.com/CookieShualon/caulibench, Release Date: 2026-06-12). Wrapped in a humorous 'cauliflower' theme with serious technical goals, it tests large language models' instruction following ability, reasoning stability, and context retention through designed conflicting instructions. The project emphasizes reproducibility and LLM evaluation mechanisms, providing references for model selection, improvement feedback, and behavioral research.

Traditional benchmarks mostly focus on standard task performance (e.g., Q&A accuracy) and struggle to capture models' behavior under complex/contradictory instructions. CauliBench uses the 'cauliflower' metaphor to test models' 'persistence' when facing strange/conflicting instructions—derived from observations of models ignoring or over-complying with instructions. Its humorous theme lowers the entry barrier for this technical tool.

CauliBench designs tests around three dimensions:

1. Instruction Following: Test whether models execute accurately or follow mechanically through constraint instructions with strange elements like 'cauliflower';
2. Reasoning Stability: Observe whether models contradict themselves or revise conclusions reasonably in multi-turn dialogues;
3. Context Retention: Monitor whether models forget initial roles/constraints in long dialogues.

The project adopts a CLI-first design (written in TypeScript), with a core architecture including:

• Test cases (defined via structured JSON);
• Execution engine (model API interaction and error handling);
• Evaluation system (LLM judgment + deterministic metrics);
• Report generation (Markdown format). Reproducibility measures: fixed random seeds, versioned test sets, complete logs, and deterministic fallbacks.

The value of CauliBench includes:

1. Model Selection: Help teams predict model behavior in edge cases;
2. Improvement Feedback: Identify model weaknesses such as instruction following;
3. Behavioral Research: Provide standardized test scenarios for scholars to compare different model mechanisms.

Current limitations: Limited test coverage (does not involve math/code generation), and LLM judgment has subjectivity. Future plans: Expand the test case library, add multi-language support, develop visualization tools, and establish a community contribution mechanism.

The project has received positive feedback from the open-source community, and the MIT license encourages contributions. Developers have already submitted PRs: adding evaluation metrics, optimizing the CLI interface, and supporting more model providers.