IFHierBench: A Benchmark for Evaluating Hierarchical Instruction-Following Capabilities of Large Language Models

IFHierBench is an open-source benchmark for evaluating the hierarchical instruction-following capabilities of large language models (LLMs). Its core innovations include introducing tree-structured output constraints (depth 0-3 layers) and deterministic Python validators to avoid subjective bias; it provides 600 test samples (evenly distributed across 4 depth levels) and an automated evaluation pipeline to help locate and improve the boundary of model capabilities.

Existing LLM instruction-following evaluations have issues such as subjectivity (relying on manual/LLM-as-judge), insufficient constraint complexity (lack of multi-layer nested evaluation), unreliable validation (fuzzy matching), and limited scalability. In real-world scenarios, user instructions often contain multi-layer constraints on format, content, structure, nesting, etc. Hierarchical instruction-following capability is a key manifestation of the practical value of models.

1. Tree-structured constraint structure: root node (basic instruction), intermediate nodes (hierarchical constraints), leaf nodes (fine-grained rules); 2. Depth stratification: 4 levels from 0 to 3 layers, 150 samples each, to precisely locate capability boundaries; 3. Deterministic Python validator: no LLM involvement, transparent rule-matching logic, reproducible and efficient, checking constraints node by node.

The dataset contains 600 samples (150 per depth level), with constraint types covering format, content, quantity, relationship, style, etc. The toolchain includes code (evaluation pipeline, validator, etc.) and data (sample files). Usage flow: environment configuration → data loading → model inference → result validation → report generation, supporting custom extensions (constraints, model interfaces, reports).

Research significance includes guiding model training (identifying shortcomings, screening data, validating iterations), evaluating application scenarios (adaptability judgment, prompt design), and guiding research directions (hierarchical attention, structured decoding, etc.). Summary: IFHierBench fills the gap in existing evaluations regarding complexity and reliability, promotes the development of instruction-following evaluations toward rigor and practicality, and is suitable for model evaluation in critical business scenarios.

IFHierBench is open-source (code and data are available on GitHub). Community contributions are welcome: submit new test samples (specific domain scenarios), improve the validator (support more constraints), share evaluation reports (establish leaderboards), and develop integration tools (integrate with other frameworks/platforms).