SupChain-Bench: A Large Language Model Benchmark for Real-World Supply Chain Management Scenarios

SupChain-Bench is an LLM evaluation benchmark developed by the AIDC-SupplyChain-AI team for real-world supply chain order management scenarios. It aims to systematically test LLMs' tool invocation and multi-step reasoning capabilities. By simulating a three-tier supply chain system and using conditional tool invocation chains and a multi-dimensional evaluation framework, it fills the gap in general benchmarks for evaluating specific industry scenarios.

Existing LLM benchmarks mostly focus on general capabilities (e.g., mathematical reasoning, code generation), but enterprise applications need to deal with structured business systems, complex data hierarchies, and strict processes. In supply chain management scenarios, a simple query may require cross-level reasoning and dynamic tool invocation, while traditional evaluations only look at the final result and ignore intermediate steps. The uniqueness of SupChain-Bench lies in evaluating both result accuracy and the rationality of tool invocation chains.

The core of SupChain-Bench is a three-tier order management system that simulates real business logic:

1. Transaction Order: The top-level customer order, containing buyer information and identifiers, which can be associated with 1-5 fulfillment orders;
2. Fulfillment Order: The logistics execution unit with independent statuses (normal/canceled/error, etc.);
3. Warehouse Order: The smallest warehousing execution unit, with each fulfillment order linked to 1-3 warehouse orders, including status and error information. In addition, it includes error logs and cancellation context auxiliary tables to accurately simulate the data organization method of e-commerce logistics.

The benchmark provides 8 OpenAI-compatible tool functions covering all aspects of supply chain queries. The tool chain reflects real business conditional logic: the model must first query the buyer and order ID, then dynamically adjust subsequent calls based on the fulfillment status (e.g., canceled status requires checking the cancellation reason, error status requires checking the error reason). This conditional branching design is the essence of the benchmark, testing the model's ability to dynamically invoke strategies.

SupChain-Bench uses fine-grained entity-level evaluation (precision and recall for transaction/fulfillment/warehouse tiers) and includes conditional logic evaluation (normal/canceled/error processes). It also supports multiple prompt strategies: standard mode, ReAct mode (think-action-observe loop), and SOP-guided mode (predefined business rules), helping to compare the impact of different prompt methods.

The project provides a synthetic data generation tool that can generate test datasets by controlling parameters such as the number of orders, cancellation rate/error rate; a deterministic result arrangement script ensures the reproducibility of standard answers. Evaluation process: Model prediction results are saved in JSONL format (including tool invocation traces), and the evaluation script automatically reconstructs the data structure and compares each field with the standard answer.

SupChain-Bench sets a new benchmark for LLM evaluation in the supply chain field, and its methodology (simulating real tiers, conditional tool chains, multi-grained evaluation) can be extended to complex scenarios such as financial risk control and medical diagnosis. For enterprises, it is both a practical evaluation tool and a benchmark design example, helping to reliably deploy LLMs in core business systems. Project open-source address: https://github.com/Damon-GSY/SC-bench