LLMEval-Logic: A New Benchmark and Adversarial Reinforcement Method for Chinese Logical Reasoning Evaluation

This article introduces LLMEval-Logic, a real-scenario-based Chinese logical reasoning benchmark constructed through expert review, Z3 solver validation, and an adversarial reinforcement process. It includes basic and hard sets. Experiments reveal a significant gap in complex logical reasoning among current cutting-edge large language models, providing a new standard for evaluating the logical reasoning capabilities of Chinese LLMs.

Natural language logical reasoning is a core capability of LLMs, and evaluating its reliability is crucial for high-risk scenarios (law, medical care, finance). However, existing benchmarks have three major issues:

1. Template-based generation leads to single data type, which is disconnected from real scenarios—models tend to recognize patterns rather than master reasoning;
2. Formal annotations are rough or un-reviewed, making the credibility of evaluation results questionable;
3. Cutting-edge models (e.g., GPT-4, Claude) achieve over 90% accuracy on traditional benchmarks, which lose their discriminatory power.

LLMEval-Logic adopts a rigorous construction process:

1. Forward creation: Create questions based on real scenarios (daily life, business, law, etc.), emphasizing authenticity and diversity;
2. Expert review and formalization: Domain experts write reference formal representations to ensure strict correspondence with natural language;
3. Z3 solver validation: Verify the correctness of answers via Microsoft Z3 theorem prover to provide formal guarantees;
4. Expert scoring standards: Develop 1400 scoring atoms covering various logical structures;
5. Adversarial reinforcement: A closed-loop system analyzes model failure patterns, adjusts question difficulty, and ensures it is challenging for cutting-edge models.

LLMEval-Logic includes two paired subsets:

• Basic Set: 246 questions, 1400 expert scoring standards, medium difficulty, covering basic logical reasoning types;
• Hard Set: 190 questions, 938 multi-step sub-questions, high difficulty, targeting closed model spaces (excluding simple pattern matching). Hierarchical evaluation is possible: the basic set tests basic capabilities, while the hard set explores limit performance.

Evaluation results for 14 cutting-edge models:

1. Low accuracy on the hard set: The best-performing model only achieves 37.5% accuracy, with an error rate exceeding 60%, which contrasts with their excellent performance on general NLP tasks;
2. Limited formal conversion capability: Even with reference symbols provided, the highest combined Z3 + scoring standard score is only 60.16%, indicating that models struggle to accurately convert natural language to formal logic;
3. Significant differences between models: Open-source models have an accuracy rate below 20% on the hard set, while closed-source models perform better but with small gaps—logical reasoning is a common shortcoming.

Contributions of LLMEval-Logic:

1. Real-scenario orientation: Ensure evaluation is relevant to actual application needs;
2. Formal verification guarantee: Z3 validation provides mathematical credibility;
3. Adversarial reinforcement mechanism: Ensure the benchmark remains challenging and avoids becoming obsolete quickly;
4. Chinese coverage: Fill the gap in non-English evaluation resources and facilitate the development of Chinese AI applications.

Limitations:

• Limited scale (436 questions);
• Domain coverage focuses on general logic, with insufficient coverage of professional fields (mathematical proof, program verification);
• The degree of automation for dynamic updates needs improvement. Future directions:

1. Expand scale and include more questions and reasoning types;
2. Multilingual expansion;
3. Establish a real-time updated adversarial reinforcement pipeline;
4. Conduct fine-grained analysis of model performance differences across different logical structures.

LLMEval-Logic sets a new standard for Chinese logical reasoning evaluation through a rigorous process and adversarial reinforcement mechanism. Experiments show that cutting-edge models still have significant room for improvement in complex logical reasoning, reminding us to attach importance to the construction of basic reasoning capabilities. Only AI systems that master strict logical reasoning can be trusted in high-risk scenarios.