HalLing Benchmark: Revealing the Deep Mechanisms of Large Model Hallucinations from a Linguistic Perspective

The HalLing (Hallucination in Linguistic Reasoning) benchmark approaches from a linguistic perspective, systematically evaluating the hallucination tendencies of large models in linguistic reasoning through six phenomena: ambiguous sentences, anaphora resolution, center embedding, garden-path sentences, quantifier scope, and first-order logic extension. Unlike traditional evaluation methods that focus on factual errors, it pays more attention to whether the model truly understands the semantic structure of the input text, revealing the deep shortcomings of current large models in language comprehension ability.

The hallucination problem of large models is a core issue in AI safety and reliability research, but mainstream evaluation methods mostly focus on factual errors and ignore the model's understanding of the semantic structure of input. HalLing provides a new evaluation paradigm: instead of checking whether the model "knows" facts, it tests whether it can correctly parse linguistically challenging inputs and reason. This shift in perspective reveals the deep shortcomings of models in language comprehension.

HalLing builds its evaluation system around six core linguistic phenomena:

1. Ambiguous Sentences: Test the model's ability to disambiguate based on context;
2. Anaphora Resolution: Hierarchically examine the referential relationship between pronouns and entities (basic, extended, and failure tests);
3. Center Embedding: Test the model's syntactic parsing ability by increasing embedding depth;
4. Garden-Path Sentences: Evaluate the model's reanalysis ability to correct initial incorrect parsing;
5. Quantifier Scope: Test the model's ability to map the logical relationships of quantifiers;
6. First-Order Logic Extension: Extend the evaluation to the level of formal reasoning.

HalLing uses a dual-track evaluation method (Multiple Choice Questions MCQ + Open-ended Questions OQ) and has evaluated four major model families: Llama, Mistral, Qwen, and GLM-4. The results show that different models have significant performance differences across various linguistic phenomena, and no model excels in all dimensions, confirming the multidimensional nature of language comprehension ability. The evaluation results are stored in Excel, supporting secondary analysis.

Conclusions: HalLing reveals that current large models still have significant gaps in the core ability of "truly understanding language." Recommendations: Developers can use HalLing to identify weak links in models' semantic understanding and make targeted improvements; in scenarios requiring precise semantic parsing such as legal texts and contract clauses, special attention should be paid to the linguistic reasoning hallucination problem of models.

HalLing has built a multi-dimensional and multi-level evaluation system for large model linguistic reasoning hallucinations. Starting from classic linguistic problems, it systematically tests six dimensions, providing a new evaluation perspective and tool for researchers and developers concerned with the reliability and safety of large models.