FACET Benchmark: Evaluating Attribution Faithfulness in Multi-Factor Reasoning of Large Language Models

FACET (Faithfulness Attribution in Complex Evaluation Tasks) is a four-probe benchmark framework designed for multi-factor reasoning scenarios of large language models (LLMs). Its core goal is to quantitatively evaluate the attribution faithfulness of models—i.e., whether the model's conclusions are based on real evidence. This benchmark covers a comparative analysis of eight cutting-edge models, focusing on the transparency and reliability of attribution chains, and provides a key evaluation tool for AI safety and alignment research.

As LLMs are increasingly applied to complex reasoning tasks, a key question emerges: when a model gives a conclusion, is it truly based on the evidence it claims? This is the Attribution Faithfulness problem. When models handle comprehensive reasoning tasks involving multiple factors, they may "hallucinate" non-existent evidence or incorrectly attribute results to irrelevant factors. In high-stakes scenarios such as medical diagnosis, legal consultation, and financial risk assessment, such attribution biases can lead to serious consequences. Therefore, developing systematic evaluation tools to measure the attribution faithfulness of models has become an important direction in AI safety and alignment research.

FACET adopts a four-probe architecture, specifically designed for multi-factor reasoning scenarios. Unlike traditional end-to-end accuracy evaluation, it focuses on the transparency and reliability of the model's internal attribution chain. The core evaluation dimensions include: attribution accuracy (whether the evidence truly supports the conclusion), attribution completeness (whether key factors are omitted), and attribution exclusivity (whether irrelevant factors are included). This benchmark has a verifiable design (all numerical claims are validated through CI processes), and the dataset has been archived on the Zenodo platform for long-term community access.

FACET conducted a systematic evaluation of eight current mainstream LLMs, revealing industry trends: there is no simple linear relationship between model size and attribution faithfulness; some small models perform better than large models in specific attribution tasks; different model families show systematic differences in attribution error patterns—some tend to over-attribute (attributing too many factors), while others tend to under-attribute (ignoring key factors).

For LLM application developers and product managers, FACET's findings have practical value: at the prompt engineering level, robust prompts can be designed for the model's attribution weaknesses (e.g., requiring "only list directly relevant factors"); at the human-machine collaboration level, strict manual review should be set up for tasks with low model faithfulness; at the model selection level, prioritize models with better attribution performance (even if other metrics are slightly inferior).

Current limitations of FACET: it mainly focuses on English scenarios, and its applicability to other languages needs to be verified; the four-probe design may not capture subtle biases in specific domains. Future directions include: expanding to multi-language scenarios, introducing dynamic adversarial testing, developing real-time attribution monitoring tools, and extending to visual-language joint reasoning scenarios.

FACET represents an important advancement in LLM evaluation methodology—shifting from focusing on "how many questions the model answers correctly" to "whether the model correctly knows why it answered correctly". This focus on attribution faithfulness reflects the AI community's emphasis on model transparency and interpretability, providing a valuable diagnostic tool for responsible AI deployment.