CCR.GB: Evaluating the Compositional Causal Reasoning Capabilities of Large Language Models

Title: CCR.GB: Evaluating the Compositional Causal Reasoning Capabilities of Large Language Models This article introduces the CCR.GB benchmark framework, which aims to systematically evaluate the performance of large language models (LLMs) on compositional causal reasoning tasks. Based on Judea Pearl's causal hierarchy (three levels: association, intervention, counterfactual), this benchmark fills the gap where existing benchmarks fail to capture complex causal structures. The project is maintained by kun-zero162, with the source code hosted on a GitHub repository, and the related paper is published at ICML 2025.

Large language models perform well in various reasoning tasks, but the core question is whether they truly understand causal relationships rather than just imitating statistical correlations. Causal reasoning is crucial for fields like healthcare and policy-making. Existing benchmarks are often simplified to binary classification or multiple-choice questions, which cannot handle the complex causal structures in the real world. The CCR.GB benchmark is proposed to provide a comprehensive framework for evaluating LLMs' capabilities in complex causal scenarios.

CCR.GB is designed based on Pearl's causal hierarchy:

1. Association Level: Focuses on the situation of Y when X is observed (statistical correlation);
2. Intervention Level: Answers the question "What would happen to Y if we do X?" (considering causal structure and confounding factors);
3. Counterfactual Level: Handles hypothetical questions (constructing a complete world model). The unique feature of this benchmark is that it requires models to reason in compositional scenarios, i.e., complex interactions between multiple causal variables and intervention points.

Causal Graph Generation

Directed Acyclic Graphs (DAGs) are used to represent causal relationships. Each test case is based on a randomly generated DAG containing multiple Binary Causal Variables (BCCs), and nodes are randomly assigned labels to separate semantics from reasoning capabilities.

Probability Calculation and Verification

100,000 simulations are performed using Structural Causal Models (SCMs) to calculate key metrics: Global PNS, Local PNS, and compositional reasoning verification (whether the global effect equals the product of local effects).

Experiment Reproduction

Two notebooks are included:

• experimental_results.ipynb: Reproduces key experimental results from the paper (validity vs. consistency scatter plots, CCT reasoning profiles, path length error scaling);
• verification.ipynb: Verifies causal DAG construction, prompt generation, and Theorem 5.1.

Key Findings

• Verification of Theorem 5.1: In serial cut-point structures, the global PNS equals the product of local PNSs. Experimental deviations stem from finite sample sampling (RAE is approximately 19%-21%);
• Cross-topic Consistency: DAG structures across different topics (e.g., FluVaccine, FlowerGarden) are matched, confirming that the benchmark can isolate semantics from reasoning capabilities;

Model Evaluation Results

Models such as o1, GPT-4o+CoT, and Llama3 are evaluated. It is found that state-of-the-art models still have gaps in compositional causal reasoning, especially their performance at the counterfactual level is significantly lower than at the intervention and association levels.

Application Significance

• Guiding model development: Diagnosing weaknesses in causal reasoning;
• Evaluation in high-risk fields: Safety assessment before deployment in healthcare, law, etc.;
• Promoting causal AI research: Standardized benchmarks facilitate fair comparisons;
• Educational value: Notebooks and visualizations serve as teaching cases.

Limitations

• Binary variable limitation;
• Simplified scenarios;
• High computational cost.

Future Directions

• Extending to multimodal causal reasoning;
• Introducing temporal causal structures;
• Efficient approximate reasoning methods;
• Combining neural and symbolic approaches to enhance capabilities.

CCR.GB is an important advancement in evaluating the causal reasoning capabilities of LLMs. By covering Pearl's hierarchy and compositional complexity, it reveals the capability boundaries of current models. For practitioners, it is necessary to carefully evaluate the causal understanding capabilities of LLMs rather than just focusing on surface task performance. The open-source implementation and documentation of this project provide valuable resources for the causal AI community and promote the development of the field.