Blanc: Evaluating Abductive Reasoning Capabilities of Large Language Models Using Deductive Proofs

The Blanc project aims to address the challenges large language models (LLMs) face in abductive reasoning (inference to the best explanation) by generating defeasible sets via deductive proofs to evaluate LLMs' abductive reasoning capabilities. Abductive reasoning is a common yet most difficult-to-evaluate type of reasoning in daily decision-making and scientific discovery; existing methods struggle to systematically assess its quality, and Blanc provides an innovative framework for this purpose.

Human reasoning is divided into three types: deductive, inductive, and abductive. Among them, abductive reasoning (inference to the best explanation) is the most common but hardest to evaluate. LLMs face challenges in abductive reasoning such as difficulties in returning to the best explanation (hard to select the optimal explanation, reliance on common explanations from training data), complex evaluation (multiple reasonable explanations, dependence on background knowledge), and limitations of existing methods (multiple-choice accuracy, end-to-end tasks, subjective manual evaluation).

Blanc transforms the evaluation of abductive reasoning into a deductive reasoning problem: generate candidate explanations from observed occurrences, construct a deductive proof for each explanation, define a set of defeasible hypotheses based on the proof, then score and compare them. Defeasible logic is a non-monotonic logic that allows new information to overturn conclusions, aligning with the essence of abductive reasoning (explanations are based on current best knowledge and can be overturned by new evidence).

Deductive Proof Generation: Build a domain knowledge base (axioms, rules, background knowledge), perform backward search for reasoning chains, and analyze hypotheses and dependencies in the proof; Defeasible Set Construction: Classify hypotheses (necessary, auxiliary, default), sort by priority, and evaluate defeasibility; Scoring Mechanism: Score from multiple dimensions including explanatory power (coverage of phenomena), conciseness (number of hypotheses, length of reasoning chain), consistency (compatibility with background knowledge), and defeasibility (sensitivity to additional information).

Blanc can be used for: 1. Model capability evaluation (diagnose weaknesses, compare models, track iterations); 2. Training data screening (identify high-quality samples, filter data with error patterns); 3. Prompt engineering optimization (evaluate the impact of prompt templates, develop few-shot examples); 4. Scientific discovery assistance (assess AI-generated hypotheses, compare competing theories, identify key hypotheses).

Blanc has the following limitations: 1. Knowledge formalization barriers (requires formalization of domain knowledge, not all domains have complete ontologies); 2. Computational complexity (high cost of proof search and set construction); 3. Explanation diversity (need to avoid over-penalizing reasonable alternative explanations); 4. Domain specificity (the general framework needs to adapt to differences across domains).

Future directions include: 1. Automatic knowledge acquisition (extract formalized knowledge from unstructured text); 2. Approximate reasoning (scalable algorithms to improve efficiency); 3. Human-machine collaborative evaluation (automatic screening + manual processing of complex cases); 4. Cross-domain migration (reduce reliance on domain experts).