From Human Explanations to AI Reasoning: Exploring New Paths of Knowledge Distillation in Natural Language Inference

This article addresses the "black box" problem and shallow reasoning limitations of Natural Language Inference (NLI) models. It studies how to transfer the reasoning capabilities of human explanations and LLM chain-of-thought to efficient encoder models via knowledge distillation, compares their effects, and explores hybrid strategies, providing a new direction for interpretable NLI.

The NLI task requires judging the logical relationship between sentences. However, existing pre-trained models (such as BERT, DeBERTa) lack interpretability in their decisions, often relying on shallow lexical heuristics (e.g., judging entailment based on identical vocabulary), and exhibit "pseudo-understanding" in complex reasoning samples.

Based on the DeBERTa architecture, four training settings are designed: 1. Baseline model (using only premise-hypothesis pairs); 2. Human explanation distillation (multi-task learning to generate human explanations); 3. LLM-CoT distillation (training with chain-of-thought generated by GPT-4); 4. Hybrid distillation (combining both).

Experiments show: 1. Reasoning supervision significantly improves performance on complex samples; 2. LLM chain-of-thought outperforms human explanations in systematization and scalability; 3. Distillation strategies (e.g., multi-task learning) affect the results.

Reasoning capabilities can be transferred to small models via distillation, and LLM chain-of-thought can complement human explanations, providing a feasible path for interpretable NLI in resource-constrained scenarios.

1. Re-examine the relationship between human annotations and synthetic data; 2. Need stricter evaluation metrics to distinguish real reasoning from surface imitation; 3. Transfer of reasoning capabilities supports trustworthy AI.

Limitations: Validated only on English SNLI, and the quality of LLM chain-of-thought depends on prompts; Future: Explore advanced distillation techniques, expand to other reasoning tasks, and optimize evaluation metrics.