Learning Dense Reasoning Reward Models from Expert Demonstrations via Inverse Reinforcement Learning

This study explores the use of Inverse Reinforcement Learning (IRL) to extract implicit reasoning reward signals from expert demonstrations, build a dense reward model that can evaluate the quality of reasoning processes, address the reward sparsity problem in LLM reasoning training, and promote models to shift from imitating expert answers to learning expert thinking processes.

Large language models have made significant progress in reasoning tasks, but core challenges remain in training complex multi-step reasoning: Traditional supervised fine-tuning can only imitate expert final answers and cannot capture decision-making logic in the reasoning process; Reinforcement learning faces reward sparsity—for tasks like mathematical proof and code generation, binary feedback is only available after completing the entire reasoning chain, and learning signals for intermediate steps are weak.

The technical framework includes three core components:

1. Expert demonstration collection: Record process-supervised data of complete reasoning paths (including exploration, trial, verification, and correction全过程);
2. Reward model learning: Use IRL algorithms to infer the reward function from expert demonstrations, making expert trajectories optimal under this function;
3. Dense reward modeling: Generate step-level dense rewards to provide fine-grained quality evaluation for each step of the reasoning chain.

The study's technical innovations include:

1. Maximum entropy IRL extension: Model expert behavior uncertainty to enhance the robustness and generalization of the reward function;
2. Hierarchical reward structure: Simultaneously model multi-scale reward signals at step, paragraph, and task levels to capture reasoning hierarchy;
3. Computational efficiency optimization: Fast solving algorithm based on approximate dynamic programming to support large-scale reasoning task applications.

Validated effectiveness across multiple reasoning benchmarks: Mathematical reasoning (GSM8K, MATH datasets) showed significant improvement over sparse reward baselines; Code generation (HumanEval, MBPP) helped models better understand structure and execution logic; Logical reasoning enabled precise error localization. Additionally, sample efficiency is high—good performance can be achieved with fewer interaction rounds.

1. Process supervision outperforms outcome supervision: The reasoning process of expert demonstrations is more valuable than final answers; future data collection should focus on process quality annotation;
2. Key to reward engineering: IRL provides a method to automatically learn reward functions from data;
3. New human-AI collaboration mode: Humans provide high-quality demonstrations, and AI learns evaluation standards to improve itself.

Limitations: Highly dependent on expert demonstration quality (reward functions inherit deviations if demonstrations are biased); IRL training is computationally intensive; cross-domain generalization ability needs verification. Future directions: Combine online learning to dynamically update reward models; explore reward learning with no/few demonstrations; integrate with training paradigms like DPO/KTO.

This study learns dense reasoning reward models from expert demonstrations via inverse reinforcement learning, providing an innovative solution to the reward sparsity problem in LLM reasoning training. It proposes a new idea of enabling models to think like experts rather than just imitate answers, and the open-source code is expected to be applied and verified in a wider range of reasoning tasks.