RRRL: Research on Best-of-N Reasoning Optimization via Structured Reasoning and Step-Aware Selection

Core Overview of the RRRL Project

The RRRL project focuses on reasoning optimization for large language models. It combines structured chain-of-thought generation and step-aware reward model selection with dual-head language models to improve Best-of-N reasoning performance, covering experimental design and validation frameworks for classification and mathematical reasoning tasks.

Keywords: Large Language Model, Reasoning Optimization, Reward Model, Chain of Thought, Best-of-N Sampling, Structured Reasoning, Step-Aware Evaluation, Dual-Head Model

Original Author: wenqi-l, Source: GitHub (https://github.com/wenqi-l/rrrm), Release Date: 2026-05-28

Current large language models face two core issues in reasoning:

1. The reasoning process lacks structural constraints, making intermediate steps difficult to track and verify;
2. Traditional reward models only focus on the final answer, ignoring the quality of intermediate steps and failing to distinguish between high-quality reasoning paths and low-quality paths that happen to be correct.

The Best-of-N sampling strategy improves accuracy by generating multiple candidates and selecting the best one, but the selector's lack of fine-grained reasoning understanding leads to a bottleneck. The RRRL project addresses this pain point by exploring an optimization path that combines structured reasoning generation with step-aware reward models.

Structured Chain of Thought (Structured CoT)

Traditional CoT is free text, while Structured CoT requires outputting steps in a clear format (e.g., marked with "### Step N:"), bringing three benefits: parseability, verifiability, and debuggability.

Step-Aware Reward Model

Unlike models that only evaluate the final answer, this model scores each reasoning step, enabling it to identify logical flaws (even if the answer is correct), distinguish the quality of reasoning paths, and provide rich supervision signals.

Dual-Head Language Model (DHRD-style)

It is divided into a head for generating reasoning processes and a head for evaluation and selection. The decoupled design allows separate optimization of the generator and selector.

A 2×2 factorial experiment is used to verify component contributions:

Condition	Unstructured CoT	Structured CoT
Outcome-only RM	A (Baseline)	B (Structured only on generator side)
Step-aware judge	C (Structured only on RM side)	D (Proposed scheme = combination)

Core logic: Compare A→B, A→C, and the relationship between D and B/C to quantify the effect of each factor. D must outperform both B and C to prove the effectiveness of the combined scheme.

Generator Configuration

• Mathematical tasks: Qwen2.5-3B-Instruct (can be upgraded to 7B/8B)
• Classification tasks: Qwen2.5-1.5B-Instruct
• Sampling parameters: temperature=0.8, top_p=0.95, generate 8 candidates per question

Prompt Design

• Unstructured: Solve the problem step by step. Put your final answer in \boxed{}.
• Structured: Solve the problem using numbered steps. Begin each step with "### Step N:". Put your final answer in \boxed{}.

Evaluation Pipeline

Three-stage design (supports resuming from breakpoints):

1. Generation phase: Output generations/*.jsonl
2. Scoring phase: Call judge and outcome RM to output scores/*.jsonl
3. Evaluation phase: Calculate metrics such as best-of-{1,2,4,8} and oracle@8

Dataset Selection

• Mathematical reasoning: MATH500 (150-question subset for rapid iteration, full set for formal use; replace with AIME-2024 if difficulty is insufficient)
• Classification tasks: BoolQ development set (300 to full) + optional ARC-Challenge

Key Metrics

• Primary metric: best-of-8 accuracy under each condition
• Auxiliary metrics: best-of curve, oracle@8, structured format compliance rate
• Consistency metric: κ coefficient between judge and oracle

Go/No-Go Criteria

• Go: Condition D improves math accuracy by ≥3 points and classification accuracy by ≥2 points compared to condition A, and D ≥ B and C
• Sanity check: oracle@8 − pass@1 >5%, otherwise replace the task or generator

Risk	Early Signal	Mitigation Strategy
Prompt fails to stably induce structured output	Low format compliance rate in first 100 outputs of generation phase	Fine-tune with LoRA SFT on NuminaMath dataset
Judge model has excessive noise	Manually check 20 (response, judge-JSON) pairs	Replace with a larger judge model (14B) or use API
oracle@8 ≈ pass@1 (no room for selection)	Calculate immediately after generation phase	Replace with AIME-2024 or a weaker generator

The RRRL project provides a systematic framework to verify the effect of structured reasoning and step-aware evaluation in Best-of-N reasoning. If Phase 1 meets the Go criteria, it will proceed to Phase 2:

1. Step-level contrast rubric synthesis: Push log-prob margins to step granularity
2. DHRD dual-head SFT fine-tuning: Fine-tune on reasoning paths with subproblem annotations
3. Collaborative-critical subproblem generator: Train step-level contrast pairs using DPO

The progressive design ensures the reliability of basic assumptions and reserves space for subsequent methodological innovations.