LongTraceRL: Long-Context Reasoning Learning Based on Search Agent Trajectories and Scoring Rewards

Abstract: LongTraceRL addresses the challenges of handling distracting information and process supervision in long-context reasoning by constructing hierarchical distracting documents and using entity-level scoring rewards, achieving excellent performance across multiple benchmarks. Keywords: Long-context reasoning, reinforcement learning, process supervision, knowledge graph, search agent, reward design, multi-hop reasoning, RLVR Core Insights: LongTraceRL targets issues like model attention dispersion and information omission in long-context reasoning. It innovatively uses search agent trajectories to construct hierarchical distractors and designs entity-level scoring rewards to achieve fine-grained process supervision, significantly enhancing the model's reasoning ability in complex scenarios.

Problem Background: Core Challenges of Long-Context Reasoning

Long-context reasoning is one of the core challenges faced by large language models. Although modern LLMs have expanded their context windows to millions of tokens, their ability to locate key information and integrate scattered evidence is limited, as shown in:

1. Attention Dispersion: Distracted by irrelevant information, unable to focus on key paragraphs
2. Information Omission: Failing to notice details critical to the answer
3. Spurious Association: Incorrectly linking irrelevant information to the question
4. Reasoning Chain Breakage: Losing logical connections between intermediate steps in multi-hop reasoning

These problems stem from information integration that humans consider 'obvious', but for models, it is a complex skill that requires explicit learning.

Limitations of Existing Methods: Shortcomings of RLVR in Long-Context Reasoning

Reinforcement Learning with Verifiable Rewards (RLVR) has great potential in reasoning tasks, but existing methods have two key limitations:

Limitation 1: Low-Confusion Distractors

Existing training data often uses random sampling or single searches to build distracting documents, resulting in low-confusion distractors that models can easily identify as irrelevant. However, real-world distractors are more deceptive (e.g., surface-relevant but actually irrelevant, containing partially relevant information but insufficient to answer the question).

Limitation 2: Sparse Outcome-Oriented Rewards

Using only final answer correctness as the reward signal leads to:

• No supervision for intermediate steps
• Reward hacking (models get correct answers through wrong reasoning)
• Inability to distinguish differences in reasoning quality among correct answers

Analogy: A teacher only tells students they 'passed' without pointing out specific mistakes or improvement directions.

Core Innovation 1: Hierarchical Distractor Construction Based on Search Agent Trajectories

Multi-hop Question Generation via Knowledge Graph Random Walk

1. Select a starting entity from the knowledge graph
2. Randomly walk through multi-relation edges to the target entity
3. Convert the path into a natural language question
4. Record intermediate entities in the reasoning chain (gold entities)

Search Agent Trajectory Collection

Deploy a search agent to attempt answering multi-hop questions, and record its complete behavior trajectory (multiple searches, reading documents, citing evidence, generating answers) for constructing hierarchical distractors.

Hierarchical Distractor Design

• High-Confusion Distractors: Documents read but not cited by the agent (surface-relevant but insufficient to support the answer, highly deceptive)
• Low-Confusion Distractors: Documents in search results not opened by the agent (surface-relevant but not worth reading, easy to identify)

This design makes training data more challenging and simulates real-world complexity.

Core Innovation 2: Entity-Level Scoring Rewards and Process Supervision

Scoring Reward Design

Core Idea: Use gold entities in the reasoning chain as checkpoints to evaluate whether the model cites correct evidence at each step:

1. The gold answer for a multi-hop question contains a sequence of key entities
2. Parse the model's answer to extract cited entities
3. Calculate the entity matching degree (fine-grained feedback)

Positive-Only Strategy

• Scoring rewards are only applied to responses with correct final answers
• Responses with wrong answers only receive sparse correctness rewards (negative feedback)
• Scoring rewards are used to distinguish reasoning quality among correct answers

This prevents reward hacking and encourages quality competition among correct answers.

Advantages of Process Supervision

Compared to sparse outcome rewards, it provides:

1. Intermediate step feedback
2. Evidence quality evaluation
3. Encouragement for reasoning completeness
4. Interpretability (analysis of reasoning behavior)

Experimental Results: Consistent Improvement in Long-Context Reasoning Ability

Experimental Setup

• Model Scales: 4B, 7B, and 30B parameter reasoning LLMs
• Benchmarks: Five long-context reasoning benchmarks
• Baselines: Strong baselines like standard RLVR and supervised fine-tuning (SFT)

Core Results

• Consistent Performance Improvement: Outperforms strong baselines across all model scales and benchmarks, with significant average improvements
• Improved Reasoning Quality: More comprehensive, evidence-based reasoning, less missing key information, and less misled by high-confusion distractors
• Scale Generalization: Advantages are maintained across different model scales, with strong universality

Ablation Experiments

• Value of Hierarchical Distractors: Compared to random distractors, it improves robustness to real distractors; performance drops on hard samples without high-confusion distractors
• Value of Scoring Rewards: Compared to sparse rewards, it improves reasoning quality; the positive-only strategy effectively prevents reward hacking
• Synergy Effect: Better results when combining data construction and reward design

Application Value and Insights

Long-Context Applications

Directly applicable to:

• Document QA systems (key information localization in legal, medical, and scientific documents)
• Multi-hop search (complex queries integrating multiple information sources)
• Evidence chain construction (scenarios requiring clear reasoning basis)

Insights for RLVR Research

1. Data quality is crucial: The difficulty and authenticity of training data affect the upper limit of model capabilities
2. Value of process supervision: Fine-grained intermediate feedback is more effective than sparse outcome rewards
3. Prevent reward hacking: Strategies like positive-only maintain reasoning honesty

AI Safety Implications

Fine-grained supervision via scoring rewards helps:

• Improve interpretability (analyze reasoning paths)
• Detect error patterns (identify common error types)
• Alignment verification (verify consistency between reasoning processes and expectations)

Limitations, Future Directions, and Conclusion

Limitations

1. Search Agent Limitations: Current basic agents are not optimal; stronger agents may improve trajectory quality
2. Entity Recognition Accuracy: Scoring rewards rely on accurate entity recognition and alignment, which may fail in complex texts
3. Domain Generalization: Experiments are mainly on general knowledge QA; specific domains (medical, legal) need adaptation
4. Computational Cost: Trajectory collection and distractor construction require large computational resources

Future Directions

Optimize search agents, improve entity recognition accuracy, expand domain applications, and reduce computational cost

Conclusion

LongTraceRL significantly enhances long-context reasoning ability through innovative data construction and fine-grained process rewards, demonstrating the key role of training data design and reward engineering in RLVR. As LLMs are increasingly used in knowledge-intensive tasks, such methods will become a reliable technical foundation. We look forward to more progress to help AI find true knowledge in massive information.