Reasoning Learning of Large Language Models Under Weak Supervision: Saturation Dynamics and Faithfulness Mechanisms

This article focuses on the reasoning learning of large language models under weak supervision. The core findings include: 1. Reward saturation dynamics in RLVR training determine the generalization ability of models; 2. Reasoning faithfulness is a key pre-training attribute for predicting the success of weak supervision learning; 3. The combination of Supervised Fine-Tuning (SFT) and continuous pre-training can effectively improve reasoning generalization under weak supervision.

Large language models have made significant progress in reasoning ability through Reinforcement Learning with Verifiable Rewards (RLVR), but constructing high-quality reward signals is becoming increasingly difficult. Weak supervision learning refers to learning without complete and precise supervision signals (such as data scarcity, noisy rewards, and proxy rewards), which is of great significance for expanding the application boundaries of models.

The research team tested the model in three weak supervision scenarios:

1. Data Scarcity Scenario: The model is trained only on extremely limited labeled samples, focusing on generalization rather than memorization;
2. Noisy Reward Scenario: Simulate imperfect reward signals to test the model's robustness to noise;
3. Self-Supervised Proxy Reward Scenario: The model learns relying on self-generated proxy signals without external verification.

1. Reward Saturation Dynamics: Models that generalize successfully exhibit an extended pre-saturation phase (training rewards and downstream performance improve synchronously), while models that saturate quickly are prone to overfitting;
2. Reasoning Faithfulness: The degree to which intermediate steps logically support the final answer is a reliable indicator for predicting the success of weak supervision learning, and mere output diversity cannot predict generalization.

1. Key Role of SFT: Supervised fine-tuning on explicit reasoning trajectories helps the model establish a framework for problem decomposition and step-by-step reasoning, laying the foundation for subsequent weak supervision learning;
2. Amplification Effect of Continuous Pre-Training: Continuous pre-training on domain data provides rich domain knowledge, which synergizes with SFT to improve weak supervision reasoning performance.

The study applied the combination of SFT and continuous pre-training to the Llama3.2-3B-Base model, which originally failed in all three weak supervision scenarios, and successfully achieved generalization in all scenarios, proving that small models can also acquire weak supervision reasoning ability through reasonable intervention.

1. Attach importance to the cultivation of reasoning faithfulness, and focus on the logical consistency of the reasoning process during the pre-training phase;
2. Monitor the reward saturation curve—extending the pre-saturation phase is more important than pursuing high training rewards;
3. Adopt phased training: first use SFT to establish a reasoning framework, then conduct continuous pre-training to expand knowledge, and finally perform weak supervision reinforcement learning.

Future explorations can include: automatic evaluation and improvement of reasoning faithfulness, development of fine-grained saturation dynamics monitoring tools, research on the impact of other pre-training attributes; expansion to complex tasks such as multimodal reasoning and code generation to verify the universality of the method.