SARL: Label-Free Reinforcement Learning via Rewarding Reasoning Topology

This article introduces SARL (Structure-Aware Reinforcement Learning), a training framework for reasoning models that requires no labels or real rewards. Traditional reinforcement learning methods (e.g., RLVR) rely on verifiable answers, limiting their application to closed-domain tasks. Moreover, overemphasis on outcomes can lead models to take shortcuts. SARL shifts the supervision focus to the structure of reasoning paths: by constructing reasoning graphs and rewarding their small-world topological properties (local clustering + global reachability), it achieves significant performance improvements in both mathematical and open-ended tasks.

Reinforcement learning methods (e.g., RLVR) have achieved success in closed-domain tasks like mathematics and coding, but they require verifiable correct answers, making them inapplicable to open-ended domains such as creative writing and ethical reasoning (where answers are ambiguous or subjective). Additionally, overemphasis on outcomes may cause models to learn shortcuts, lacking generalizable reasoning abilities; their reasoning trajectories have no effective constraints, leading to fragile paths.

Construction of Reasoning Graphs

Extract reasoning graphs from the model's intermediate thinking steps. Nodes represent reasoning states/subgoals, edges represent transition relationships, which can capture branches, loops, and hierarchical structures (different from linear chain-of-thought).

Small-World Topological Reward

Drawing on complex network theory, reward reasoning graphs that simultaneously exhibit high local coherence (logical connection between adjacent steps) and high global efficiency (no redundant paths).

Label-Free Training Paradigm

1. The model generates responses with intermediate steps; 2. Extract reasoning graphs; 3. Calculate topological features; 4. Optimize the strategy using topological quality as a reward, breaking free from dependency on labels.

Experiments on the Qwen3-4B model:

• Mathematical tasks: PPO algorithm improved by 9.1%, GRPO by 11.6% (surpassing traditional RL methods even without real rewards);
• Open-ended tasks: PPO improved by 34.6%, GRPO by 30.4% (traditional RLVR cannot be applied here);
• Training dynamics: Lower KL divergence (stable learning without catastrophic forgetting), higher policy entropy (maintaining exploration ability).

Paradigm Shift

Shifting focus from outcomes to reasoning processes, similar to education moving from exam-oriented to quality-oriented education, fostering correct thinking methods.

Cross-Domain Generalization

Not relying on domain-specific answers, reasoning abilities can be transferred to scientific reasoning, logical puzzles, etc.

Neuroscience Connection

Inspired by the functional organization of the human brain; future work can combine neuroscientific findings to enhance models.

• Reasoning graph extraction: The accuracy of extracting structured reasoning graphs from free text needs improvement;
• Computational overhead: Calculation of topological features increases resource consumption; training efficiency for ultra-large-scale models needs optimization;
• Method combination: Future work can explore mixing with outcome-based methods to form a more powerful paradigm.

SARL breaks through the limitations of traditional RLVR in open-ended domains, cultivating generalizable reasoning abilities by rewarding reasoning topology. In the pursuit of general intelligence, teaching models 'how to think' is more critical than 'what to think', and SARL provides technical support for this.