ScaleLogic: Unveiling the Power Law of Reinforcement Learning Training for Long-Range Reasoning

The ScaleLogic framework, via a controlled logical reasoning environment, finds that RL training computation and reasoning depth follow a power law relationship, where the richness of logical expressiveness determines the power law exponent; more expressive training setups can lead to a performance improvement of up to 10.66 points. This study provides a new perspective for understanding the scaling laws of large models' long-range reasoning capabilities.

Open Questions

In recent years, RL has been widely used to enhance large models' reasoning capabilities, but a core question remains unresolved: How does RL training scale with task difficulty?

Existing Limitations

Lack of controlled, scalable evaluation environments: Real-world reasoning tasks (e.g., math competition problems) are difficult to control for difficulty precisely, have high evaluation costs, and cannot systematically study the relationship between training computation and task difficulty.

ScaleLogic Framework Design

Provides two independently controlled difficulty dimensions:

1. Reasoning Depth: Number of planning steps required to complete a proof
2. Logical Expressiveness: Supports logical systems from simple to complex (implication, conjunction, disjunction, negation, universal quantifiers)

Methodological Highlights

• Independent Variable Control: The synthetic environment can manipulate reasoning depth and expressiveness independently
• Reproducibility: Synthetic task generation mechanism ensures experimental reproducibility
• Cross-Method Validation: The power law relationship applies to multiple RL methods like PPO and GRPO
• Curriculum Learning Gain: Transitioning from simple to complex tasks improves training efficiency

Core Findings

Training computation T and reasoning depth D follow a power law relationship: T ∝ D^γ (with R² > 0.99), and the power law exponent γ increases monotonically with improved expressiveness (from 1.04 to 2.60)

Impact of Expressiveness

1. Training Efficiency: Simple implication logic (γ≈1.04) shows linear growth, while first-order logic (γ≈2.60) shows superlinear growth
2. Downstream Transfer: Models trained with high expressiveness achieve up to 10.66 points improvement on math/general reasoning benchmarks and perform better under the same computation budget

Experimental Details

• Task: Construct logical proof sequences with automatic correctness verification
• Model: Transformer architecture (millions to billions of parameters)
• Hyperparameters: Systematic exploration of learning rates, batch sizes, etc., to ensure generalizability

ScaleLogic is the first to reveal the power law relationship between RL training computation and reasoning depth in a controlled environment, expanding the understanding of neural network Scaling Laws; the core findings challenge the intuition of 'simple tasks first', indicating that the 'quality' and 'quantity' of training content are equally important. This framework provides tools for predicting and optimizing the scaling of reasoning capabilities.

1. Training Data Selection: Prioritize high-expressiveness, challenging tasks rather than a large number of simple tasks
2. Evaluation Benchmark Design: Cover different levels of expressiveness to avoid underestimating the model's true capabilities
3. Computation Resource Allocation: Optimize training configurations based on the power law exponent γ, combined with target reasoning depth and budget

Limitations

There is a gap between synthetic logical tasks and real-world reasoning tasks (e.g., mathematical proofs, scientific reasoning)

Future Directions

• Validate the findings' applicability in complex real-world domains
• Explore universal methods for quantifying expressiveness
• Study optimal curriculum design strategies
• Combine model architecture innovations (reasoning modules, memory mechanisms) to improve performance