Inference-Time Computational Optimization for Reasoning Models: A Comparative Study of SFT and GRPO Fine-Tuning Strategies

This study focuses on the impact of different inference-time computational strategies (majority voting, Best-of-N, PRM-guided beam search, budget enforcement) on reasoning accuracy under a fixed inference computational budget, and compares the effect differences between two fine-tuning methods: SFT and GRPO. The core question is: Does the optimal inference-time strategy depend on the fine-tuning method? The study reveals the interaction effect between fine-tuning methods and inference-time strategies, providing references for the design of efficient reasoning systems.

Research Background and Core Questions

In recent years, large language models have shown improved performance in reasoning tasks (mathematics, code, logic), but the inference cost has increased dramatically. How to maximize accuracy within a limited computational budget has become a key challenge for deployment.

Inference-time computational strategies improve accuracy at a low additional cost by generating and filtering multiple candidate answers during the inference phase.

Core questions: Which inference-time strategy achieves the highest accuracy under a fixed budget? Does the choice of optimal strategy depend on the fine-tuning method (SFT vs GRPO)?

Overview of Inference-Time Computational Strategies

Four mainstream strategies are evaluated:

1. Majority Voting

A simple integration strategy: generate multiple independent answers and select the one with the highest frequency. Advantages: easy to implement without additional models; Disadvantages: poor performance when the correct answer does not account for the majority.

2. Best-of-N with PRM

Generate N candidates and select the one with the highest score using a Process Reward Model (PRM). PRM evaluates the rationality of the reasoning process, making it more reliable for complex tasks.

3. PRM-Guided Beam Search

Maintain a candidate beam at each step, use PRM to guide the search direction, and prioritize exploring promising paths. It uses the budget more effectively than independent sampling but is complex to implement.

4. Budget Enforcement

Dynamically adjust the generation length/thinking depth to control computational consumption, balancing efficiency and quality.

Comparison of SFT and GRPO Fine-Tuning Paradigms

Supervised Fine-Tuning (SFT)

A mainstream method that learns task patterns through supervised learning on high-quality annotated data. Advantages: stable training, fast convergence, directly learning expert thinking; Disadvantages: limited generalization ability (out-of-distribution problems).

GRPO Fine-Tuning

Based on reinforcement learning, it optimizes strategies to maximize rewards. It does not directly learn fixed patterns but explores diverse problem-solving strategies; Challenges: unstable training, reward hacking.

Research Findings and Insights

Core finding: There is a significant interaction effect between fine-tuning methods and inference-time strategies.

• For SFT models: Majority voting can achieve considerable accuracy improvement (consistent answer patterns).
• For GRPO models: Complex PRM-guided strategies are better (high answer diversity requires fine-grained filtering).

Impact of budget size: Simple strategies have high cost-effectiveness when the budget is small; complex search strategies can better utilize resource value when the budget is large.

Practical Application Significance

It provides direct guidance for the deployment of large reasoning models:

• Developers need to select inference strategies based on the model training method, rather than considering the strategy in isolation.
• Resource-constrained scenarios: Find the strategy that achieves the maximum accuracy improvement with the minimum computational overhead.
• Extreme performance scenarios: Understand the upper limits and boundaries of strategies to design efficient reasoning systems.

Future Research Directions

• Design adaptive inference-time strategies: dynamically adjust computational allocation based on problem difficulty.
• Build hybrid reasoning frameworks: combine the advantages of multiple strategies.
• Adapt to model capability improvements: evolve inference-time strategies to match new model characteristics.

Conclusion

Against the backdrop of enhanced reasoning capabilities of large language models, efficient utilization of computational resources is a key issue. This study provides empirical evidence and decision-making references for efficient reasoning systems by systematically comparing the combined effects of inference-time strategies and fine-tuning methods. We look forward to the emergence of more intelligent and efficient reasoning paradigms.