David-GRPO: A Low-Cost Reinforcement Learning Scheme for Small Models to Excel at Complex Reasoning

This post introduces the David-GRPO framework, which leverages budget-efficient reinforcement learning to enable small language models (under 10B parameters) to perform multi-hop reasoning. It provides a new approach for Agent development in resource-constrained scenarios, challenging the traditional view that small models lack strong reasoning capabilities.

While large models like GPT-4 and Claude 3 Opus excel in reasoning benchmarks, their high inference costs make them unsuitable for edge devices, real-time applications, or large-scale deployments. Traditional wisdom holds that small models (<10B parameters) have weak reasoning abilities, but David-GRPO aims to change this perception.

GRPO (Group Relative Policy Optimization) is an RL algorithm by DeepSeek that estimates advantage functions via intra-group relative comparison, eliminating the need for an independent reward model. David-GRPO builds on this with optimizations for multi-hop reasoning and budget efficiency:

• Dynamic reasoning path planning: Autonomous decision-making on information retrieval, stopping, and integration.
• Budget-aware training: Cost constraints to balance reasoning quality and resource consumption.
• Small model-specific architecture: Optimized training strategies for models under 7B parameters to avoid mismatches from large model techniques.

Multi-hop reasoning requires meta-cognition (awareness of knowledge boundaries) and flexible reasoning chains. Traditional methods use fixed retrieval-generate patterns, but David-GRPO uses RL to let models explore optimal reasoning-retrieval strategies, dynamically adjusting resource allocation for different problem difficulties.

David-GRPO controls costs (computation + external API calls) through:

• Early exit: Terminate reasoning when answer confidence is sufficient.
• Query selectivity: Distinguish necessary vs redundant external queries.
• Adaptive reasoning depth: Use shallow reasoning for simple problems and deep for complex ones, avoiding one-size-fits-all resource waste.

Experiments show optimized small models via David-GRPO can match or outperform unoptimized larger models. Key applications:

• Enterprise knowledge QA: Cross-department information integration.
• Intelligent customer service: Multi-system query (orders, inventory, logistics).
• Research assistant: Literature review and cross-paper concept association.
• Education辅导: Dynamic adjustment of explanation depth based on student knowledge.

Limitations:

• Requires domain-specific reward function design.
• RL sample efficiency issues need more interaction data.
• Currently focused on text reasoning; multi-modal expansion pending.

Future directions:

• Integration with tool learning.
• Enhanced online learning capabilities.
• Scaling to larger models.

David-GRPO embodies a pragmatic AI philosophy: prioritizing algorithm innovation over model scale expansion. It unlocks small models' potential for complex reasoning, offering cost-effective solutions for resource-limited teams, edge developers, and cost-conscious enterprises.