Nemotron Reasoning Pipeline: Deterministic Solver and GRPO Training Scheme for Kaggle Competitions

The nemotron-reasoning-pipeline project introduced in this article is a complete training pipeline designed for the NVIDIA Nemotron Model Reasoning Challenge (Kaggle competition). It integrates deterministic solvers, supervised fine-tuning (SFT), and iterative GRPO reinforcement learning training, aiming to win the DGX Spark Award (a top-tier computing resource prize provided by NVIDIA).

The nemotron-reasoning-pipeline is a solution developed for the NVIDIA Nemotron Model Reasoning Challenge (Kaggle competition), which requires participants to use NVIDIA Nemotron series models to build AI systems with strong reasoning capabilities. The core objective of the project is to win the DGX Spark Award, so a systematic training pipeline integrating multiple advanced technologies is adopted.

The training pipeline consists of three key phases: Deterministic Solver → Supervised Fine-tuning (SFT) → Iterative GRPO Reinforcement Learning Training. The deterministic solver provides accurate answers for specific reasoning tasks such as mathematical problems and logic puzzles, with functions including: generating high-quality training data (for SFT), serving as a validation and evaluation benchmark (providing reward signals for RL), and adopting a hybrid strategy (symbolic solvers, search solvers, rule engines, and external tool integration).

The goal of the SFT phase is to enable the model to master basic reasoning patterns and output formats. The training data includes chain-of-thought examples (showing step-by-step reasoning trajectories), format specification examples (training outputs to meet requirements), and domain-specific examples (targeting competition tasks). The fine-tuning strategy is progressive: first domain adaptation (familiarizing with competition terminology and question types), then reasoning pattern learning (chain-of-thought and structured reasoning), and finally format alignment (meeting the requirements of the evaluation system).

GRPO (Group Relative Policy Optimization) is an improved RL algorithm proposed by NVIDIA, optimized for reasoning tasks: intra-group comparison (relative evaluation of multiple answers generated for the same problem within a group), no need for a value model (simplified implementation), and sparse reward handling (adapting to the characteristics of reasoning tasks). The iterative process is: generate candidate answers → evaluate quality using solvers/rules → calculate relative rewards → update model parameters → repeat iteration to achieve self-improvement.

Competition optimization strategies include: integrated reasoning (multi-answer voting and ranking to improve reliability), post-processing optimization (format standardization, answer extraction, consistency check), and computational efficiency optimization (gradient accumulation, mixed-precision training, inference caching). At the same time, the Nemotron model features are fully utilized: long context support (handling complex multi-step reasoning), tool usage capability (calling Python interpreter, etc.), and NVIDIA ecosystem optimization (TensorRT acceleration, multi-GPU parallelism, CUDA kernel optimization).

Project Significance: Demonstrates the value of multi-phase training (progressive complementary improvement), the potential of the GRPO algorithm in reasoning tasks (simplified process and effectiveness), and competition-driven technological innovation (integrating the latest technologies to solve problems). Future Outlook: Automatic solver discovery, exploration of more efficient RL algorithms, multi-modal reasoning expansion, and migration of competition technologies to production environments.