NVIDIA Nemotron Inference Challenge: A Two-Stage Training Methodology for Deterministic LoRA Fine-Tuning

This project proposes a LoRA fine-tuning solution for the NVIDIA Nemotron Inference Challenge. Its core innovation lies in the adoption of a two-stage training strategy and a fully deterministic data generation process. Training data is generated via independent scripts without relying on external teacher models, achieving reliable and reproducible training results.

NVIDIA Nemotron is a large language model optimized for inference tasks, excelling in mathematical reasoning, logical deduction, etc. However, it requires a well-designed fine-tuning strategy to transform into a domain-specific expert. This project presents a complete LoRA fine-tuning pipeline, with the core innovation being a fully deterministic data generation process that does not rely on external teacher models.

First Stage: Knowledge Injection and Methodology Construction

Target: Inject domain knowledge and establish a methodological framework Configuration: 1 epoch, LoRA fine-tuning, learning rate 1e-4, LoRA rank/alpha=32, training data phase1_train.csv

Second Stage: Chain-of-Thought (CoT) and Synthetic Data Enhancement

Target: Refine reasoning capabilities via CoT trajectories and synthetic data Configuration: 1 epoch, LoRA fine-tuning, learning rate 5e-5, initialize weights with first-stage adapter, LoRA rank/alpha=32, training data train_sft_phase2_75_10_15.csv

Advantages

1. Reproducibility: Same script generates consistent dataset
2. Cost control: No need for expensive API services

Data Split Strategy

75% supervised fine-tuning, 10% GRPO training, 15% evaluation, stored in splits_75_10_15.csv and config.json

Generation Scripts

1. Generate split files: make_splits.py splits hierarchically by ratio
2. Prepare phase1 data: prepare_phase1_training_dataset.py
3. Prepare phase2 data: prepare_phase2_sft_dataset.py generates training set with CoT trajectories

LoRA Configuration

Both stages use rank=32 and alpha=32, balancing parameter efficiency and expressive power

Learning Rate Scheduling

First stage uses 1e-4 to accelerate knowledge absorption; second stage uses 5e-5 for fine adjustment

Data Validation Mechanism

Use --validate-only command in train_sft.py and train_grpo.py to check data format and configuration integrity, quickly identify issues without loading the base model

Cautious Use of Synthetic Data

• Added after splitting, not involved in training/validation/test splits
• Audited real splits do not contain synthetic data
• All synthetic data is confirmed via validation scripts

Differences from External Teacher Models

No use of external teacher models like GPT-4 to generate CoT; training trajectories come from deterministic scripts and selected CSV files, improving controllability and interpretability

1. Domain Adaptation: Adapt general models to specific domains like healthcare and law
2. Reasoning Enhancement: Improve explicit reasoning performance for strong reasoning tasks like math and programming via CoT training
3. Resource-Constrained Environments: LoRA fine-tuning reduces computational resource requirements
4. Reproducible Research: Deterministic process provides a foundation for reproducible academic research

Technical Highlights

1. Fully deterministic: Entire process is reproducible
2. Phased strategy: Separating goals reduces training complexity
3. Independent data generation: No external API dependency
4. Strict validation: Multiple mechanisms prevent training failures
5. Efficient LoRA fine-tuning: Achieve professional results with limited resources

Conclusion

This project demonstrates a pragmatic and rigorous fine-tuning methodology with clear code structure and transparent processes. It is an excellent learning case for large model fine-tuning technology, suitable for practical projects and academic research.