NVIDIA Nemotron Reasoning Challenge: A New Benchmark for Open-ended Reasoning Ability Evaluation

NVIDIA Nemotron Reasoning Challenge: A New Benchmark for Open-ended Reasoning Ability Evaluation

• Original Author/Maintainer: barada02
• Source Platform: GitHub
• Original Project Title: nvidia-nemotron-model-reasoning-challenge
• Original Link: https://github.com/barada02/nvidia-nemotron-model-reasoning-challenge
• Release/Update Time: 2026-06-05

NVIDIA's open-source reasoning challenge, based on the Nemotron-3-Nano-30B model and a new reasoning benchmark, invites the community to explore technical paths such as prompt engineering, data filtering, and lightweight fine-tuning to advance reproducible research on structured reasoning capabilities.

Reasoning benchmark tests are important tools for measuring the structured task capabilities of language models. However, current reasoning research is scattered across independent projects, using different datasets, prompt strategies, and evaluation settings, making direct comparisons difficult. NVIDIA launched this challenge to establish a shared benchmark environment and a unified baseline model, allowing technologies to be tested and compared under consistent conditions.

Core Objectives

Participants need to develop solutions to improve reasoning accuracy, evaluated on the new reasoning benchmark based on the Nemotron 3 Nano baseline.

Technical Routes

Explore paths such as prompt strategy optimization, data filtering and organization, synthetic data generation, reinforcement learning, LoRA lightweight fine-tuning, etc.

Base Model

Based on Nemotron-3-Nano-30B with LoRA adapters loaded (must include adapter_config.json).

Inference Engine

Use the vLLM high-performance inference engine.

Answer Extraction

Prioritize extracting answers from \boxed{}, if none, fall back to heuristics or the last numerical value.

Scoring Criteria

Predictions that exactly match the standard answer or are within the tolerance range are considered correct; scores are calculated based on the proportion of correct answers.

Reproducibility

Clear documentation (notebooks + reports) is a necessary condition for winning, supporting research reproducibility.

Collaborative Iteration

The open environment allows reusing and extending others' work, forming a positive cycle.

Open Workflow

Nemotron provides public models, datasets, and recipes, allowing participants to build and adjust freely.

Recommended participation path:

1. Familiarize yourself with the features of Nemotron-3-Nano-30B
2. Start with prompt engineering and explore advanced technologies
3. Use LoRA for rapid iteration to verify ideas
4. Record experimental processes and results in detail

The challenge provides a unified experimental field for reasoning research, open to both experts and researchers. Participants' contributions will be accumulated into reusable community knowledge, driving the development of AI open-ended research.