Zing Forum

Reading

NVIDIA Nemotron Reasoning Challenge Silver Medal Solution Review: Analysis of Delta-SVD Adapter Merging Technology

This article reviews the silver medal solution from the NVIDIA Nemotron Reasoning Challenge, which had 4355 participating teams. It focuses on analyzing the Delta-SVD adapter merging technology and its key role in enhancing the model's generalization ability.

LoRANemotronDelta-SVD模型集成推理能力Kaggle适配器合并泛化性
Published 2026-06-16 09:06Recent activity 2026-06-16 09:24Estimated read 5 min
NVIDIA Nemotron Reasoning Challenge Silver Medal Solution Review: Analysis of Delta-SVD Adapter Merging Technology
1

Section 01

Introduction / Main Floor: NVIDIA Nemotron Reasoning Challenge Silver Medal Solution Review: Analysis of Delta-SVD Adapter Merging Technology

This article reviews the silver medal solution from the NVIDIA Nemotron Reasoning Challenge, which had 4355 participating teams. It focuses on analyzing the Delta-SVD adapter merging technology and its key role in enhancing the model's generalization ability.

3

Section 03

Competition Overview and Results

NVIDIA Nemotron Model Reasoning Challenge is a Featured-level competition on the Kaggle platform with a prize pool of up to $106,388, attracting 4355 teams worldwide to compete. The core task of the competition is to train a LoRA adapter (rank ≤32) on the Nemotron-3-Nano-30B-A3B base model to maximize the model's accuracy on the "Alice's Wonderland" reasoning puzzle benchmark.

The solution achieved the following results:

Metric Value
Private Leaderboard Score (Final) 0.86
Medal Silver 🥈
Public Leaderboard Rank 480 / 4355 (Top 11%)
Number of Submissions 22
Score Evolution 0.54 → 0.86

It is worth noting that 1627 teams tied at 0.86 on the public leaderboard, making the competition extremely fierce. The real gap was in the private leaderboard performance—solutions with more stable generalization maintained 0.86 on the hidden test set, successfully advancing from the edge of the bronze medal on the public leaderboard to the silver medal zone.


4

Section 04

Solution Evolution Process

From the complete record of 22 submissions, the iterative path of the solution can be clearly seen:

5

Section 05

Phase 1: Foundation Building (v23-v59)

  • v23: Established a 30B rank32 LoRA baseline, first ensuring correct output format, private score 0.62
  • v53: Tried the hybrid replay solver-distill distillation route, with poor results (private score 0.55)
  • v59: Failed to reproduce the public 0.85 baseline, identified key gaps
6

Section 06

Phase 2: Breaking the Bottleneck (v60-v62)

  • v60: Faithfully reproduced Mohamed's 0.86 baseline, successfully crossing the 0.8 threshold
  • v62: The first solution to stably reach 0.86, becoming the cornerstone for subsequent merging
7

Section 07

Phase 3: Integration Optimization (v65-v79)

The core of this phase was exploring how to robustly merge multiple adapters:

  • v65: Introduced Hammad's SVD cleaning idea
  • v68: VNG Refine clean relay, but encountered overfitting issues (public score 0.86, private score 0.84)
  • v71-v75: Began to try adapter weighted merging and selective Delta-SVD merging
  • v78/v79: Three-way DSVD integration solution, with both public and private scores stably at 0.86, becoming the final solution
8

Section 08

Phase 4: Verification and Finalization (v84)

  • v84: Re-verification found that a single adapter was prone to inflated public scores and dropped private scores, which conversely proved the necessity of multi-way integration