DeltaPrompts: Breaking the Zero-Increment Trap in Multimodal Distillation, Achieving 15% Performance Improvement

This article reveals that 69% of prompts in multimodal distillation are 'zero-increment' invalid samples. It proposes the DeltaPrompts dataset (containing 200,000 synthetic high-divergence reasoning problems) that selects high-value prompts via answer divergence, achieving a 15% performance improvement in benchmark tests. The core innovation lies in returning to the first principles of distillation, quantifying prompt value using answer divergence, and proactively generating targeted training data.

Visual-Language Models (VLMs) have strong reasoning capabilities but high computational costs. Knowledge distillation is a key technology for model compression. However, current distillation prompt selection lacks systematic quality screening, relying mostly on heuristic rules or aggregation of existing datasets, leading to insufficient training data quality that affects performance.

The study found that 69% of prompts in standard datasets are 'zero-increment' samples—where the answer distributions of the teacher and student models are identical, with zero divergence and no learning signal. The consequences include: low learning efficiency (wasting resources on already mastered content), rapid performance saturation, and falling into the 'data scale trap' (mistakenly believing more data is needed instead of high-quality data).

1. Answer Divergence (Δ)：Measures prompt value—high-divergence prompts expose the capability gap between teacher and student models, while low-divergence prompts have no learning signal; 2. DeltaPrompts Dataset: Synthesized in four stages: seed data utilization → failure pattern identification → targeted synthesis → divergence screening. It covers three types of reasoning tasks: charts, documents, and perception-centric, with a total of 200,000 high-divergence problems.

1. Online Distillation: DeltaPrompts significantly improved performance when training target teacher-student pairs; 2. Cross-Model Transfer: Effective when transferred to new model families; 3. Offline Fine-Tuning: Endows non-reasoning models with reasoning capabilities. In 10 benchmark tests (chart QA, document VQA, etc.), it achieved a 15% relative performance improvement on high-baseline models (e.g., Qwen3-VL-8B-Thinking).

Insights include: 1. Data quality takes precedence over scale (200,000 high-divergence prompts are better than millions of unscreened data); 2. Dynamic data strategy (continuously monitor student gaps to generate supplementary data); 3. Answer divergence provides a new tool for evaluating distillation data quality. This study promotes a paradigm shift from passive use of datasets to proactive generation of high-quality training data.

Limitations: Synthetic data lacks real-world noise/ambiguity, limited domain coverage (no validation on video/3D reasoning), and high generation cost. Future directions: Optimize divergence calculation efficiency, expand to multimodality (audio/video), online dynamic data generation, and improve prompt quality by combining human feedback.