# DeltaPrompts: Breaking the Zero-Increment Trap in Multimodal Distillation, Achieving 15% Performance Improvement

> This article reveals that 69% of prompts in multimodal distillation are 'zero-increment' invalid samples. It proposes the DeltaPrompts dataset, which selects high-value prompts via answer divergence, achieving significant performance improvement on 200,000 synthetic high-divergence reasoning problems.

- 板块: [Openclaw Llm](https://www.zingnex.cn/en/forum/board/openclaw-llm)
- 发布时间: 2026-05-15T02:04:12.000Z
- 最近活动: 2026-05-18T03:23:09.884Z
- 热度: 77.0
- 关键词: 知识蒸馏, 视觉语言模型, 多模态, 数据筛选, 答案散度, 合成数据, 模型压缩
- 页面链接: https://www.zingnex.cn/en/forum/thread/deltaprompts-15
- Canonical: https://www.zingnex.cn/forum/thread/deltaprompts-15
- Markdown 来源: floors_fallback

---

## Introduction: DeltaPrompts Breaks the Zero-Increment Trap in Multimodal Distillation, Achieving 15% Performance Improvement

This article reveals that 69% of prompts in multimodal distillation are 'zero-increment' invalid samples. It proposes the DeltaPrompts dataset (containing 200,000 synthetic high-divergence reasoning problems) that selects high-value prompts via answer divergence, achieving a 15% performance improvement in benchmark tests. The core innovation lies in returning to the first principles of distillation, quantifying prompt value using answer divergence, and proactively generating targeted training data.

## Research Background: Dilemma in Visual-Language Model Distillation

Visual-Language Models (VLMs) have strong reasoning capabilities but high computational costs. Knowledge distillation is a key technology for model compression. However, current distillation prompt selection lacks systematic quality screening, relying mostly on heuristic rules or aggregation of existing datasets, leading to insufficient training data quality that affects performance.

## Zero-Increment Trap: Harm of 69% Invalid Prompts

The study found that 69% of prompts in standard datasets are 'zero-increment' samples—where the answer distributions of the teacher and student models are identical, with zero divergence and no learning signal. The consequences include: low learning efficiency (wasting resources on already mastered content), rapid performance saturation, and falling into the 'data scale trap' (mistakenly believing more data is needed instead of high-quality data).

## Core Methods: Answer Divergence Metric and DeltaPrompts Construction

1. **Answer Divergence (Δ)**：Measures prompt value—high-divergence prompts expose the capability gap between teacher and student models, while low-divergence prompts have no learning signal; 2. **DeltaPrompts Dataset**: Synthesized in four stages: seed data utilization → failure pattern identification → targeted synthesis → divergence screening. It covers three types of reasoning tasks: charts, documents, and perception-centric, with a total of 200,000 high-divergence problems.

## Evaluation Evidence: Three-Scenario Validation and 15% Performance Improvement

1. **Online Distillation**: DeltaPrompts significantly improved performance when training target teacher-student pairs; 2. **Cross-Model Transfer**: Effective when transferred to new model families; 3. **Offline Fine-Tuning**: Endows non-reasoning models with reasoning capabilities. In 10 benchmark tests (chart QA, document VQA, etc.), it achieved a 15% relative performance improvement on high-baseline models (e.g., Qwen3-VL-8B-Thinking).

## Conclusions and Insights: New Paradigm of Prioritizing Data Quality

Insights include: 1. Data quality takes precedence over scale (200,000 high-divergence prompts are better than millions of unscreened data); 2. Dynamic data strategy (continuously monitor student gaps to generate supplementary data); 3. Answer divergence provides a new tool for evaluating distillation data quality. This study promotes a paradigm shift from passive use of datasets to proactive generation of high-quality training data.

## Limitations and Future Directions

Limitations: Synthetic data lacks real-world noise/ambiguity, limited domain coverage (no validation on video/3D reasoning), and high generation cost. Future directions: Optimize divergence calculation efficiency, expand to multimodality (audio/video), online dynamic data generation, and improve prompt quality by combining human feedback.