Data Organization Strategies in Multimodal Instruction Tuning: A Controlled Study on Capability Trade-offs

This article explores the impact of data organization order on capability trade-offs in Multimodal Large Language Models (MLLMs) training. By comparing four training strategies (direct mixing, curriculum training, balanced sampling, reverse curriculum), it finds that data scheduling should be regarded as a first-order design variable for multimodal model adaptation, and the curriculum training strategy performs best in structured reasoning, providing important guidance for multimodal model training.

In recent years, Multimodal Large Language Models (MLLMs) have made significant progress in tasks such as general visual understanding, chart reasoning, and document perception. However, these capabilities come from heterogeneous supervised data sources, which have large differences in task structure and learning requirements. A long-neglected question is: How does the temporal organization of data during training affect the final performance of the model? Traditional training uses a simple data mixing strategy, but different visual tasks (general visual understanding, structured chart reasoning, fine-grained OCR recognition) have essentially different cognitive requirements for the model. Core research question: Does data organization affect capability trade-offs in multimodal instruction tuning?

To isolate the data organization variable, a three-stage training framework was designed: the model backbone, trainable modules, and optimization process are fixed, and only the temporal arrangement of post-alignment supervised data is changed.

Four strategies are compared:

1. Direct Mixing: Randomly mix all data (mainstream approach)
2. Curriculum Training: From simple to complex—first general visual understanding, then structured reasoning, finally OCR-intensive supervision
3. Balanced Sampling: Maintain equal sampling ratio for each data type
4. Reverse Curriculum: First complex tasks, then simple tasks

Evaluation dimensions: General visual instruction following, chart reasoning, mathematical chart understanding, scene text QA, document QA.

Key findings from experimental results:

1. Data organization is a first-order design variable: Changing only the data order significantly affects model capability performance, challenging the "more data is better" mindset.
2. Curriculum training is optimal: It achieves the best overall capability trade-off, especially outstanding in structured reasoning.
3. Balanced sampling has bias: It performs well in OCR tasks but weakens the overall capability balance.
4. Reverse curriculum fails: It has the worst performance and unstable optimization, verifying the scientific nature of curriculum training.

Analysis of training dynamics shows that building general understanding and reasoning capabilities first, then introducing OCR-intensive supervision leads to smoother optimization and faster convergence, echoing the "scaffolding theory" in cognitive science.

Specifically: The foundation from general visual tasks enables the model to learn to extract semantic information, providing a good initialization for subsequent tasks; conversely, initial exposure to OCR data easily leads to over-focus on local details and neglect of high-level semantic understanding.

Research implications:

1. Data scheduling is a core decision: Its impact is no less than architectural improvements.
2. Curriculum training should be widely applied: A low-cost and high-yield improvement method.
3. Beware of simple mixing strategies: When data types differ greatly, carefully designed curricula bring significant improvements.
4. Explicitly manage capability trade-offs: Control trade-offs through data organization strategies to adapt to application scenario needs.

Limitations: The experiments are based on specific architectures and datasets, so the generalizability of the conclusions needs to be verified; only four simple strategies are considered, and complex dynamic scheduling may be better.

Future directions: Automatically discover the optimal curriculum sequence; combine data organization with architectural design; maximize specific task performance while maintaining general capabilities.