Mega Data Factory: An Open-Source Multimodal Data Pipeline Solution for SOTA Foundation Models

Mega Data Factory (MDF) is an open-source multimodal data processing pipeline built on Ray, accelerated by Rust, and optimized for GPU. It aims to reproduce the data cleaning processes of top foundation models such as FineWeb, LAION-5B, and DataComp, supporting large-scale data governance for text, images, and videos. It addresses the industry pain points of scattered data processing workflows and the lack of unified, reproducible implementations.

In the training of large language models and multimodal foundation models, data quality often determines the final performance more than the model architecture. However, industry-leading processes like FineWeb's 15T token quality filtering and LAION-5B's CLIP-based selection are mostly scattered across different codebases and research papers, lacking unified, reproducible open-source implementations. The MDF project was born to address this pain point, providing an end-to-end multimodal data pipeline that allows researchers and engineers to reproduce the data cleaning processes of SOTA foundation models.

MDF uses Ray as the distributed computing foundation, which can scale to hundreds of nodes to process datasets of tens of billions of scale. Dual acceleration strategy: Rust is used for CPU-intensive operations (text extraction, deduplication hash calculation), and GPU for deep learning inference (CLIP/SigLIP embedding generation, aesthetic scoring), balancing development efficiency and runtime performance.

Text processing covers from rule-based filtering to model evaluation: it implements the full set of RefinedWeb heuristic filters (URL blacklist, text length, letter ratio, etc.), and plans to include KenLM perplexity scoring and model quality classifiers.

Image processing goes beyond standard workflows: it implements CLIP/SigLIP filtering from LAION-5B/DataComp, and also detects technical and visual quality issues such as compression artifacts, image entropy, color cast, and blurriness; it integrates an AIGC detector to filter synthetic images, and a CLIP aesthetic scoring module to select images based on visual quality.

MDF commits to the reproducibility of academic papers and maintains a detailed implementation status table:

• FineWeb/FineWeb-Edu: 15T token educational content classifier (in progress)
• RefinedWeb: URL filtering, trafilatura extraction, deduplication (URL filter completed)
• DCLM, Dolma, RedPajama-V2: planned
• Z-Image, Imagen 3: image generation foundation model workflows (implemented)
• LAION-5B, DataComp: CLIP filtering, deduplication (implemented)
• Qwen-VL, Seed1.5-VL, etc.: visual-language data workflows (in progress/planned) Transparent progress tracking allows the community to clearly understand feature availability.

MDF provides a concise CLI interface, allowing workflow definition via YAML configuration (e.g., mdf run --config configs/z_image.yaml), and supports command-line parameter overrides (limiting sample count, adjusting batch size).

The operator system provides extension interfaces: adding new Refiners (enriching record fields), Filters (data selection), and Deduplicators (deduplication) can follow a unified pattern. The documentation lists the functions of each operator and reference papers, lowering the entry barrier for users.

Engineering-level optimizations: Rust accelerates hot paths such as text extraction and hash calculation, not all operations are rewritten; GPU tasks are scheduled reasonably, and CLIP/SigLIP embedding generation is designed to fully utilize GPU throughput.

It provides an interactive report via HuggingFace Spaces, visualizing pipeline operation metrics and performance statistics, improving observability and helping debug large-scale data processing tasks.

MDF fills a gap in the open-source ecosystem: it is a unified, scalable, high-performance multimodal data processing framework that integrates scattered implementations and re-implements them with modern engineering practices, lowering the threshold for high-quality data processing.

Future adaptation to the trend of multimodal large models: The modular design can be extended to video processing and multimodal quality assessment models, providing a solid starting point for teams training foundation models.