Queue as Dataset: A Multi-Stage Pipeline System for Intelligent Conversion and Processing of Web Data

This article introduces an innovative multi-stage queue data processing system, whose core is to use "queue" as the central abstraction for data processing. Through a pipeline architecture, it realizes the crawling, cleaning, conversion, and formatting of web data, and finally generates interleaved format data suitable for machine learning (especially large language model training). This system solves the problems of low efficiency and difficulty in scaling of traditional batch processing, providing an efficient solution for large-scale web data processing and AI training data preparation.

The Internet contains massive and diverse web data, but converting it into training datasets usable for machine learning faces multiple challenges: diverse web formats, uneven quality, and complex content structures; traditional batch processing methods are inefficient and difficult to scale. These problems have spurred the innovative design of the Queue as Dataset system.

The system uses "queue" as the core abstraction, replacing traditional file/database transfer methods, bringing advantages such as asynchronous processing, horizontal scaling, and failure retries. The pipeline includes five stages:

1. Web Crawling: Handles scenarios like dynamic content, login restrictions, and anti-crawling, outputting unified raw HTML;
2. Content Extraction: Extracts text, images, tables, and other information through DOM rules, machine learning recognition, and framework-specific parsers;
3. Data Cleaning: Removes noise (ads/navigation), standardizes formats, corrects encoding, and deduplicates;
4. Structured Conversion: Generates formats suitable for downstream tasks (e.g., plain text for language models, SQuAD format for question-answering systems, triples for knowledge graphs);
5. Formatted Output: Customizes the final data form according to requirements. Each stage is decoupled via queues and can be scaled independently.

The interleaving feature is specifically designed for large language model training, improving model generalization by interleaving content from multiple sources/themes. It supports three strategies:

• Random Interleaving: Selects sources with uniform distribution to balance themes;
• Weighted Interleaving: Assigns sampling probabilities based on source importance;
• Theme-Aware Interleaving: Maintains topic coherence between adjacent segments while ensuring diversity. Additionally, the system optimizes the use of context windows to reduce padding waste and uses boundary markers to help models identify document boundaries.

• Elastic Scaling: Automatically scales based on Docker/Kubernetes to handle traffic fluctuations;
• Performance Optimization: Asynchronous non-blocking I/O, batch queue operations, compiled language for critical paths, multi-threaded parallel computing, hierarchical caching;
• Fault Tolerance: Idempotent design for each stage, dead-letter queues for failed messages, state checkpoints for resume-from-breakpoint recovery. These optimizations enable the system to process thousands of pages per second.

• Academic Field: Building domain-specific corpora for law, medicine, etc.;
• Industrial Applications: Web indexing for search engines, user content collection for recommendation systems, training data acquisition for large language models in AI companies;
• Typical Cases: Multilingual pre-training datasets (controlling the proportion of each language), programming Q&A knowledge bases (extracting instruction-tuning format data from Stack Overflow).

• vs Scrapy: Scrapy focuses on crawling and requires self-implementation of subsequent processing; this system provides an end-to-end solution;
• vs Apache Spark: Spark excels at general distributed processing, but web-specific tasks require extensive customization; this system has built-in web processing optimizations;
• vs Commercial Data Services: Provides full control and customization capabilities, no vendor lock-in, and independent data ownership.

Future Directions:

1. Intelligence: Use machine learning to optimize processing workflows (automatic content area recognition, anomaly detection, reinforcement learning to adjust interleaving strategies);
2. Real-Time Processing: Support stream processing with sub-second latency to explore scenarios like real-time news analysis;
3. Multimodal Processing: Extend to extraction and alignment of image, audio, and video content. Conclusion: The Queue as Dataset system is an important advancement in the engineering of web data processing, providing an efficient solution from raw web pages to AI training data and facilitating the development of data-driven intelligent applications.