Panoramic Guide to Large Language Model Training Datasets: A Complete Resource Library from Pre-training Corpus to Alignment Data

This article systematically organizes the four major categories of data required for LLM training: pre-training corpus, instruction fine-tuning data, code datasets, and alignment data, emphasizing the critical role of data quality in model performance. The guide covers the characteristics, scale, license agreements, and applicable scenarios of various datasets, providing a one-stop data resource reference for researchers and developers to help understand the role of different data types and how to obtain and use them.

In LLM training, data quality often determines the final performance more than the model architecture. Excellent models require high-quality, diverse, and widely covered training data. From the GPT series to open-source models (such as Llama and Qwen), optimizing data strategies is a key driver of capability improvement. Different types of data play distinct roles in each training phase (pre-training, instruction fine-tuning, alignment), collectively supporting the model's abilities in language understanding, task execution, safety, and controllability.

Pre-training Corpus is the core for building the foundation of language understanding, requiring wide coverage, reliable quality, linguistic diversity, and strong timeliness. Mainstream resources include Common Crawl (large scale but needs cleaning), The Pile (high-quality and diverse), RedPajama (open-source replication of Llama data), RefinedWeb (deeply cleaned web data), and Chinese resources like WuDaoCorpora. Preprocessing steps include text cleaning, quality filtering, deduplication, and language identification.

Instruction Fine-tuning Data helps models learn to converse, requiring diverse instructions, high-quality responses, and standardized formats. Representative datasets include Alpaca (synthetic data), Dolly (manual annotation), FLAN (multi-task instruction set), ShareGPT (real conversations), and Chinese resources like BELLE and COIG. Construction strategies include manual annotation, model generation, data conversion, and user feedback.

Code Datasets can improve the model's reasoning and structured thinking abilities, with strict syntax and clear logic. Main sources include The Stack (open-source code with license filtering), GitHub Repositories (note license issues), StackOverflow (code Q&A), CodeSearchNet (code-natural language parallel corpus), and programming competition data. Processing steps include syntax parsing, license filtering, AST-level deduplication, and language identification.

Alignment Data ensures the model is safe and controllable (usefulness, honesty, harmlessness), with preference data (e.g., HH-RLHF, SHP) as the core. Construction faces challenges such as annotator consistency, cultural differences, adversarial samples, and dynamic updates.

Selection by Phase: For pre-training, choose large-scale general corpus (TB-level, hundreds of billions of tokens); for instruction fine-tuning, choose high-quality and diverse data (tens of thousands to hundreds of thousands of entries); code enhancement accounts for 10-20% of pre-training data; alignment uses preference datasets with RLHF/DPO.

Quality Evaluation: Language quality (perplexity), content diversity (clustering/topic models), repetition rate, and proportion of harmful content.

Legal and Ethical Considerations: Copyright compliance (choose appropriate licenses), privacy protection (anonymize PII), content moderation, and transparency (record sources and processing workflows).

English Resources: HuggingFace Datasets (largest platform), Papers with Code (links papers and code), Google Dataset Search (search engine), Kaggle Datasets (community-contributed), UCI Repository (classic datasets).

Chinese Resources: ModelScope (Magic Community), BAAI (Beijing Academy of Artificial Intelligence), CLUE (Chinese Language Understanding Evaluation), C-EVAL (Chinese evaluation benchmark).

Continuous Updates: GitHub repositories like awesome-llm-datasets regularly maintain the latest dataset lists; it is recommended to follow them for dynamic updates.