LLM Continued Pretraining Production-Grade Pipeline: A Domain Adaptation Solution Based on PyTorch FSDP

This project is open-sourced by josephGoke (GitHub link: https://github.com/josephGoke/llm-continued-pretraining) and aims to provide a production-ready LLM continued pretraining pipeline. Its core goal is to achieve efficient distributed training via PyTorch FSDP, addressing the adaptation challenges of general-purpose LLMs in specific domains (e.g., healthcare, law). The project integrates modular components such as data preprocessing, training, and evaluation, supports configuration-driven management and a reliable checkpoint mechanism, and provides an engineering foundation for domain-specific LLM development.

General-purpose LLMs (e.g., GPT, Llama) perform well in general tasks but struggle to cover terminology, knowledge, and expression styles in professional domains. Continued pretraining is a key technology to solve this problem—by continuing training with domain data, it retains general capabilities while learning domain knowledge. However, production-level continued pretraining faces challenges such as distributed efficiency, memory management, and checkpoint saving, which this project is designed to address.

Continued pretraining is the process of further training with domain data after basic pretraining. Its differences from fine-tuning include:

• Larger data scale (millions to billions of tokens)
• Same training objective (next-token prediction)
• Lower learning rate (to avoid destroying general knowledge)
• Longer training cycles (multiple epochs) This method can deeply encode domain knowledge instead of relying solely on prompt engineering or lightweight adaptation.

Core Training Framework: PyTorch FSDP FSDP reduces single-GPU memory requirements via parameter, gradient, and optimizer state sharding, supporting training of models with hundreds of billions of parameters. Its principles include parameter sharding, on-demand collection, gradient sharding, and optimizer state sharding. Data Pipeline: Uses JSONL format (one JSON object per line), including data-prep.py (preprocessing), data-utils.py (loading batches), and the config directory (configuration files). Training and Evaluation: Modular scripts: train.py (main training), evaluate.py (evaluation), inference.py (inference testing), facilitating independent optimization and debugging.

Configuration-Driven: Manages model architecture, training hyperparameters, FSDP settings, and data configurations via configuration files, facilitating experiment management and hyperparameter search. Checkpoint Management: Regularly saves model states, supports resuming training from checkpoints, saves optimizer states, and enables multi-version management. Distributed Support: Multi-node multi-GPU training, automatic process group initialization, gradient synchronization optimization, and communication compression.

Domain-Specific Models:

• Healthcare: Trained on medical literature/medical records to enhance Q&A and diagnostic assistance capabilities
• Law: Trained on legal provisions/case precedents to improve contract review and consulting capabilities
• Finance: Trained on financial reports/research reports to support investment analysis and risk assessment Multilingual Expansion: Continued pretraining on low-resource language corpora to enhance understanding and generation capabilities Code Models: Trained on specific programming languages/framework codebases to build dedicated code assistants.

Best Practices:

• Data preparation: Cleaning and deduplication, unified format (JSONL), consistent tokenizer
• Hyperparameters: 10-100x lower learning rate, batch size adapted to hardware, monitoring validation loss to avoid overfitting
• Hardware: Recommended A100/H100, high-speed interconnection, sufficient CPU memory, NVMe storage Challenges and Solutions:
• Catastrophic forgetting: Extremely low learning rate, mixing general/domain data, LoRA supplementation
• Training stability: Gradient clipping, loss scaling, learning rate warm-up and decay
• Data quality: Establish evaluation metrics, strict cleaning processes, monitor abnormal samples.

This project translates the concept of continued pretraining into production-ready code, solving engineering problems such as distributed training and memory management, and providing a solid foundation for domain-specific LLM development. Future directions include:

• Multimodal expansion (text + images)
• Instruction alignment (to improve controllability)
• Quantization support (to reduce deployment costs)
• Model merging (merging weights of multi-domain models) As large model technology evolves, continued pretraining will become an important means of model customization, and such production-grade tools will play a key role.