Practical Guide to LLM Pre-training: Continued Pre-training with Hugging Face

This article focuses on the Hugging Face ecosystem and provides an in-depth explanation of practical methods for LLM pre-training and continued pre-training, covering core content such as conceptual differences, project architecture implementation, training monitoring and evaluation, cost planning, and best practices. It helps AI practitioners understand the complex but critical process of pre-training.

Pre-training is the foundation of LLM capabilities and is divided into two methods: pre-training from scratch and continued pre-training. Pre-training from scratch requires terabytes of data, huge computing resources (hundreds of thousands to millions of dollars), and weeks/months of time, which is suitable for creating new models or domain-specific base models. Continued pre-training is based on existing model weights, leveraging their general capabilities, with significantly reduced data volume, cost, and time, and can inject domain-specific knowledge. This project uses continued pre-training based on the TinySolar-248m-4k model.

The project uses the TinySolar-248m-4k lightweight open-source model (248 million parameters, 4K context) for easy demonstration and learning. The training data is unstructured text (needs to be domain-relevant, cleaned, and preprocessed). The core workflow is implemented using the Hugging Face Transformers library and Trainer API: load model weights → convert data to token sequences → set hyperparameters → training loop → save checkpoints. The default hardware is CPU, but GPU acceleration is recommended (code: device_map="auto"), and dataloader_num_workers can be adjusted to optimize loading efficiency.

The project integrates Weights & Biases (W&B) to monitor training, which can track metrics such as loss and learning rate in real time, visualize the process, and compare experiments. Example training metrics show that loss gradually decreases (ideal case), grad_norm reflects the magnitude of parameter updates, and the learning rate uses a cosine annealing schedule. Note that the example only has 30 steps; in practice, thousands/millions of steps are needed to show results.

Pre-training costs are high; even small models can cost hundreds of thousands of dollars. Hugging Face provides an estimation tool, and cloud service providers need to be consulted for the latest pricing. Pre-training is suitable for injecting new domain knowledge, while fine-tuning is more suitable for specific task formats; in domains with existing knowledge bases, fine-tuning is more efficient.

1. Prioritize data quality: strictly clean, deduplicate, and select high-quality sources; 2. Learning rate scheduling: continued pre-training uses a lower learning rate to avoid catastrophic forgetting, and cosine annealing scheduling is robust; 3. Save checkpoints regularly: to handle interruptions and evaluate intermediate versions; 4. Ethical safety: consider data copyright, harmful content generation by the model, and compliance.

Pre-training is a core technology for LLMs. Although it has high thresholds and costs, it is indispensable for customized models. Continued pre-training can build domain-specific models on open-source models. In the future, pre-training costs will decrease, and small and medium-sized organizations may be able to afford it; parameter-efficient fine-tuning technologies (LoRA, QLoRA) provide economical options for non-deep customization scenarios. Practicing pre-training can deepen understanding and assist in technology selection.