Pretrain-Experiments: A Modular Framework for Continual Pre-training Experiments of Large Language Models

Pretrain-Experiments is an open-source framework developed by Sebastian Bordt and Martin Pawelczyk, focusing on continual pre-training experiments of large-scale language models. Its core design philosophy is 'One Training, Multiple Experiments': by injecting different data interventions into the base training, it enables parallel execution of multiple experiments at minimal additional cost, significantly saving computing resources. The framework supports OLMo and OLMo-Core training backends, and the entire workflow—from data injection to evaluation—can be completed via YAML configuration (no code modification needed). It also features precise data intervention capabilities and automated evaluation functions.

Large language model pre-training faces many challenges: single experiments consume significant computing resources, making it difficult to efficiently validate hypotheses with limited budgets; traditional workflows require manual modification of training code, checkpoint management, and writing evaluation scripts—processes that are tedious and error-prone. Additionally, the field of continual pre-training lacks standardized tools, leading many teams to reinvent the wheel, which hinders the improvement of research efficiency.

The framework's core mechanisms include:

1. Precise Data Intervention: Define inserted text via JSONL files (e.g., {"text": "Question: An astronomer observes that a planet rotates faster after a meteorite impact..."}). It supports three insertion modes: random distribution, range restriction, and precise position. You can set repetition counts or random subsampling to control exposure levels; it also supports combining multiple JSONL files from different sources.
2. Modular Configuration: All experiment workflows (training, intervention, evaluation) are configured via YAML files, no code modification required.
3. Multi-backend Support: Natively supports OLMo and OLMo-Core, and can be adapted to other frameworks via extensions.

The framework has a built-in automated evaluation pipeline: configure evaluation tasks (e.g., specify scripts, tasks, splits) via YAML, which can run automatically before/after training and at each checkpoint; all metrics are synced to the Weights & Biases platform for easy monitoring. Example application: Inserting ARC-Challenge questions into the OLMo-3 7B mid-training checkpoint—with just a concise YAML configuration and execution command (pretrain-experiments config/OLMo-3-1025-7B-midtrain.yaml), you can complete the entire workflow of checkpoint downloading, data injection, training, and evaluation.

The value of Pretrain-Experiments for LLM research:

• Lowering Barriers: Enables complex experiments without deep modification of training code, allowing more teams to participate in large model research.
• Resource Efficiency: The 'One Training, Multiple Experiments' mode significantly reduces computing costs.
• Improved Reproducibility: Standardized YAML configurations and automated workflows facilitate academic collaboration and result validation.
• Accelerated Discovery: Fast iteration capabilities allow researchers to test more hypotheses in a short time, deepening their understanding of model mechanisms.

Current Limitations: It is mainly oriented towards research scenarios; additional work is needed for production environment deployment. It only supports models with OLMo architecture—support for popular architectures like Llama and Mistral is still under development. Future Directions: Expand support for more training backends and model architectures; introduce distributed training support; add data intervention strategies such as adversarial insertion and curriculum learning; integrate more evaluation benchmarks and custom metrics.