AsymCheck: Asymmetric Partition Checkpointing Technology for Large Language Model Training

AsymCheck proposes an innovative asymmetric partition checkpointing mechanism that optimizes large language model training efficiency by assigning different-sized partitions to forward and backward propagation, while further reducing overhead through selective partition compression and batch flushing techniques. This technology has been open-sourced (GitHub link: https://github.com/zqming-cs/AsymCheck), and related results were published at the DAC2026 conference. This article will introduce it from aspects such as background, core ideas, technical architecture, and experimental verification.

With the expansion of large language model (LLM) scales, training fault tolerance becomes crucial. However, traditional full checkpoint schemes have huge storage overhead and I/O latency issues. Existing incremental checkpointing and partitioning strategies mostly adopt symmetric designs, ignoring the essential differences between forward and backward propagation, leading to insufficient resource utilization efficiency. How to minimize performance loss while ensuring fault tolerance has become a key issue.

The core innovation of AsymCheck is to break the symmetric partitioning paradigm and assign partition sizes based on the different needs of forward and backward propagation: smaller partitions for forward propagation (to capture intermediate states finely) and larger partitions for backward propagation (to reduce management overhead and adapt to reverse data access patterns).

AsymCheck adopts a decoupled hierarchical storage design, including four modules: 1. Asymmetric partition snapshot module (dynamically adjusts partition sizes); 2. Selective partition compression module (intelligently compresses based on data importance); 3. Optimal batch flushing module (merges write operations to reduce I/O latency); 4. Fault recovery module (quickly reconstructs states to reduce recomputation).

The experiments cover six models: GPT-2, BERT, RoBERTa, BLOOM, ResNet, and ViT (with parameter scales up to 10 billion), and provide modular experimental scripts to lower the threshold for reproduction. Compared with seven mainstream schemes such as ExCP and DataStates-LLM, AsymCheck shows advantages in storage efficiency and training speed.

Dependencies include Python3.12+, PyTorch1.3+, CUDA12.6, DeepSpeed0.14.5, NCCL2.20.5, Hadoop3.3.6, and Hugging Face Transformers0.24.6. Installation steps: Clone the repository → Install dependencies via pip → Run the setup script; DeepSpeed provides an NCCL integration guide to solve installation issues.

The results of AsymCheck were published at the 63rd Design Automation Conference (DAC2026, a top conference in the field of architecture and design automation). The project repository provides a standard BibTeX citation format for researchers to cite easily.

AsymCheck provides a new design idea for LLM training infrastructure, and its asymmetric partitioning concept can inspire other training optimization technologies. As model scales grow, this technology can reduce training waiting time and costs, and the open-source code provides engineering practice references for the community.