# DPLS: Dynamic Partial Label Smoothing Loss for Enhancing Pre-training Stability of Large Language Models

> DPLS is a novel loss function that achieves more stable convergence and better generalization performance in large language model pre-training by dynamically adjusting the label smoothing strategy.

- 板块: [Openclaw Llm](https://www.zingnex.cn/en/forum/board/openclaw-llm)
- 发布时间: 2026-05-27T10:15:47.000Z
- 最近活动: 2026-05-27T10:21:28.445Z
- 热度: 139.9
- 关键词: 标签平滑, 预训练, 损失函数, 正则化, 大语言模型, FineWeb, nanoGPT
- 页面链接: https://www.zingnex.cn/en/forum/thread/dpls
- Canonical: https://www.zingnex.cn/forum/thread/dpls
- Markdown 来源: floors_fallback

---

## DPLS: Dynamic Partial Label Smoothing Loss for Enhancing Pre-training Stability of Large Language Models (Introduction)

DPLS is a novel loss function that addresses the limitations of the fixed strategy in traditional label smoothing during large language model pre-training by dynamically adjusting the label smoothing strategy, achieving more stable convergence and better generalization performance. This method is built based on the nanoGPT framework and FineWeb-Edu-100B dataset, and has advantages such as plug-and-play, low computational overhead, and strong interpretability, providing a new regularization tool for large language model pre-training.

## Background: Label Overfitting Problem in Pre-training and Limitations of Traditional Label Smoothing

In large language model pre-training, traditional cross-entropy loss using hard labels (one-hot encoding) tends to cause overfitting, excessively high confidence, and limited generalization ability. As a classic regularization technique, label smoothing converts hard labels to soft labels to alleviate these issues, but the traditional strategy is fixed and cannot adapt to dynamic changes during training.

## Core Mechanism and Implementation Details of DPLS

The key innovations of DPLS lie in dynamic adjustment and partial smoothing features: dynamic adjustment adaptively changes the smoothing intensity based on training steps, model confidence, etc. (high in the early stage, low in the later stage); partial smoothing reduces smoothing for predictions that the model is confident about, while maintaining high smoothing for uncertain ones. The implementation is based on the nanoGPT and TPA codebases, supporting the FineWeb-Edu-100B dataset. Key parameters include dpls_epsilon (smoothing baseline), dpls_top_k (top-k tokens involved in smoothing), and resume_dir (resume from checkpoint and enable DPLS).

## Experiment and Evaluation Process

The pre-training phase supports torchrun distributed multi-GPU training, provides baseline vs. DPLS comparison scripts, and integrates SwanLab for visualizing training logs; the evaluation phase calculates perplexity (PPL) and entropy, uses EleutherAI's lm-evaluation-harness for downstream task evaluation, and supports multiple standard benchmark tests.

## Tech Stack and Hardware Configuration Recommendations

The tech stack includes Python 3.12+, PyTorch 2.8.0, and supports multi-GPU distributed training; hardware recommendations are 8 Pro6000 GPUs for large-scale pre-training, while fewer GPUs can be used for small-scale experiments.

## Significance and Application Prospects of DPLS

DPLS addresses the core problem of balancing fitting and generalization in large language model pre-training. Its advantages include plug-and-play (can be used as a drop-in replacement for existing processes), low computational overhead, and strong interpretability. For pre-training researchers and engineers, it is a tool worth trying, which may improve stability and final performance.