STQuant: Adaptive Spatio-Temporal Quantization Framework Redefines Memory Efficiency in Large Model Training

Memory is often a bottleneck when training large multimodal models, with optimizer states consuming a significant amount of memory. STQuant reduces the memory footprint of optimizer states by 84.4% while maintaining model quality through a spatio-temporal adaptive precision allocation strategy, providing an efficient quantization solution for large model training.

In large model training, optimizer states (e.g., first/second moments of Adam) account for a high proportion of memory. Traditional fixed-precision quantization fails to adapt to inter-layer numerical distribution differences (shallow vs. deep layers) and dynamic changes across training phases (large fluctuations in early stages, convergence in later stages), easily leading to accuracy loss or resource waste.

Spatial Dimension: Dynamically allocate precision based on the sensitivity of layers and state variables—higher bits are used for sensitive layers/states; Temporal Dimension: Monitor training statistics (gradient norm, variance, etc.), use high precision in early training stages to ensure stability, and gradually reduce precision in later stages.

Challenge 1: Quantization noise affects training stability → Adopt progressive quantization (high precision initially, gradually reduced) + error compensation mechanism; Challenge 2: Exponential search space → Focus on key factors (layer depth, state type) via factor selection strategy + dynamic transfer decision algorithm with linear complexity.

• Memory efficiency: Optimizer state memory reduced by 84.4%, average bit width of 5.1 bits; - Model quality: Performance comparable to full-precision trained models (difference within statistical error); - Computational overhead: Additional cost O(N/K) (N = total steps, K = adjustment cycle), additional space O(1).

Multimodal models have more urgent memory requirements; STQuant can automatically adapt to the numerical characteristics of different modal encoders. For complex training strategies (e.g., contrastive learning), the temporal adaptive capability can increase precision in key stages to ensure stability.

Limitations: Factor selection strategy can be optimized; only targets optimizer states; adapts to Adam variants; Future Directions: Extend to parameter/activation quantization; adapt to other optimizers (LARS/LAMB); distributed training scenarios; synergy with parallel technologies; hardware-aware strategies.

STQuant achieves a balance between significant reduction in resource consumption and preservation of model quality, which is of great significance for economic and environmental sustainability in the era of large models. Its methodology of identifying key factors and designing adaptive strategies provides a reference for similar optimization problems.