Deep Neural Network Pruning Technology Based on Random Matrix Theory: Implementation of RMT Pruning and Multi-Architecture Validation

This article introduces an innovative neural network pruning method—pruning technology based on Random Matrix Theory (RMT). The project provides implementations of multiple pruning strategies, supporting various mainstream architectures such as Vision Transformer, DeiT, Swin, ConvNeXt, Hiera, and ResNet, offering a new technical path for model compression and edge deployment.

With the expansion of deep learning model scales, maintaining performance while reducing computational costs has become an important issue. Traditional pruning is based on weight magnitude or gradients; this project introduces Random Matrix Theory (RMT) to guide pruning. RMT studies the spectral distribution of large random matrices. The eigenvalue distribution of neural network weight matrices contains structural information: parts that deviate from random distribution may carry important information, while parts that conform to random distribution can be safely pruned.

The project implements multiple pruning strategies: 1. Hybrid Magnitude-SER Pruning (combines magnitude and Spectral Energy Ratio SER); 2. Classic RMT Pruning (strictly follows the RMT framework to analyze eigenvalue distribution); 3. Classic Magnitude Pruning (baseline method); 4. Spectral Edge Budget Pruning (dynamically adjusts pruning ratio for each layer); 5. Dynamic Threshold Variant (gradually reduces threshold during training to achieve progressive pruning).

The project supports various mainstream architectures: Vision Transformer (ViT), DeiT, Swin Transformer, ConvNeXt, Hiera, and ResNet. Multi-architecture validation shows that RMT pruning has good generality, applicable to attention mechanism Transformers and traditional convolutional networks.

Application Value: Edge device deployment (small size and fast inference), reduced energy consumption (decreased computation), real-time applications (low latency); Theoretical Contributions: Provides a new mathematical perspective for pruning, helping to understand the internal structure and redundancy of models.

The project provides complete reproduction code with a clear structure. Developer recommendations: First understand the characteristics of each strategy, select the appropriate method according to the model architecture and performance requirements, and refer to the document configuration instructions and hyperparameter tuning suggestions.

In the future, RMT methods are expected to be extended to large-scale models (such as GPT, CLIP), and combined with technologies like quantization and knowledge distillation to achieve more extreme optimization, providing support for lightweight deployment of giant models.