SubFit: A New Paradigm for LLM Compression at the Submodule Level, Breaking Hierarchical and Continuity Constraints

SubFit is a new paradigm for LLM compression at the submodule level. By breaking the full-layer granularity and continuous selection constraints of traditional hierarchical compression, it adopts submodule-level non-continuous selection and lightweight residual replacement strategies. At 25% sparsity, it retains 84.6% downstream accuracy, significantly outperforming traditional hierarchical compression methods and providing an efficient solution for large model deployment.

Basic Information:

• Original author team (arXiv submission)
• Source: arXiv, original title: From Layers to Submodules: Rethinking Granularity in Replacement-Based LLM Compression
• Release date: June 1, 2026
• Open-source code: https://github.com/eliacunegatti/SubFit
• Original link: http://arxiv.org/abs/2606.02559v1

Post-training compression of large language models aims to reduce inference costs, but existing replacement-based methods have two constraints: full-layer granularity (taking entire Transformer layers as units) and continuous selection (removed components must be distributed continuously).

The authors' analysis found that pre-trained Transformer redundancy has non-uniform distribution characteristics:

1. Uneven spatial distribution: Redundancy is scattered across different depths
2. Component type differences: Attention and FeedForward have different redundancy characteristics
3. Non-continuous patterns: Removable components do not need to be continuous

Traditional hierarchical compression is too coarse and misses fine-grained optimization opportunities.

Core design principles of SubFit (Submodule-level Fitted residual replacement):

1. Submodule granularity: Refine the compression unit to Attention and FeedForward submodules, and evaluate importance independently
2. Non-continuous selection: Allow submodule compression at any position to accurately locate redundancy
3. Lightweight residual replacement: Replace selected submodules with fitted residual bypasses (retain residual connections + lightweight fitting module + calibration data-driven)

Implementation flow: Importance evaluation → Submodule selection → Residual bypass design → Calibration training → Iterative optimization.

Experimental Setup: Cover 10 LLMs (5 base + 5 instruction-tuned), 12.5%-37.5% sparsity, compare with 4 baseline methods, evaluate perplexity and downstream accuracy.

Key Results:

• At 25% sparsity: 84.6% downstream accuracy retention (strongest baseline: 81.6%, +3% improvement), perplexity degradation of 2.42x (baseline:4.34x, 44% reduction)
• Inference efficiency: Improve inference speed, save KV cache memory, deployment-friendly

Ablation Experiments: Submodule granularity, non-continuous selection, and residual replacement are all key contributions.

Technical Advantages:

1. Fine-grained optimization: Accurate redundancy localization, type-aware strategy, retain key capabilities
2. Post-training friendly: No retraining needed, small amount of calibration data, plug-and-play, progressive compression

Comparison with Other Methods:

• vs Pruning: No fine-tuning required to maintain performance
• vs Quantization: Structural compression (can be complementary)
• vs Distillation: Directly compress the original model, retain architecture and weights

Applicable Scenarios: Resource-constrained deployment (edge/mobile), high-throughput services, long-context applications, cost-sensitive applications

Deployment Recommendations:

1. Start adjusting from 25% sparsity
2. Prepare a small amount of target domain calibration data (thousands of samples)
3. Validate performance on downstream tasks
4. Can combine with quantization technology for extreme compression

Current Limitations:

1. Significant performance drop at extremely high sparsity (>50%)
2. Greater impact on tasks sensitive to specific submodules
3. Dependence on calibration data quality

Future Directions:

1. Dynamic compression (input-adaptive submodule activation)
2. Mixed granularity compression
3. Adaptive sparsity learning
4. Multi-task joint compression optimization

SubFit breaks traditional hierarchical and continuity constraints, proving that fine-grained submodule compression can significantly improve performance while maintaining post-training convenience. In today's era where LLM deployment costs are a concern, SubFit provides a practical and efficient solution, and will play an important role in lowering deployment thresholds and expanding application scope in the future.