GPart: A New Paradigm for End-to-End Isometric Fine-Tuning via Global Parameter Partitioning

GPart proposes a brand-new parameter-efficient fine-tuning method aimed at solving the low-rank bottleneck problem of current mainstream PEFT methods (such as LoRA). Its core innovation is to directly map trainable vectors to the full weight space via a single isometric partition matrix, enabling end-to-end isometric fine-tuning, simplifying the process while maintaining optimization effectiveness.

Fine-tuning large language models is costly. As a mainstream PEFT method, LoRA reduces parameters via low-rank matrices, but its bilinear structure leads to non-distance-preserving mapping, distorting the optimization landscape. Improvement attempts like Uni-LoRA have not solved the core problem of end-to-end isometry.

GPart abandons the low-rank bottleneck. Its core is to use a random projection matrix to directly map d-dimensional trainable vectors to the full weight space, ensuring end-to-end isometry. The fine-tuning process is extremely simple: only random projection and a single hyperparameter d are needed, with storage cost of only d+1 values (trainable vector + random seed).

GPart is based on the theoretical premise that effective fine-tuning can emerge from random low-dimensional subspaces of the full weight space, without imposing low-rank matrix structures, breaking the traditional low-rank assumption and opening up a new theoretical path.

The research team validated GPart's effectiveness across multiple tasks: it performs equivalently or better in natural language understanding (GLUE benchmark); excels in computer vision tasks; and shows strong adaptability in mathematical reasoning tasks.

Compared to LoRA and its variants, GPart has four advantages: 1. Theoretical simplicity (eliminating low-rank constraints); 2. Parameter efficiency (extremely low storage overhead of d+1 values); 3. Simple implementation (no need for complex matrix decomposition); 4. Competitive performance (achieves or exceeds existing methods in multiple tasks).

GPart provides an elegant solution for PEFT, reducing storage and computation costs, and offering a new perspective for understanding the essence of LLM fine-tuning. Practical applications include lower deployment costs, faster fine-tuning speed, and better interpretability. It challenges the traditional view that PEFT relies on low-rank structures and demonstrates the potential of random subspace projection.