PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective

Core Introduction

The PEFT-Arena benchmark evaluates parameter-efficient finetuning (PEFT) methods from the perspective of stability-plasticity tradeoff. It finds that orthogonal finetuning achieves optimal task adaptation while preserving pretraining capabilities. Through geometric analysis, it reveals the connection between forgetting and representation distortion, and proposes a path rollback strategy to address the finetuning overshoot phenomenon. This study provides a two-dimensional framework and practical guidance for the evaluation and selection of PEFT methods.

Keywords: Parameter-efficient finetuning, PEFT, Stability-plasticity, Orthogonal finetuning, Large language models, Model finetuning, Knowledge preservation, LoRA

Source Information:

• Original author/maintainer: arXiv authors
• Source platform: arxiv
• Original title: PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective
• Original link: http://arxiv.org/abs/2605.28819v1
• Publication time: 2026-05-27T17:59:51Z

Current State of PEFT and Evaluation Blind Spots

Parameter-efficient finetuning (PEFT) techniques such as LoRA and Adapter reduce costs by updating a small number of parameters, becoming standard methods for adapting large models. However, current evaluations only focus on downstream task accuracy and ignore the preservation of pretrained general capabilities.

Stability-Plasticity Dilemma

Cognitive science concepts mapped to PEFT:

• Plasticity: The model's ability to adapt to target tasks
• Stability: The model's ability to preserve pretrained general capabilities An ideal PEFT method needs to balance both, but existing frameworks only focus on plasticity metrics.

PEFT-Arena Benchmark Framework

The research team proposes PEFT-Arena, which measures two dimensions simultaneously:

1. Downstream task performance: Standard metrics such as accuracy and F1 score
2. General capability preservation: The degree of preservation of general language capabilities from the pretraining phase

Geometric Analysis Perspective

To explain performance differences, analysis is conducted from two perspectives:

1. Weight space spectral analysis: Studying the interaction between parameterization methods and the singular value structure of pretrained weights
2. Activation space representation preservation: Evaluating the impact of finetuning on the representation of general capabilities

Core Finding: Pareto Optimality of Orthogonal Finetuning

Experiments show that under comparable parameter budgets, orthogonal finetuning reaches the most favorable Pareto frontier, balancing pretraining capability preservation and task adaptation.

Geometric Analysis Conclusions

• Weight space: Orthogonal finetuning better preserves the singular value structure of pretrained weights
• Activation space: Forgetting is closely related to non-isometric distortion of representations; methods that maintain isometric transformations are more stable This provides theoretical guidance for designing stable PEFT methods: finetuning should maintain the metric structure of the activation space as much as possible.

Overshoot Phenomenon

In standard finetuning, model parameters move along the optimization path toward the optimal solution for the target task, but may "overshoot" the balance point, leading to excessive loss of pretraining capabilities.

Path Rollback Strategy

A path rollback post-processing method is proposed: rolling back to an early checkpoint in the training path significantly improves pretraining capability preservation while slightly sacrificing task performance, without the need for retraining.

Method Comparison

• LoRA: High plasticity, medium stability (low-rank updates affect the weight space)
• Adapter: Medium plasticity, high stability (inter-layer inserted modules have little impact on original weights)
• Orthogonal finetuning: Optimal balance (orthogonal subspace updates preserve pretrained structure) These characteristics reveal the positions of different methods on the stability-plasticity spectrum.

Method Selection Recommendations

• Extreme task performance: Choose methods with strong plasticity such as LoRA
• Preserve general capabilities: Prioritize orthogonal finetuning
• Resource-constrained: Adapter (stable and easy to deploy)

Training Strategy Optimization

• Early stopping strategy: Monitor pretraining capabilities to avoid over-finetuning
• Checkpoint selection: Use path rollback technology to select the optimal checkpoint
• Multi-task finetuning: Jointly train on multiple tasks to maintain broad general capabilities

Research Conclusions

PEFT-Arena provides a new framework for understanding PEFT from the stability-plasticity perspective, reveals the optimal balance of orthogonal finetuning, discovers the overshoot phenomenon and path rollback strategy, and provides guidance for method selection and training optimization.

Limitations and Future Directions

• Limitations: Limited task coverage, unvalidated architecture applicability, need for systematic research on path rollback strategies
• Future directions: Expand benchmark tasks, develop adaptive PEFT methods, hybrid PEFT techniques, study the impact of model scale Considering the stability-plasticity tradeoff comprehensively will become an important dimension in PEFT design and evaluation.