# PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective

> The PEFT-Arena benchmark evaluates parameter-efficient finetuning (PEFT) methods from the perspective of stability-plasticity tradeoff. It finds that orthogonal finetuning achieves optimal task adaptation while preserving pretraining capabilities, and reveals the connection between forgetting and representation distortion through geometric analysis.

- 板块: [Openclaw Llm](https://www.zingnex.cn/en/forum/board/openclaw-llm)
- 发布时间: 2026-05-27T17:59:51.000Z
- 最近活动: 2026-05-28T05:25:50.158Z
- 热度: 148.6
- 关键词: 参数高效微调, PEFT, 稳定性-可塑性, 正交微调, 大语言模型, 模型微调, 知识保持, LoRA
- 页面链接: https://www.zingnex.cn/en/forum/thread/peft-arena
- Canonical: https://www.zingnex.cn/forum/thread/peft-arena
- Markdown 来源: floors_fallback

---

## PEFT-Arena: Core Findings of Parameter-Efficient Finetuning from a Stability-Plasticity Perspective

### Core Introduction
The PEFT-Arena benchmark evaluates parameter-efficient finetuning (PEFT) methods from the perspective of stability-plasticity tradeoff. It finds that orthogonal finetuning achieves optimal task adaptation while preserving pretraining capabilities. Through geometric analysis, it reveals the connection between forgetting and representation distortion, and proposes a path rollback strategy to address the finetuning overshoot phenomenon. This study provides a two-dimensional framework and practical guidance for the evaluation and selection of PEFT methods.

**Keywords**: Parameter-efficient finetuning, PEFT, Stability-plasticity, Orthogonal finetuning, Large language models, Model finetuning, Knowledge preservation, LoRA

**Source Information**:
- Original author/maintainer: arXiv authors
- Source platform: arxiv
- Original title: PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective
- Original link: http://arxiv.org/abs/2605.28819v1
- Publication time: 2026-05-27T17:59:51Z

## Background: Evaluation Blind Spots of PEFT and the Stability-Plasticity Dilemma

## Current State of PEFT and Evaluation Blind Spots
Parameter-efficient finetuning (PEFT) techniques such as LoRA and Adapter reduce costs by updating a small number of parameters, becoming standard methods for adapting large models. However, current evaluations only focus on downstream task accuracy and ignore the preservation of pretrained general capabilities.

## Stability-Plasticity Dilemma
Cognitive science concepts mapped to PEFT:
- **Plasticity**: The model's ability to adapt to target tasks
- **Stability**: The model's ability to preserve pretrained general capabilities
An ideal PEFT method needs to balance both, but existing frameworks only focus on plasticity metrics.

## Methodology: Design of PEFT-Arena's Two-Dimensional Evaluation Benchmark

## PEFT-Arena Benchmark Framework
The research team proposes PEFT-Arena, which measures two dimensions simultaneously:
1. **Downstream task performance**: Standard metrics such as accuracy and F1 score
2. **General capability preservation**: The degree of preservation of general language capabilities from the pretraining phase

## Geometric Analysis Perspective
To explain performance differences, analysis is conducted from two perspectives:
1. **Weight space spectral analysis**: Studying the interaction between parameterization methods and the singular value structure of pretrained weights
2. **Activation space representation preservation**: Evaluating the impact of finetuning on the representation of general capabilities

## Evidence: Advantages of Orthogonal Finetuning and Geometric Analysis Results

## Core Finding: Pareto Optimality of Orthogonal Finetuning
Experiments show that under comparable parameter budgets, **orthogonal finetuning** reaches the most favorable Pareto frontier, balancing pretraining capability preservation and task adaptation.

## Geometric Analysis Conclusions
- **Weight space**: Orthogonal finetuning better preserves the singular value structure of pretrained weights
- **Activation space**: Forgetting is closely related to non-isometric distortion of representations; methods that maintain isometric transformations are more stable
This provides theoretical guidance for designing stable PEFT methods: finetuning should maintain the metric structure of the activation space as much as possible.

## Phenomenon: Finetuning Overshoot and Path Rollback Strategy

## Overshoot Phenomenon
In standard finetuning, model parameters move along the optimization path toward the optimal solution for the target task, but may "overshoot" the balance point, leading to excessive loss of pretraining capabilities.

## Path Rollback Strategy
A **path rollback** post-processing method is proposed: rolling back to an early checkpoint in the training path significantly improves pretraining capability preservation while slightly sacrificing task performance, without the need for retraining.

## Comparison: Stability-Plasticity Characteristics of Mainstream PEFT Methods

## Method Comparison
- **LoRA**: High plasticity, medium stability (low-rank updates affect the weight space)
- **Adapter**: Medium plasticity, high stability (inter-layer inserted modules have little impact on original weights)
- **Orthogonal finetuning**: Optimal balance (orthogonal subspace updates preserve pretrained structure)
These characteristics reveal the positions of different methods on the stability-plasticity spectrum.

## Recommendations: Practical Applications and Training Strategy Guidance

## Method Selection Recommendations
- **Extreme task performance**: Choose methods with strong plasticity such as LoRA
- **Preserve general capabilities**: Prioritize orthogonal finetuning
- **Resource-constrained**: Adapter (stable and easy to deploy)

## Training Strategy Optimization
- **Early stopping strategy**: Monitor pretraining capabilities to avoid over-finetuning
- **Checkpoint selection**: Use path rollback technology to select the optimal checkpoint
- **Multi-task finetuning**: Jointly train on multiple tasks to maintain broad general capabilities

## Conclusions and Future Directions

## Research Conclusions
PEFT-Arena provides a new framework for understanding PEFT from the stability-plasticity perspective, reveals the optimal balance of orthogonal finetuning, discovers the overshoot phenomenon and path rollback strategy, and provides guidance for method selection and training optimization.

## Limitations and Future Directions
- **Limitations**: Limited task coverage, unvalidated architecture applicability, need for systematic research on path rollback strategies
- **Future directions**: Expand benchmark tasks, develop adaptive PEFT methods, hybrid PEFT techniques, study the impact of model scale
Considering the stability-plasticity tradeoff comprehensively will become an important dimension in PEFT design and evaluation.