LLM Forgetting Technology: A New Privacy Protection Method Based on Soft Prompts

This article introduces an innovative Large Language Model (LLM) forgetting method—privacy protection technology based on soft prompts—aimed at addressing AI privacy protection and compliance challenges. By training special soft prompts to guide the model to precisely forget specific knowledge, this method does not require modifying the main parameters of the model. Compared to traditional machine forgetting methods, it has advantages such as high parameter efficiency, composability, and reversibility, providing new ideas for the "right to be forgotten" under data protection regulations like GDPR.

As LLMs become more integrated into daily life, their training data contains massive amounts of information, which may include personal privacy, copyrighted content, or harmful knowledge, leading to privacy and compliance issues (such as the "right to be forgotten" requirement under GDPR). Machine forgetting technology has emerged to enable trained models to forget specific samples/knowledge without retraining. Traditional methods have limitations: retraining is extremely costly; gradient ascent methods impair performance on other tasks; knowledge distillation methods depend on the quality of the teacher model—all struggle to balance efficiency and effectiveness.

Soft prompts are continuous vectors optimized in the embedding space (non-human-readable text) that can guide model behavior. Steps for soft prompt-based forgetting:

1. Define forgetting targets (privacy information, copyrighted content, etc., form the forgetting set);
2. Build a contrastive framework: train forgetting prompts (to make the model "unaware" of the forgetting set) and retention prompts (to maintain performance on non-target content);
3. Optimization objectives: ensure forgetting effectiveness, retention integrity, and clear boundaries;
4. Lightweight adaptation: only train a small number of prompt parameters (from thousands to tens of thousands) without modifying the main model parameters.

1. Parameter Efficiency: Only a small number of prompt parameters are optimized, with low cost and can be done on ordinary GPUs;
2. Composability: Different forgetting prompts can be trained independently and combined flexibly;
3. Reversibility: Stopping the use of prompts allows recovery without permanent damage;
4. Fine-Grained Control: Precisely forget specific content (e.g., personal privacy while retaining public facts).

1. Privacy Compliance: Respond to users' "right to be forgotten" requests and meet regulations like GDPR at low cost;
2. Copyright Protection: Quickly forget infringing content to avoid legal risks;
3. Harmful Content Filtering: Suppress the model from generating harmful/biased content;
4. Model Personalization: Customize exclusive forgetting configurations in multi-tenant scenarios.

Current Challenges:

• Forgetting Completeness: Does the model truly forget rather than hide knowledge?
• Generalized Forgetting: How to forget semantically related content?
• Evaluation Standards: Lack of unified benchmarks and quantitative indicators;
• Adversarial Robustness: Resist attacks that recover forgotten knowledge. Future Directions: Extend to multi-modal models (images, audio, etc.) to build more responsible AI systems.

Soft prompt-based forgetting technology is an important advancement in the field of AI privacy protection, solving the "right to deletion" problem of large models in an efficient way. In the future, this technology will play a key role in protecting user privacy and ensuring compliance. Forgetting ability is as important as learning ability, providing a foundation for building trustworthy AI.