Latent Circuit Disruption: A New Robust Unlearning Method for Large Language Models

This article introduces a model unlearning technique called Latent Circuit Disruption (LCD), whose core is to achieve secure deletion of sensitive information by precisely locating and modifying specific knowledge circuits in large language models while preserving other capabilities of the model. Compared to traditional methods, LCD has significant advantages in unlearning completeness, side effect control, and robustness, providing a new direction for privacy protection and controllability of large language models.

Large language models memorize a large amount of data during training, including privacy, copyright, or harmful content, so specific knowledge needs to be efficiently removed. Traditional retraining is costly, and existing model unlearning methods face four major challenges:

1. Incomplete unlearning: Simple fine-tuning makes it easy to recover target knowledge via prompt engineering;
2. Severe side effects: Impairing the model's general capabilities;
3. Insufficient robustness: Weak resistance to attacks and extraction techniques;
4. Poor scalability: Difficult to adapt to large-scale models.

LCD is based on a key insight: Knowledge exists in Transformer models in the form of specific computational circuits (combinations of attention heads and FFN neurons). Unlike traditional coarse-grained modifications at the parameter level, LCD precisely locates and disrupts at the circuit level, achieving:

• Precision: Only affects target knowledge circuits;
• Minimal side effects: Preserves the functions of other circuits;
• Robustness: Fundamentally breaks the knowledge extraction path.

Circuit Discovery and Localization

1. Attention Head Analysis: Identify attention heads contributing to target knowledge via causal intervention (activation patching, path tracing) (attribution analysis, contrastive activation difference, clustered collaborative heads);
2. FFN Neuron Localization: Detect neurons storing specific facts, and locate relevant neurons using sparse activation characteristics and inter-layer correlations.

Latent Space Disruption

1. Attention Pattern Modification: Weight distribution adjustment, selective masking, structured pruning;
2. Neuron Activation Suppression: Threshold adjustment, activation direction perturbation, orthogonal subspace projection.

Optimization Objectives

Adopt multi-objective optimization: L_total = L_forget + λ*L_retain + μ*L_robust

• L_forget: Maximize the perplexity of target knowledge;
• L_retain: Minimize performance degradation on retained datasets;
• L_robust: Enhance resistance to adversarial attacks.

Evaluation Scenarios

Covers four scenarios: fact unlearning, copyrighted text unlearning, harmful content unlearning, and category unlearning.

Evaluation Metrics

Unlearning success rate, retained performance (perplexity/accuracy), resistance to membership inference attacks, resistance to model extraction.

Key Results

• Unlearning success rate is close to 100%;
• General benchmark performance degradation is controlled within 2-5%;
• Stronger resistance to attacks such as prompt injection and fine-tuning recovery;
• Maintains stable performance on large models.

Method Type	Representative Work	Advantages	Disadvantages	LCD Improvements
Gradient Ascent	GradAscent	Simple and direct	Severe side effects, incomplete unlearning	Circuit-level precise localization
Contrastive Learning	Contrastive	Good retention effect	High computational cost	Efficient latent space disruption
Knowledge Distillation	Knowledge Distillation	Strong interpretability	Requires a teacher model	No additional model needed
Parameter Editing	ROME, MEMIT	Effective for single-point editing	Batch editing conflicts	Supports batch circuit editing
Influence Functions	Influence Functions	Theoretically complete	Computationally infeasible	Efficient approximate implementation

Privacy Compliance

• Respond to GDPR's right to be forgotten;
• Remove personally identifiable information (PII);
• Protect sensitive medical data.

Copyright and Law

• Remove the impact of copyrighted training content;
• Handle expired data authorization;
• Reduce litigation risks.

Safety and Alignment

• Remove the ability to generate harmful content;
• Mitigate biases;
• Correct factual errors.

Current Limitations

• Circuit identification relies on heuristics, prone to omissions/misjudgments;
• Interference exists in multi-knowledge unlearning;
• High computational cost;
• Cross-model architecture generalization needs verification.

Future Directions

• Develop automatic circuit discovery algorithms;
• Support incremental unlearning;
• Provide mathematical proof of unlearning effects;
• Explore distributed unlearning in federated learning scenarios.