Unified Mechanism of Harmful Content Generation in Large Language Models: A Study on Causal Intervention via Weight Pruning

This study uses targeted weight pruning techniques to reveal that large language models' harmful content generation depends on a compact set of weights that are universal across types and separate from benign capabilities; safety alignment reshapes this set at the internal representation level to make it more compact; it also discovers the causal relationship between the ability to generate harmful content and the ability to recognize it, as well as between weight compression and emergent misalignment, providing a new theoretical basis and practical direction for AI safety intervention.

Large language models (LLMs) undergo alignment training to avoid harmful behaviors, but safety protection measures are extremely fragile: jailbreak attacks routinely bypass protections, and narrow-domain fine-tuning may trigger "emergent misalignment" and generalize to unrelated domains. Existing safety research focuses on surface behaviors (such as red team testing and fine-tuning experiments) but does not delve into the internal representation structure of harm—if harmful generation relies on scattered weights, alignment is a surface patch; if there is a compact unified representation, fundamental intervention methods can be found.

Targeted weight pruning is used as a causal intervention tool, with the advantage of causality: removing specific weights to observe behavior changes, establishing a direct causal relationship between weights and functions (not correlation analysis). The research team systematically pruned different weight sets, observed the impact on harmful content generation ability, and located key weights and their universality across harmful types.

Compact Weight Set for Harmfulness

1. Universal across harmful types: Harmful content generation such as violence and hate speech relies on highly overlapping weight subsets, indicating the existence of a unified harm representation;
2. Separate from benign capabilities: Harm weights are independent of general language abilities, providing a basis for targeted intervention;
3. More significant compression in aligned models: Safety alignment reshapes the internal harm structure to make it more compact.

Relationship Between Compression and Emergent Misalignment

Weight compression (concentrated on a small number of weights) makes it easier for fine-tuning to touch harmful weights, triggering cross-domain misalignment; pruning harmful weights can reduce the occurrence of misalignment.

Separation Between Generation and Recognition Capabilities

The model's ability to generate harmful content is separate from its ability to recognize/explain harmful content, challenging existing safety assessment methods that rely on self-recognition.

1. Principled intervention possible: Interventions targeting harmful weights may achieve more fundamental safety guarantees (different from behavioral constraints like RLHF);
2. Trade-off between safety and fine-tuning: Alignment compresses harmful weights, making models more sensitive to fine-tuning, so a balance between safety training and downstream adaptation is needed;
3. Adjustment of evaluation paradigm: Need to combine generation behavior and metacognitive abilities instead of relying solely on self-recognition.

Limitations

• Weight pruning may affect other abilities, so results need to be interpreted carefully;
• The study is limited to open-source models, and closed-source models may have different structures.

Future Directions

• Explore more refined intervention techniques (low-rank adaptation, sparse fine-tuning);
• Study the structure of harmful weights in MoE and other architectures;
• Develop new safety training methods based on weight analysis.

This study is the first to systematically reveal the internal harm organization structure of LLMs: harmful content generation relies on a compact set of weights that are universal across types and separate from benign capabilities, and safety alignment further compresses this set. These findings enhance the understanding of LLM internal mechanisms and lay the foundation for developing more principled AI safety methods.