Single Training Session Can Undermine Large Model Alignment: GRPO Security Vulnerability Study Reveals Post-Training Fragility

Original Author & Source:

• Original Author/Maintainer: arXiv authors
• Source Platform: arXiv
• Original Title: It Takes One to Bias Them All: Breaking Bad with One-Shot GRPO
• Original Link: http://arxiv.org/abs/2606.10931v1
• Source Publication/Update Time: 2026-06-09T14:44:01Z

Key Takeaway: Latest research shows that a single GRPO training session on one biased data sample is sufficient to override the safety alignment mechanisms of large language models, leading to systemic bias that generalizes across multiple dimensions, revealing the fundamental fragility of current post-training alignment paradigms.

Modern large language models (LLMs) need post-training to achieve "alignment" after large-scale pre-training, ensuring outputs align with human values. Common methods include Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF). However, core questions remain: Are these safety mechanisms indestructible? Can a small amount of malicious data break the protective measures? Does the current alignment paradigm have fundamental flaws?

GRPO is a training method in the field of reinforcement learning. It does not require a separate reward model; instead, it optimizes strategies by comparing the relative quality of multiple responses to the same prompt. Its core idea is to update parameters using relative advantages within a group, offering high computational efficiency and excellent performance. It has been adopted as a core post-training algorithm by mainstream large models, but its widespread use means potential vulnerabilities have far-reaching impacts.

The most critical finding of the study: A single GRPO training session on one biased sample is enough to undermine the model's safety alignment mechanism. Experiments show that this minimal attack can induce systemic bias that generalizes across attributes, categories, and benchmark tests. Attackers do not need large-scale data poisoning or complex strategies; a single malicious sample can make an aligned model "defect".

Stereotypes learned from a single GRPO training session spread through the model's internal representations in the form of "reasoning chains". When faced with related prompts, the model activates and reuses stereotype-driven reasoning patterns, which migrate to related attributes/categories (e.g., gender bias generalizes to occupation and ability evaluation). This suggests that structured bias representations exist inside the model and spread rapidly once activated.

There are significant differences in vulnerability among different models, with the key factor being the prior probability of biased outputs in the initial state. Models that have learned more stereotype associations during pre-training are more vulnerable to single GRPO attacks, as their parameter space already has "pre-set" bias patterns, and the attack only activates and reinforces them. This reminds model providers to pay attention to pre-training data quality and bias issues.

Current post-training alignment methods have fundamental fragility; a single malicious sample can override the results of safety training. Defense recommendations:

1. Training Data Filtering: Strengthen bias detection and filtering;
2. Adversarial Training: Introduce adversarial samples during the GRPO phase to enhance robustness;
3. Continuous Monitoring: Monitor for abnormal bias in outputs after deployment;
4. Multi-Layer Protection: Build a multi-dimensional security system.

The study reveals a serious security vulnerability in the GRPO framework: a single biased sample can undermine alignment and generalize across dimensions, posing challenges to safety practices in academia and industry. In the future, collaborative efforts are needed in training algorithms, data governance, monitoring mechanisms, and other dimensions to build reliable artificial intelligence systems.