Seirênes: Enhancing LLM Reasoning Robustness via Adversarial Self-Play and Evolutionary Perturbation

Researchers propose the Seirênes framework, whose core is a parameter-sharing adversarial self-play mechanism—allowing the model to simultaneously learn to generate perturbed contexts and extract core logic from them, turning contextual perturbations from failure modes into training signals. This framework achieves an average improvement of 7-10 percentage points across 7 mathematical reasoning benchmarks, significantly enhancing the model's reasoning robustness.

In recent years, reinforcement learning based on verifiable rewards has improved LLM reasoning capabilities, but models are vulnerable when facing perturbations like redundant information and irrelevant instructions in real-world scenarios. Traditional solutions involve adding perturbed samples, but they suffer from issues such as the difficulty in exhausting the diversity of real-world perturbations and static data augmentation failing to keep up with model evolution.

The core idea of Seirênes is to turn perturbations into training signals. Its technical architecture uses parameter-sharing adversarial self-play: the same model acts as both a perturbation constructor (generating reasonable, relevant, and misleading perturbed contexts) and a solver (eliminating perturbations and restoring correct reasoning logic). Through a co-evolutionary adversarial loop, it automatically generates training curricula with increasing difficulty, forcing the model to go beyond surface pattern matching and establish deep logical reasoning capabilities.

Across 7 mathematical reasoning benchmarks, models of different scales all achieved improvements: 4B models had an average +10.2% gain, 7B +9.1%, and 30B +7.2%. Additionally, perturbations generated by the 4B Seirênes model reduced the accuracy of GPT and Gemini by approximately 4-5%, indicating that its perturbation construction capability has cross-model generalization and can diagnose common reasoning blind spots.

The perturbations constructed by Seirênes include four types: information overload, statistical correlation traps, semantic misleading, and instruction contamination. Compared with traditional methods, Seirênes has advantages such as dynamically generating perturbations (evolving with the model), adversarial design (targeting current weaknesses), and end-to-end integration (unifying perturbation generation and training).

Seirênes has issues such as high computational overhead, perturbation diversity being limited by the model's creativity, and domain limitations (currently focused on mathematical reasoning). Future directions include exploring efficient adversarial algorithms, expanding to more reasoning domains, studying the interpretability of perturbation construction, developing evaluation tools, and attempting multi-model adversarial play.

Seirênes provides three implications for AI safety: 1. It can be used as a red-teaming tool to automatically discover model weaknesses; 2. Integrating adversarial sample generation into the training loop is an effective strategy to improve robustness; 3. The self-play mechanism demonstrates the potential for models to self-improve.