Red Set ProtoCell: An Automated AI Red Teaming Engine with Dual-Agent Architecture

Red Set ProtoCell is an open-source AI red teaming engine that uses a unique Sniper/Spotter dual-agent architecture. Through evolutionary algorithms and adaptive attack strategies, it systematically detects unknown failure modes of large language models (LLMs), providing reproducible and analyzable vulnerability discovery capabilities for AI security research. Its core value lies in shifting from passive response to proactive discovery, helping build more reliable AI systems.

With the widespread application of LLMs across industries, AI security issues have become increasingly prominent. Traditional security testing methods can only detect known problems, while unknown failure modes are the real risks. Red Set ProtoCell (RSP) emerged as a solution—it is not just a compliance checking tool, but also a proactive AI security research platform designed to address the limitations of traditional testing.

• Sniper Agent: The attack initiator, using evolutionary algorithms (genetic algorithms + mutation strategies) to generate diverse adversarial prompts. It optimizes for policy violations, jailbreak attacks, etc., and explores new attack paths.
• Spotter Agent: The result evaluator, which objectively analyzes model responses through a three-layer scoring classification method (language security layer, security exploitability layer, cognitive stability layer) to quantify the severity of failures.

This division of labor forms a complete attack-evaluation loop, ensuring systematicity and objectivity.

RSP's workflow is a continuously optimized closed loop:

1. Generation: The Sniper generates adversarial prompts based on evolutionary strategies to probe specific security boundaries.
2. Execution: Sends prompts to the target LLM via API interfaces (supports mainstream models like OpenAI, Anthropic, etc.).
3. Evaluation: The Spotter applies the three-layer scoring method to analyze responses, recording whether a failure occurred and its severity.
4. Evolution: Based on evaluation results, a fitness-guided selection mechanism optimizes the next generation of attack strategies to discover deeper vulnerabilities.

RSP's technical features include:

• Evolutionary Intelligence: Genetic algorithms + iterative fitness scoring to dynamically adjust attack strategies and improve effectiveness.
• Locked Strategy Model: Attack rules, fitness functions, etc., are versioned and immutable at runtime, ensuring reproducibility and auditability of results.
• Ethical Boundary Protection: Built-in EGG mechanism to prevent the generation of non-compliant content such as CSAM (Child Sexual Abuse Material) and biological weapons, ensuring testing is within ethical frameworks.
• Secure-by-Default Design: Default target isolation, scope limits (number of iterations/token budget), and non-persistent sensitive data.

RSP's application scenarios and value:

• Enterprise-level AI Risk Assessment: Early detection of high-impact failures, provision of quantitative security assessments, replacement of ad-hoc testing processes, and reduction of post-deployment exposure.
• AI Security Research: Standardized vulnerability discovery framework, support for reproducible research results, and exploration of new failure modes that are difficult to identify with traditional methods.

It is necessary to clarify RSP's limitations and positioning: RSP is not a compliance tool, content filter, infrastructure penetration testing framework, malware generator, etc. Its positioning is an offensive security research tool, used only in controlled environments to discover weaknesses in AI models. It is not a real-time protection tool for production environments, nor can it replace human security researchers.

Red Set ProtoCell represents a new paradigm for AI security testing: shifting from passive response to proactive discovery, and from static testing to dynamic evolution. By combining a dual-agent architecture with evolutionary algorithms, it systematically probes the security boundaries of LLMs, providing strong support for AI security research and risk assessment. In today's era of rapid AI development, proactive security testing is crucial for building reliable AI systems.