LLM Red Teaming: A Modular Open-Source Toolkit for Adversarial Testing of Large Language Models

A red team testing framework designed specifically for AI security researchers and machine learning engineers. It supports character-level, word-level, sentence-level, and semantic-level adversarial attacks, integrates JailbreakBench jailbreak evaluation and an automated judgment system, and provides a structured, reproducible solution for security assessment of large language models.

• Original Author/Maintainer: minw0607
• Source Platform: GitHub
• Original Title: llm_red_teaming
• Original Link: https://github.com/minw0607/llm_red_teaming
• Publication Date: June 6, 2026

---

As large language models (LLMs) like ChatGPT and Claude are increasingly deployed in sensitive scenarios—from medical consultation to financial decision-making—a fundamental question has long been overlooked: How vulnerable are these models to malicious inputs?

Traditional software security testing has mature penetration testing methodologies, but the evaluation of AI systems' "adversarial robustness" is still in its infancy. Attackers can bypass security restrictions through carefully designed prompts or alter model outputs via tiny text perturbations. The LLM Red Teaming project was created to address this pain point; it provides a structured, reproducible framework that allows researchers and engineers to systematically evaluate the security boundaries of models.

---

The core design philosophy of this toolkit is "layered attack"—covering all dimensions of text adversarial attacks from the character level to the semantic level. The project uses a clear modular architecture where each component can be used independently or combined into a complete evaluation pipeline.

The attack modules are divided into four levels based on perturbation granularity, implementing a total of 7 classic adversarial attack methods:

Character-level Attacks

• TextBugger: Generates adversarial samples via random character replacement to test the model's robustness against spelling errors
• DeepWordBug: Uses editing operations like insertion, deletion, and swapping to simulate input noise in real-world scenarios

Word-level Attacks

• TextFooler: Based on WordNet synonym replacement, changes model predictions while preserving semantics
• BERTAttack: Uses BERT's mask filling mechanism to generate candidate replacement words, then filters via cosine similarity to ensure semantic consistency

Sentence-level Attacks

• CheckList: Appends random noise words to the end of text to test if the model over-relies on position bias
• StressTest: Adds redundant tautological text to evaluate the model's sensitivity to information density

Semantic-level Attacks

• SemanticAttack: Based on part-of-speech (POS) tagging for synonym replacement, changes semantic expression while keeping grammatical structure unchanged

In addition to traditional NLP adversarial attacks, the project also integrates JailbreakBench—a standardized LLM jailbreak evaluation benchmark. This module includes:

• Predefined jailbreak targets (e.g., inducing the model to output harmful content)
• Known attack templates (e.g., PAIR, GCG, etc.)
• Standardized evaluation metrics and report formats

This allows researchers to compare the performance differences of different models when facing jailbreak attacks, providing quantitative basis for safety alignment research.

---

The key challenge in red team testing is: How to automatically determine whether an attack is "successful"? Manual review is costly, and simple keyword matching is prone to misjudgment. The project uses a two-stage judgment pipeline:

Stage 1: Rule Matching Uses regular expressions to quickly identify explicit refusal responses (e.g., "I cannot", "This violates my guidelines") and known violation patterns, achieving millisecond-level initial screening.

Stage 2: Zero-shot Classification For boundary cases, the facebook/bart-large-mnli model is called for natural language inference (NLI) to determine whether the response violates preset behavioral guidelines. This hybrid strategy balances efficiency and accuracy.

The judgment results are divided into five categories: violation, refusal, blocked, uncertain, benign.

The project defines a set of unified evaluation metrics to support comparative analysis across models and attack methods:

• Accuracy Drop: The difference between the original accuracy and the accuracy after attack
• Attack Success Rate (ASR): The proportion of queries that successfully induce violations
• Refusal Rate: The proportion of queries where the model explicitly refuses to answer
• Blocked Rate: The proportion of requests blocked by the security layer

For JailbreakBench evaluation, it also provides statistical reports broken down by attack category.

---