Guardrail-Under-Fire: Automated Red Teaming Platform for Evaluating Adversarial Prompt Risks in Large Models

Guardrail-Under-Fire is an open-source automated red teaming dashboard focused on evaluating the vulnerabilities of large language models (LLMs) under adversarial prompt attacks, helping developers identify and fix security loopholes. Its core value lies in automating and visualizing the red teaming process, lowering the barrier to security assessment, and supporting integration with local Ollama models, providing a practical tool for LLM security.

With the widespread application of LLMs in customer service, content generation, and other scenarios, adversarial prompt attacks have become a major security threat—attackers construct inputs to induce models to output harmful content, leak sensitive information, or bypass restrictions. Traditional security testing relies on manual audits and static rules, which struggle to cope with evolving attack methods; manual red teaming is time-consuming, labor-intensive, and requires professional knowledge. Guardrail-Under-Fire was created to address this pain point.

This is an open-source Python project with a modular design:

• dashboard.py: Visual dashboard displaying test results
• prompt_library.csv: Predefined adversarial prompt library
• test_vulnerabilities.py: Core testing engine
• runollama.py: Ollama model integration interface
• prompt_cleaner.py: Prompt preprocessing tool

The adversarial prompt techniques it covers include:

1. Jailbreak attacks: Bypassing safety training to output non-compliant content
2. Prompt injection: Embedding malicious instructions to override system rules
3. Data extraction: Obtaining sensitive information from training data
4. Model behavior manipulation: Inducing unintended responses

The automated testing workflow is as follows:

1. Load predefined adversarial prompt sets
2. Connect to local models under test via the Ollama interface
3. Execute batch tests and record responses
4. Analyze whether responses contain violations or leaks
5. Visually display risk distribution and detailed results

Advantages of integrating with Ollama:

• Offline testing to protect data privacy
• Support for fine-tuned/custom model evaluation
• Early-stage security validation
• Easy CI/CD continuous monitoring

The tool's value for different roles:

• AI Developers: Automated scanning before release to fix prompt injection vulnerabilities and ensure security baselines
• Security Teams: Systematic red teaming, quickly understanding security posture via the dashboard, and prioritizing high-risk vulnerabilities
• Researchers: Experimental platform to test new attack methods, verify defense strategies, and open-source code supports expansion
• Enterprise Compliance: Provide security test records to meet AI regulatory requirements (e.g., EU AI Act)

Current limitations and improvement directions:

• Prompt library scale: Community contributions are needed to cover the latest attack variants
• Evaluation automation: The judgment of "harmfulness" in model responses needs to be combined with manual review
• Multimodal support: Currently only focuses on text prompts; needs to expand to image and other multimodal attacks
• Model coverage: Needs to adapt to closed-source API testing

Guardrail-Under-Fire is the open-source community's pragmatic response to AI security, providing ready-to-use tools to help developers detect vulnerabilities before deployment. As LLMs enter production environments, such security testing tools will become standard components of the AI engineering stack. Security should run through the entire lifecycle of models, and the open-source spirit of this project democratizes security capabilities, allowing every developer to conduct professional-level security assessments.