Large Language Model System Prompt Security Dataset: Research on Defending Against Prompt Injection and Jailbreak Attacks

This article introduces an open-source dataset project focused on LLM system prompt security, aiming to provide researchers with standardized tools to evaluate and improve models' ability to defend against prompt injection and jailbreak attacks. The project covers key content such as dataset design, evaluation framework, and defense strategies, helping to enhance the security protection level of LLM agents.

System prompts are the core configuration of LLM agents (including role definitions, behavioral guidelines, sensitive information, etc.), determining the boundaries of model behavior. Malicious users can leak system prompts, induce harmful outputs, or perform unauthorized operations through prompt injection (direct/indirect/role-playing) or jailbreak attacks (gradient/template/encoding, etc.), leading to serious security consequences.

The dataset aims for systematic evaluation, reproducibility, practicality, and scalability. It includes various attack samples (direct/indirect injection, jailbreak, multimodal) and defense benchmarks (input filtering, output monitoring, etc.). Evaluation metrics include Attack Success Rate (ASR), prompt leakage rate, harmful output rate, and false positive rate.

Attack samples are stored in a structured JSON format, including fields such as attack_id, category, and attack_text. The evaluation process is: load the model → set system prompts → run attack tests → analyze responses → generate reports. Python integration examples are provided, allowing attack category filtering and evaluation of target models.

Input layer defense: pattern detection, semantic analysis, length limitation, etc.; Model layer defense: adversarial training, instruction reinforcement, multi-layer verification, etc.; Architecture layer defense: permission separation, sandbox execution, audit logs, etc. These multi-dimensional measures enhance system prompt security.

Enterprise deployment recommendations: security assessment, continuous monitoring, emergency response, security training. Compliance requirements need to align with standards such as GDPR (prevent data leakage), AI Act (security assurance), and NIST AI Risk Management Framework.

Current limitations: incomplete attack coverage, subjective evaluation, model specificity, context dependence. Future directions: general defense mechanisms, real-time protection, formal verification, multi-agent security research.