LLM Security Firewall: A Prompt Injection Attack Protection Scheme Based on Semantic Embedding and XGBoost

This article introduces the Sentinel-AI project, a lightweight and high-speed security firewall designed specifically for large language models (LLMs). This solution targets prompt injection attacks (including jailbreak attacks) and uses Sentence Transformers' semantic embedding technology combined with an XGBoost classifier to achieve real-time protection. With the widespread deployment of LLMs, prompt injection has become a major security risk. Traditional rule/keyword methods are difficult to handle this, and Sentinel-AI provides an effective solution through semantic understanding and machine learning classification.

The core of prompt injection attacks is to use LLMs' natural language understanding capabilities to change model behavior through semantic manipulation. It does not rely on code vulnerabilities but instead uses language ambiguity and context dependency. Typical methods include: direct injection (embedding malicious instructions to override security prompts), jailbreak attacks (role-playing to break boundaries), and indirect injection (transmitting malicious instructions via external data sources). Traditional rule-based or keyword-based detection methods are easily bypassed and struggle to handle the concealment and diversity of attacks.

Sentinel-AI uses a two-stage detection pipeline:

1. Semantic Embedding: Uses the all-MiniLM-L6-v2 model to convert input text into 384-dimensional vectors, capturing deep semantics, identifying synonymous expressions and context changes, and outputting fixed-length vectors for subsequent processing.
2. XGBoost Classification: Inputs the embedded vectors into a trained XGBoost model for classification. The advantages of XGBoost include fast inference, strong interpretability, friendliness to high-dimensional data, and low memory usage. Technical components include: an app.py dashboard built with Streamlit, a models directory (storing models and caches), a notebook directory (training process), and requirements.txt (dependencies). Workflow: Text preprocessing → Semantic encoding → Threat classification → Response decision (forward to LLM or intercept), with controllable latency (millisecond level).

Comparison between Sentinel-AI and traditional methods:

Protection Method	Working Principle	Advantages	Limitations
Keyword Filtering	Matching blacklisted words	Simple to implement	Easily bypassed, high false positive rate
Rule Engine	Regular expressions + logical rules	Strong interpretability	High maintenance cost, limited coverage
Prompt Engineering	Embedding security instructions in system prompts	No additional components needed	Relies on model following instructions, can be overridden
Sentinel-AI	Semantic understanding + machine learning classification	Understands intent, strong adaptability	Requires training data and model maintenance
This solution can identify deformed/obscure attacks and is not limited to fixed patterns.

Sentinel-AI is easy to deploy, and applicable scenarios include:

• API Gateway Layer: Pre-filtering to form the first line of defense;
• Microservice Architecture: Independent security microservice for easy expansion and update;
• Edge Deployment: Small model size and fast inference, suitable for edge nodes to reduce latency;
• Development and Testing: Quickly test new attack samples via the Streamlit interface.

Project Value: Reflects the trend of AI security from passive defense to active intelligent defense, providing open-source security tools for enterprises/developers, lowering security thresholds, promoting best practices, and supporting community collaboration. Limitations:

• Adversarial sample risk: May be deceived by adversarial samples;
• Multilingual support: Currently mainly for English;
• Continuous learning requirement: Needs regular retraining with new data to deal with evolving attack methods.

Future improvement directions include:

• Integrating multi-model integration strategies to improve robustness;
• Introducing active learning mechanisms to automatically identify edge cases requiring manual review;
• Developing customized detection models for specific business scenarios.