AgentWatcher: A Rule-Based Monitoring System for Prompt Injection Attacks

AgentWatcher is a monitoring system for prompt injection attacks. It focuses on key context segments through causal attribution, combines explicit rule-based reasoning, and achieves scalable and interpretable detection in long-context scenarios, effectively balancing security and practicality. This article will introduce it from aspects such as background, methods, and experimental verification.

Threats of Prompt Injection

In large language model (LLM) and agent applications, prompt injection attacks exploit the sensitivity of LLMs to inputs. They overwrite original instructions via malicious inputs, inducing models to perform unauthorized operations (such as leaking information or calling dangerous APIs). These attacks can be executed without technical vulnerabilities, and the risks are even greater in agent scenarios.

Two Major Issues with Existing Defenses

1. Insufficient Scalability: As context length increases, the effectiveness of existing detection methods decreases significantly, making it difficult to handle tens of thousands of tokens of conversation history or complex documents.
2. Lack of Interpretability: Methods based on black-box models or implicit matching cannot explain detection results, making it hard to diagnose and improve when false positives/negatives occur.

AgentWatcher identifies the minimal context subset that has a decisive impact on model output through causal attribution, addressing the long-context challenge:

• Attribution Logic: Identify key segments whose changes would significantly affect the output, narrowing the detection scope.
• Advantages: Greatly reduces computational burden (only processes hundreds of tokens), improves detection accuracy (eliminates irrelevant interference), and enhances interpretability (marks suspicious segments).

AgentWatcher adopts an explicit rule-based reasoning framework:

• Rule Design Principles: Understandable (security experts can evaluate the logic), verifiable (performs well in independent test scenarios), and modifiable (rules can be updated without retraining).
• Reasoning Process: The monitoring LLM analyzes attribution segments based on predefined rules and outputs judgments with reasoning basis (citing rules + application logic), ensuring transparency.

Evaluation results on tool-using agent benchmarks and long-context datasets:

• Detection Effectiveness: Effectively identifies direct/indirect injection attacks, with stable performance in long-context scenarios.
• Practicality: Low false positive rate, no impact on normal operations.
• Comparative Advantages: Higher accuracy than pattern matching methods, better interpretability than end-to-end models, and better scalability than full-context processing methods.

Deployment Considerations

• Modular Architecture: Attribution, rule engine, and monitoring model can be updated independently.
• Controllable Overhead: Attribution reduces computational requirements; the monitoring model is lightweight (no need to call the main LLM).
• Flexible Configuration: Adjustable attribution sensitivity and rule strictness to adapt to different scenarios.

Ecosystem Significance

• Provides developers with practical security tools without the need for large-scale modifications to existing systems.
• Open-source nature supports community contributions to rule and algorithm improvements.
• Interpretability helps security research and promotes the construction of trustworthy AI.

Current Limitations

• Insufficient attribution accuracy in complex scenarios (e.g., multi-segment interactions affecting output).
• Rule sets only cover known attack patterns, making it difficult to handle zero-day attacks.
• Only supports text input; not adapted for multimodal agents.

Future Directions

• Optimize attribution methods to capture complex causal structures.
• Explore the combination of rules and learning to improve robustness against unknown attacks.
• Extend the framework to support multimodal injection detection.