ManyIH: A Multi-Level Instruction Hierarchy to Solve the Challenge of Agent Instruction Conflicts

Core Insights

ManyIH proposes an instruction conflict resolution paradigm that supports any number of permission levels. The accompanying ManyIH-Bench benchmark shows current state-of-the-art models achieve only about 40% accuracy under 12 levels of conflicting instructions, revealing key challenges for agent safety. This article will analyze from aspects of background, methodology, testing, results, and safety implications.

Instruction Conflict Issues in the Agent Era

Large language model agents can receive instructions from multiple sources such as system messages, user prompts, and tool outputs. They need to follow the highest-priority instructions to ensure safety and effectiveness, but traditional Instruction Hierarchies (IH) face limitations:

• Source Diversity: Unable to accommodate dozens of sources like tool returns and memory retrieval
• Context Dynamics: Permission levels change with scenarios (e.g., medical vs. creative writing)
• Fine-Grained Conflicts: Need to balance multiple constraints (safety policies, user needs, privacy regulations)

Traditional IH assumes fixed and limited permission levels (≤5 layers), making it difficult to adapt to complex scenarios.

ManyIH: Multi-Level Instruction Hierarchy

Core Design Principles

• Scalability: Supports any number of permission levels
• Context Awareness: Permission evaluation depends on the execution scenario
• Fine-Grained Control: Resolves conflicts at the level of instruction components
• Interpretability: Transparent conflict resolution process

Technical Implementation Mechanisms

• Dynamic Permission Evaluation: Calculates permission scores by combining source type, historical trust, and task domain
• Structured Instruction Parsing: Identifies components like constraints and goals
• Conflict Resolution Algorithm: Makes decisions based on permission, conflict nature (hard constraints vs. soft preferences), and impact scope

ManyIH-Bench: The First Multi-Level Instruction Benchmark

Test Task Composition

• Programming Tasks (427): Conflict handling in code generation/modification/debugging scenarios
• Instruction Following Tasks (426): General scenarios like information retrieval and content generation

Permission Level Design

Involves 12 permission levels, simulating multi-source instructions in real agent environments

Constraint Generation

AI-generated plus human-validated, covering 46 real application scenarios (customer service, code assistants, etc.)

Experimental Results: Limitations of Current Models

• Accuracy Only ~40%: State-of-the-art models have low average accuracy on ManyIH-Bench, contrasting with traditional simple tests (90%+)
• Error Patterns:
  • Permission Confusion: Confusing priorities of different sources
  • Recency Bias: Prioritizing the most recent instruction
  • Over-Simplification: Trying to compromise instead of executing high-priority instructions
  • Context Neglect: Applying permission labels statically

Implications for Agent Safety

Safety Risks

• Leaking private information
• Executing dangerous operations
• Being deceived by prompt injection attacks
• Spreading errors in multi-agent collaboration

Improvement Directions

• Built-in explicit conflict detection and resolution modules
• Training models to learn complex permission relationships
• Implementing auditable decision-making processes
• Establishing system-level hierarchical safety architectures

Conclusions and Outlook

ManyIH and ManyIH-Bench reveal the complexity of agent instruction hierarchy issues. Current models perform poorly in multi-level conflict handling, highlighting the urgency of agent safety research. Future developments are needed in:

• More refined permission modeling methods
• More robust conflict resolution algorithms
• More effective model training strategies

Solving the instruction hierarchy problem is key to the reliable and safe application of agents.