TraceSafe: A Systematic Evaluation of LLM Safety Guardrails in Multi-step Tool Calling Trajectories

This article focuses on the safety issues of intermediate trajectories in multi-step tool calling by LLM agents, filling the gap in domain evaluation. Key contributions include proposing the first trajectory-level safety benchmark TraceSafe-Bench (12 risk categories, 1000+ instances), and discovering three key patterns: guardrail effectiveness depends on structured data capabilities rather than semantic alignment; model architecture is more important than scale; accuracy improves with execution steps.

LLMs have evolved from static chatbots to autonomous tool-calling agents, and safety risks have shifted from final outputs to intermediate trajectories. Traditional guardrails focus on final content, but malicious tool-calling sequences may cause damage early, and existing evaluations of intermediate trajectory safety are almost non-existent.

TraceSafe-Bench is the first benchmark for evaluating the safety of multi-step tool calling trajectories, with the concept of evaluating every step of execution in depth. It covers 12 risk categories: safety threats (prompt injection, privacy leakage, privilege abuse) and operation failures (hallucination-induced incorrect calls, interface inconsistencies, etc.); it contains over 1000 execution instances with annotated risk points.

1. Structured capability outperforms semantic alignment: Strongly correlated with structured tests (ρ=0.79), irrelevant to jailbreak robustness; 2. Architecture is better than scale: General-purpose LLMs are superior to specialized safety guardrail models, and medium-scale general-purpose models may be more optimal; 3. Accuracy improves with steps: In long trajectories, models shift from static definitions to dynamic behavior observation, and information gain enhances recognition rates.

1. Prioritize evaluating structured data processing capabilities when selecting guardrails; 2. Innovate evaluation methods and establish standards for trajectory structuring/temporal reasoning; 3. Utilize information gain from long trajectories to design guardrails that dynamically integrate historical context.

Limitations: Limited modal coverage (lack of multimodality), based on static datasets; Future directions: Develop trajectory safety training methods, multimodal trajectory evaluation, and human-machine collaborative guardrail mechanisms.