Safety Tooling: A Unified Inference API and Empirical Toolkit for AI Safety Research

Safety Tooling is an open-source toolkit developed by safety research institutions, providing a unified LLM inference API interface and supporting empirical research tools. It aims to solve tool dilemmas in AI safety research (such as fragmented model interfaces and poor experimental reproducibility), enabling multi-model comparative evaluation, automated experimental workflows, and security testing to facilitate academic research in the AI safety field. The project is open-source and actively maintained, with the original code repository on GitHub (https://github.com/safety-research/safety-tooling) and released on May 29, 2026.

With the improvement of large language model capabilities, AI safety research has become a core issue, but there are three major tool challenges:

1. Fragmented Interfaces: Different model providers (OpenAI, Anthropic, etc.) have independent API designs and authentication mechanisms, requiring specific calling code;
2. Poor Reproducibility: Lack of standardized experimental records and configuration management;
3. Sensitive Content Handling: Security testing involves sensitive content, requiring strict isolation and audit mechanisms. Safety Tooling is designed to address these pain points.

Core Value

Encapsulate interface differences between vendors through an abstraction layer to achieve consistent code-style calls for various models.

Supported Model Ecosystem

• Commercial models: OpenAI (GPT-4/o1/o3), Anthropic (Claude 3/3.5 series), Google (Gemini Pro/Ultra);
• Open-source models: Llama, Mistral, Qwen, etc. (integrated via vLLM).

Interface Consistency

All models use the same parameter passing, retry strategies, and error handling logic to ensure experimental fairness and eliminate confounding variables introduced by calling methods.

Provides a series of auxiliary tools:

1. Prompt Management: Version control system to record modifications and experimental results, supporting backtracking and comparison;
2. Experiment Reproduction: Declarative configuration + deterministic random seeds to ensure result reproducibility;
3. Output Parsing: Built-in structured extraction strategies (JSON, classification labels, etc.) for quantitative analysis;
4. Concurrent Batch Processing: Maximize throughput under API rate limits, supporting large-scale experiments.

Designed for scenarios like adversarial testing:

1. Isolated Execution: Docker containerization support to prevent harmful outputs from affecting the host;
2. Audit Logs: Detailed records of model calls and experiment runs to support compliance reviews;
3. Content Filtering: Configurable mechanisms to balance research exploration and ethical responsibility.

Applicable to multiple AI safety scenarios:

1. Red Teaming: Unified API to compare the resistance of multiple models to jailbreak prompts and social engineering attacks;
2. Capability Evaluation: Complete toolchain supports custom benchmark construction;
3. Alignment Research: Batch/concurrent capabilities improve the efficiency of collecting human feedback data;
4. Multimodal Safety: Architecture supports expansion to vision-language model scenarios.

Feature	Safety Tooling	Direct use of vendor SDKs	Other research frameworks (e.g., EleutherAI Harness)
Unified multi-model interface	Yes	No	Partial support
AI safety-specific features	Strong	None	Medium
Experimental reproducibility	Built-in support	Need to implement manually	Partial support
Isolation and security	Built-in Docker support	None	Varies by framework
Community activity	Actively maintained	N/A	Active
Documentation and examples	Comprehensive	Official documentation	Comprehensive
Positioned between vendor SDKs and general frameworks, balancing convenience and AI safety research optimization.

Limitations

1. Model coverage needs continuous updates to adapt to newly released models;
2. Multimodal (image/audio) support needs improvement;
3. Lack of built-in visualization tools;
4. Large team collaboration features need refinement.

Future Directions

The community will participate in improvements together, evolving continuously with the development of the AI safety field.

Conclusion

Safety Tooling lowers the technical threshold for AI safety research, allowing more researchers to participate in key areas. It is a reliable starting point for research.