SecuriFine: A Safety Alignment Evaluation Toolkit for Fine-tuning Large Language Models in Cybersecurity

SecuriFine is an AI safety evaluation toolkit specifically designed for cybersecurity scenarios. It aims to help developers maintain safety alignment when fine-tuning large language models (LLMs) and prevent potential security risks and misuse. It fills the gap where traditional fine-tuning evaluations ignore the safety dimension, providing a complete framework to assess and maintain the safety alignment of fine-tuned LLMs in cybersecurity scenarios.

The application of large language models in the cybersecurity field is growing rapidly, but fine-tuning faces a "double-edged sword" challenge: while improving professional capabilities, it may lose safety guardrails or even create new risks (e.g., generating attack code, exploit programs). Traditional evaluations focus on task performance and ignore the safety dimension, so SecuriFine provides a systematic solution for this.

SecuriFine is built around three core modules:

1. Automated Safety Benchmark Testing: Covers test cases for multiple scenarios such as harmful content generation and malicious code generation, simulates real adversarial scenarios, and supports batch execution and trend analysis;
2. Dataset Safety Scanning: Identifies toxic samples, sensitive data, adversarial samples, and data contamination before fine-tuning;
3. Differential Regression Analysis: Compares the safety behavior differences between the base model and the fine-tuned model, and quantifies the details of changes.

The technical implementation integrates advanced AI safety technologies:

1. Red Team Testing Automation: Builds a test template library (covering multiple attack vectors) + intelligent mutation algorithms to generate new variants;
2. Safety Alignment Metrics: Defines quantifiable indicators such as rejection rate, safety consistency, boundary clarity, and robustness score;
3. Continuous Monitoring and Auditing: Integrates CI/CD for automated testing and provides complete audit logs to meet compliance requirements.

The practical value of SecuriFine is reflected in multiple scenarios:

• Security vendors: Ensure product safety and prevent misuse risks;
• Enterprise security teams: Establish internal evaluation standards to avoid internal models becoming risk points;
• Research and education: Serve as a research tool to understand LLM safety characteristics;
• Compliance auditing: Generate evaluation reports to support compliance documents.

Limitations: The evaluation cannot cover all attack vectors, there are false positives and false negatives, and context-dependent judgments have limitations; Best Practices: Data scanning before fine-tuning, establishing baseline evaluations, integrating iterative evaluations into the fine-tuning process, combining manual reviews, and continuously monitoring deployed models.

SecuriFine is an open-source project, and community contributions are welcome. Conclusion: In today's era of powerful AI capabilities, safety alignment is crucial. SecuriFine helps developers hold the safety bottom line, which determines the long-term development of AI in the cybersecurity field.