MACyber: Multi-source Alignment Benchmark and 12B Large Model for Cybersecurity Domain

Addressing the challenges in cybersecurity AI capability evaluation, the MACyber project has built a comprehensive benchmark system covering seven major security domains, developed a supporting 12B-parameter threat intelligence-enhanced large model, and provided a standardized evaluation toolchain, establishing a new paradigm for cybersecurity AI capability evaluation. It consists of two core dual-drive components: MACyber-INT (multi-source alignment benchmark dataset) and MACyber-12B (dedicated large model).

Cybersecurity data is highly heterogeneous and professional, and existing general benchmarks (such as MMLU) lack in-depth coverage of the security domain. The MACyber team proposed the concept of "multi-source alignment", integrating multi-scenario data through a unified framework. Open-sourced by the qcydm team, it is positioned as a standardized evaluation system driven by both "benchmark + model".

1. Unified Data Schema: Includes five components: metadata, feature data, label information, reasoning process (evidence chain + analysis logic), and response suggestions;
2. MACyber-12B Model: Built-in RAG dual-channel architecture (exact matching for known attacks / similarity reference for unknown attacks);
3. Evaluation System: Four-dimensional weighted scoring (reasoning: 40%, threat classification: 30%, disposal suggestions: 20%, severity level:10%), using Qwen3-Max as the judge, supporting automated batch evaluation.

Covers 31 datasets, including seven core domains:

• Network traffic security: Identify anomalies such as DDoS and port scanning;
• IoT security: Analyze device behavior patterns and anomalies;
• System log security: Detect events like privilege escalation and abnormal login;
• DNS security: Identify abuses such as tunneling and DGA;
• Web security: Covers OWASP Top10 attacks;
• Vulnerability intelligence: CVE description and risk assessment;
• Threat intelligence: Comprehensive analysis of multi-source information.

• Security vendors: Objectively evaluate model capabilities to assist product selection;
• Researchers: Fill the gap of standardized benchmarks in the security domain and support experimental comparison;
• Practical scenarios: The model can be directly used for SOC intelligent decision-making, threat intelligence analysis, and audit report generation; the dataset can be used for fine-tuning security models.

The project is fully open-sourced (GitHub), providing data conversion tools and Schema validation mechanisms. Future plans include expanding the dataset to emerging domains such as cloud security and supply chain security, exploring larger-parameter dedicated security models, with the goal of becoming the de facto standard for AI evaluation in the security domain.