CVE-LMTune: A Vulnerability Classification Framework for Multi-Taxonomy Systems Based on Hierarchical Fine-Tuned Language Models

This article introduces CVE-LMTune—a vulnerability classification framework for multi-taxonomy systems based on hierarchical fine-tuned language models—aimed at automating the annotation of vulnerability descriptions into three authoritative security taxonomies: MITRE ATT&CK, CWE, and CAPEC. Using a hierarchical cascading strategy and shared embedding technology, the framework achieves weighted F1 scores of 90% for CWE, 92% for CAPEC, and 93% for MITRE ATT&CK on the SecureBERT model, effectively addressing the issues of class imbalance and large label space in multi-label classification.

With the evolution of cybersecurity threats, the number of newly disclosed vulnerabilities is growing rapidly. However, vulnerability descriptions are mostly unstructured text, making them difficult to directly use in security operations. The industry relies on taxonomies like MITRE ATT&CK (Attack Tactics and Techniques), CWE (Common Weakness Enumeration), and CAPEC (Common Attack Pattern Enumeration and Classification) to improve management efficiency, but manual mapping has problems of high complexity and long time consumption. Additionally, vulnerabilities often involve multiple labels, with a large label space and class imbalance, and traditional machine learning and general large language models have limited performance in handling such extreme multi-label tasks.

The CVE-LMTune framework consists of three stages: 1. Data Pipeline: Automatically integrate vulnerability information from multiple sources to build an annotated dataset covering multiple taxonomies; 2. Standardized Fine-Tuning and Evaluation Protocol: Specifically address the issue of extreme multi-label imbalance; 3. Hierarchical Cascading Architecture: Decompose the large classification space into smaller subproblems, gradually refine labels following the hierarchical structure of the taxonomy, and reduce learning difficulty.

Experimental comparisons show that fine-tuned encoder models (e.g., BERT series) are significantly better than generative models. On SecureBERT (a cybersecurity-optimized BERT variant), the hierarchical cascading strategy shows obvious improvements over flat classification: weighted F1 reaches 90% for CWE (12% improvement), 92% for CAPEC (8% improvement), and 93% for MITRE ATT&CK (12% improvement). This indicates that using the hierarchical structure of taxonomies can effectively improve the performance of fine-grained categories.

The core innovations of CVE-LMTune include: 1. Hierarchical Cascading Architecture: Decompose decisions according to the natural structure of the taxonomy (e.g., for CWE, first determine the major category then refine subcategories), using divide-and-conquer to reduce the complexity of subtasks; 2. Shared Embedding Mechanism: Classifiers at different levels share the underlying text representation, requiring only the addition of lightweight classification heads, making the computational overhead of hierarchical reasoning close to that of flat models and improving deployment feasibility.

The application value of CVE-LMTune is reflected in: 1. Security Vendors/Vulnerability Databases: Reduce manual annotation costs and shorten the time window from vulnerability disclosure to classification; 2. Enterprise Security Teams: Achieve accurate priority ranking and correlation analysis through standardized labels, quickly identifying high-risk vulnerabilities; 3. Robustness: Shows good generalization ability on zero-day vulnerabilities and emerging threat patterns, and can handle vulnerability types not seen during training.

CVE-LMTune has been open-sourced, providing reproducible baselines and tools for the community to promote the standardization of vulnerability classification. Future directions include: combining the semantic understanding of generative models with the classification stability of encoder models; exploring cross-language vulnerability classification and multi-modal analysis (code snippets, PoC videos, etc.); designing dedicated architectures and strategies for security domain needs to improve accuracy and interpretability.