Can Large Language Models Explain AI? Research on Explainable AI in Cybersecurity Scenarios

This article is an empirical study exploring the ability of large language models (LLMs) to support explainable AI (XAI) in the cybersecurity domain, comparing the effectiveness differences between LLMs and traditional SHAP/LIME methods. The core question is whether LLMs can reliably replace or enhance traditional XAI methods. Through experimental design and human evaluation, the study reveals the hallucination problem in LLM explanations and the key role of traditional XAI data, and provides best practices for using LLMs for XAI.

Machine learning models are widely used in the cybersecurity domain (e.g., intrusion detection, malware analysis), but most operate as "black boxes" with decision-making processes that are difficult to understand. In high-risk cybersecurity scenarios, interpretability is a core requirement for trust and action basis—security analysts need to understand the reasons behind model decisions, and predictions without explanations are hard to guide practical responses.

Datasets and Scenarios

The experiment uses three cybersecurity datasets: Network_logs.csv (network traffic features and anomaly labels), cybersecurity_intrusion_data.csv (intrusion detection features and labels), and the KDD Cup dataset (a classic intrusion detection benchmark).

Comparative Methods

1. Pure LLM Explanation: Generate explanations directly from model inputs and prediction results, without feature importance information;
2. LLM-Enhanced Explanation: Generate explanations by combining SHAP/LIME feature importance;
3. Traditional XAI Output: Directly use SHAP/LIME's original feature importance charts and numerical values.

Evaluation Models

Two representative LLMs are selected: GPT-5 (closed-source) and GPT-OSS-20B (open-source).

Key Finding: LLM Hallucination Issues

Pure LLM explanations have serious hallucinations:

• Feature Importance Hallucination: Fabricate feature importance that is inconsistent with the actual model logic;
• Semantic Bias: Infer based on the semantics of feature names (e.g., assuming "packet_size" means larger packets are suspicious), which is disconnected from the model's real patterns;
• Lack of Consistency: Contradictory explanation logic across different samples.

After adding SHAP/LIME data, the coherence and alignment of explanations improve, and hallucinations are significantly reduced.

Human Evaluation Results

Preferences of 38 participants:

• Enhanced GPT-5 explanations are the most popular (accurate and easy to understand);
• GPT-OSS-20B performs competitively;
• Original SHAP/LIME outputs are generally considered difficult to understand due to obscurity.

Implications from user needs: Accuracy and understandability need to be balanced, and the LLM-enhanced approach is a feasible path.

Based on the research findings, we propose best practices for using LLMs for XAI:

1. Never use LLMs alone: Direct use of LLMs without traditional XAI feature importance data is prone to hallucinations;
2. LLMs are enhancers, not replacements: Translate the outputs of traditional XAI technologies into natural language and supplement context;
3. Maintain verifiability: Explanations should be traceable to specific feature importance data;
4. Consider audience background: Support multiple explanation granularities (technical experts prefer raw outputs, decision-makers prefer natural language).

Implications for Cybersecurity Practice

• Intrusion Detection Systems: LLM-enhanced XAI accelerates decision-making for security analysts;
• Compliance Audits: Meet the requirements for decision interpretability and auditability;
• Human-AI Collaboration: Improve the interface efficiency between AI and human analysts.

Limitations and Future Directions

• Limited Model Scope: Only GPT-5 and GPT-OSS-20B were tested; other LLMs need verification;
• Domain Specificity: Results are based on cybersecurity datasets, which may differ in other domains;
• Long-Term Stability: LLM behavior changes with versions and needs continuous monitoring.

Future directions: Multi-modal explanations, XAI optimizing LLMs, standardized evaluation benchmarks.

LLMs are powerful but not omnipotent. In XAI scenarios, blind trust in LLMs easily leads to hallucinations; their real value is as an "explanation layer" for traditional XAI methods, converting technical outputs into understandable knowledge. It is recommended that organizations integrate LLMs with traditional methods like SHAP/LIME when using them, to balance accuracy and understandability.