LLM-Powered Threat Intelligence Collection System: Empowering Cybersecurity Defense with Large Language Models

This project builds a large language model-based threat intelligence collection system. Its core goal is to use LLMs (e.g., Llama3) to automatically collect and process cyber threat intelligence from public data sources like NVD and AlienVault OTX. The system adopts a local deployment model (Ollama + Llama3) to address challenges in traditional threat intelligence work such as data overload, diverse formats, and insufficient timeliness, providing an automated, privacy-friendly, and customizable solution.

In today's digital age, cybersecurity threats are becoming increasingly complex and frequent (ransomware, APTs, zero-day vulnerabilities, etc.). Effective threat intelligence collection and analysis are at the core of defense. However, traditional methods face four major challenges:

• Data overload: The volume of security logs, vulnerability announcements, etc., is enormous, making manual processing difficult
• Diverse formats: Data from different sources has varying formats, making integration challenging
• High timeliness requirements: Threat situations change rapidly, requiring real-time updates
• Professional knowledge threshold: Accurately understanding threats requires deep security expertise

The natural language understanding and generation capabilities of large language models provide new possibilities for addressing these challenges, enabling automated collection, standardized processing, and intelligent analysis.

Project Objectives

Use LLMs to automatically collect and process threat intelligence from public sources, supporting local deployment to ensure privacy and security.

Technology Stack

• Local LLM engine: Ollama + Llama3 (fully offline analysis)
• Data sources: NVD (National Vulnerability Database), AlienVault OTX (Open Threat Exchange platform)
• NLP processing: spaCy (entity extraction, text preprocessing)
• Development environment: Python 3.11+

System Architecture

Data Collection Layer:

• NVD collector: Connects to the API to obtain CVE information (description, CVSS score, impact scope)
• OTX collector: Collects IOCs (malicious IPs/domains/hashes) and pulse information

Data Processing Layer:

• Preprocessing: spaCy tokenization, entity recognition, format standardization, deduplication
• LLM analysis: Llama3 understands technical details, generates structured summaries, evaluates severity, and correlates similar threats

Output Layer: Structured reports (JSON), natural language summaries, alert notifications

• Data privacy protection: Sensitive data does not need to be uploaded to third parties; analysis is done locally
• Controllable costs: Uses open-source models and free APIs to reduce costs
• Flexible customization: Can customize collection strategies, analysis rules, and output formats
• Offline capability: Can still analyze when network-isolated or disconnected, ensuring functional continuity

Small and Medium Enterprise (SME) Security Operations

• Automatically monitor public intelligence sources
• Identify vulnerability threats related to assets
• Generate easy-to-understand reports
• Lower technical thresholds

Security Research and Education

• Demonstrate the threat intelligence lifecycle
• Teach LLM applications in the security field
• Research model performance
• Develop new algorithms

Red Team and Penetration Testing

• Quickly understand known vulnerabilities of targets
• Track the latest attack techniques
• Generate test background intelligence

Environment Configuration Steps

1. Install Ollama (local LLM runtime environment)
2. Clone the code repository
3. Create a Python 3.11+ virtual environment
4. Install dependencies from requirements.txt
5. Download spaCy's en_core_web_sm model
6. Pull Llama3 via Ollama
7. Configure NVD/OTX API keys
8. Test collector functionality

Scalability Design

• Add new data sources: Implement standard interfaces to add new sources
• Replace LLMs: Support open-source models compatible with Ollama
• Custom analysis: Insert specific logic and rules
• Output adaptation: Support multiple formats and downstream integration

Current Limitations

• Limited data sources: Only integrates NVD and OTX
• Analysis depth: Local models lag behind cloud models
• Real-time performance: Batch processing struggles to meet real-time detection needs
• False positive control: Lacks mature filtering mechanisms

Future Improvements

• Integrate more intelligence sources (MISP, ThreatConnect, etc.)
• Introduce RAG architecture (vector database to store historical intelligence)
• Multi-model fusion: Cross-validation to improve accuracy
• Real-time stream processing: Message queues + stream frameworks to enable real-time updates
• Visual interface: Web dashboard for display and interaction

Conclusion

This system represents an important direction for cybersecurity automation. By combining LLMs with traditional intelligence sources, it provides a low-cost, customizable, and privacy-friendly solution. Although there is room for improvement, AI-enhanced security operations are an industry trend and will play a more important role in the future.

Industry Insights

• Balance between automation and intelligence: LLMs handle repetitive tasks, allowing analysts to focus on decision-making
• Value of open-source ecosystem: Using open-source technology lowers the threshold for innovation
• Importance of data sovereignty: Local deployment addresses data sovereignty concerns and aligns with regulatory trends