HNAM: Implementation of an AI Intrusion Detection System Based on Heuristic Network Analysis Model

HNAM is an open-source cybersecurity research project that implements a heuristic network analysis model combined with machine learning techniques for real-time intrusion detection. Its core features include: extracting over 185 network features via a multi-layer routing architecture, supporting PCAP file analysis and real-time network traffic monitoring, and aiming to address the problem that traditional rule-based IDS struggle to handle new types of attacks.

Today's cybersecurity threats are complex, and traditional rule-based IDS struggle to cope with new attacks. The HNAM project emerged as a solution; its core innovation lies in combining heuristic analysis with AI technology to propose a multi-layer heuristic network analysis framework. Unlike traditional IDS that rely on known attack signature databases, HNAM identifies anomalies by dynamically analyzing traffic behavior patterns combined with ML, enabling detection of zero-day attacks and unknown threats. This reflects a paradigm shift from passive response to active prediction, and from static rules to dynamic learning.

HNAM adopts a modular architecture, with its core centered on the innovative "Router Hierarchy":

1. Protocol Router: Classifies packets by protocols such as TCP/UDP/ICMP, complies with the ISO-OSI seven-layer model, and performs queue management for transport layer protocols;
2. Host Router: Groups traffic by source IP, internally contains a connection router to manage all connections of the host, enabling tracking of single-host behavior;
3. General Router: Receives all packets and provides a global traffic analysis perspective;
4. Connection Router: Subdivides connections by five-tuple (protocol, source IP, destination IP, source port, destination port) to accurately track the characteristics of each session.

HNAM can extract over 185 network features, covering three levels:

• Connection Layer: Packet length distribution, arrival time interval, payload statistics, TCP flag analysis, entropy/skewness/kurtosis, etc.;
• Host Layer: Statistics on the number of addresses/ports/protocols, aggregated TCP flag information;
• Protocol/General Layer: Statistics on the number of source/destination addresses and ports, overall distribution of TCP flags.

For ML integration, it uses Random Forest, Isolation Forest (for anomaly detection and classification), and a PyTorch-based autoencoder (learns normal traffic representations and identifies anomalies via reconstruction errors). The system assigns dynamic priorities to routers through a priority calculation module that combines ML scores and traffic frequency, focusing resources on suspicious activities.

HNAM supports two operation modes:

1. Real-time Capture Mode: Integrates tshark and pyshark to capture real-time traffic from network interfaces (requires root privileges), supports graceful termination via Ctrl+C and ensures proper log closure;
2. PCAP File Analysis Mode: Offline analysis of pre-recorded traffic files, supports "real-time playback" to replay PCAP at original time intervals.

For batch processing scenarios, multiple Shell scripts are provided: run_all_pcaps_hunter.ids.sh (sequential processing), run_all_pcaps_parallel_hunter.ids.sh (parallel processing), process_pcaps_in_dir.sh (directory batch processing).

HNAM introduces a sliding window analysis mechanism:

• Small Window (default 10 seconds): Used for fine-grained feature extraction;
• Large Window (default 30 seconds): Contains multiple small windows and is used for QueueStats statistical calculations.

The QueueStats module runs in an independent thread and calculates over 40 metrics per large window cycle, divided into:

• General Queue Statistics (prefix gq_): Average queue length, number of hosts, protocol distribution percentage, etc.;
• General Connection Statistics (prefix gq_cq_): Total number of connections, average connection queue length, etc.;
• Selected Router Statistics (prefix sq_): Queue length of high-priority routers, priority score, etc.;
• Selected Connection Statistics (prefix sq_cq_): In-depth analysis indicators for key monitored connections.

Multi-time scale analysis can capture instantaneous anomalies and long-term trend changes.

HNAM offers flexible configuration:

• Core Parameters: Adjust analysis window size, minimum packet threshold, etc., via configuration.json;
• Protocol Support: protocol_list.conf defines supported protocols (classified by ISO-OSI seven layers), and unrecognized protocols are recorded in Monitoring.json for easy expansion;
• Priority Weights: Adjust alpha_frequency and alpha_s1_priority to balance traffic frequency and ML score weights.

Technical dependencies include Python 3.10+, tshark, pyshark, pandas, numpy, scikit-learn, PyTorch, etc.; optional tools include hyperfine (performance testing), jq (JSON processing). The real-time mode requires appropriate permissions during deployment, and parallel processing and hardware optimization are recommended for large-scale deployments.

HNAM has a wide range of application scenarios:

• Academic Research: Provides an experimental platform to verify the effectiveness of heuristic analysis and ML in intrusion detection;
• SOC: Supplements existing security facilities and provides AI anomaly detection capabilities;
• Threat Hunting: Helps analysts deeply analyze suspicious traffic and discover potential APT activities;
• Security Training: A teaching tool to understand traffic analysis, feature engineering, and the application of ML in security.

Summary: HNAM combines heuristic analysis, multi-layer routing architecture, and ML technology to provide an innovative solution for modern cybersecurity challenges. Its open-source nature promotes community collaboration, and it will adapt better to evolving network threats in the future, serving as a reference implementation of AI-driven cybersecurity technology.