Network Anomaly Detection Using Isolation Forest Algorithm: An Unsupervised Machine Learning Security Solution

This article introduces a network anomaly detection system developed as an undergraduate course project for the Cybersecurity and Digital Forensics program at Kingston University, UK. Built on Python, the system combines Scapy for PCAP file parsing and the Isolation Forest algorithm (unsupervised machine learning) to implement anomaly detection, achieving an accuracy rate of 86.44% in tests with 50,000 data packets. The project source code is available on GitHub (link: https://github.com/anamrifzan27-lang/network-anomaly-detection).

Traditional signature-based Intrusion Detection Systems (IDS) can only identify known attack patterns and have limited effectiveness against new/variant attacks. Anomaly detection identifies suspicious activities by establishing a baseline of normal behavior and has potential detection capabilities for zero-day attacks. This project is an undergraduate course project for the Cybersecurity and Digital Forensics program at Kingston University, UK, aiming to explore the application of unsupervised machine learning in network anomaly detection.

The system adopts a modular design, including five core components:

1. Data Ingestion Layer: Uses Scapy to load and parse PCAP files, extracting raw data packets;
2. Feature Engineering Layer: Extracts features such as source/destination IP, port, protocol type, packet size, and TCP flags;
3. Model Inference Layer: Applies the Isolation Forest algorithm to calculate anomaly scores;
4. Alert Generation Layer: Records anomalies to log files in real time;
5. Evaluation & Visualization Layer: Generates confusion matrices and scatter plots for performance analysis.

Isolation Forest is an unsupervised algorithm specifically designed for anomaly detection. Its core idea is that anomalous points are easier to isolate. By building multiple random decision trees, anomalous points are isolated at a shallower layer of the tree due to their large feature differences, and the shorter the path, the higher the anomaly score. Its advantages include: no need for labeled data, linear time complexity (suitable for large-scale data), high memory efficiency, and friendliness to high-dimensional data (adapting to network traffic features).

Feature Extraction: Extracts features from PCAP files such as network layer (source/destination IP), transport layer (source/destination port, protocol type), and metadata (packet size, TCP flags). Workflow: 1. Read benign.pcap (normal traffic) and attack.pcap (attack traffic); 2. Convert features into numerical vectors; 3. Fit the Isolation Forest model; 4. Calculate anomaly scores and predicted labels; 5. Write anomaly events to alerts.log.

The project was evaluated on a test set of 50,000 data packets, achieving an accuracy rate of 86.44%. The system provides visualization analysis tools:

• Confusion Matrix: Shows the distribution of true positives, false positives, true negatives, and false negatives;
• Scatter Plot: Visualizes the separation of normal and anomalous points in the feature space, providing a basis for model tuning.

Application Scenarios: Enterprise network monitoring, supplementary module for Security Operations Centers (SOC), teaching case for cybersecurity courses, post-penetration testing analysis. Potential Improvements: Real-time traffic processing (integrating sniffing functions), deep learning enhancement (autoencoders, etc.), feature engineering optimization (introducing time series patterns), multi-model fusion (improving robustness).

This project transforms machine learning theory into a practical security tool. The choice of Isolation Forest aligns with the characteristics of network data (no need for labeled samples). An accuracy rate of 86.44% is considerable for a course project, and it provides a complete end-to-end implementation framework. As attack methods evolve, machine learning-based anomaly detection is an important direction for security defense, and this project provides a runnable and extensible foundation for learners.