Real-time Network Intrusion Detection System Based on Machine Learning: A Practical Combination of Scapy, Random Forest, and Streamlit

This project was developed by Karthik Chowdari and is open-sourced on GitHub (link: https://github.com/KarthikChowdari/realtime-network-ids) under the MIT license. It combines Scapy packet capture, Random Forest algorithm, and Streamlit visualization to provide a lightweight machine learning solution for network security protection, aiming to address the issues of traditional IDS's insufficient ability to handle new attacks and the high cost of hardware solutions.

In today's digital age, network security threats are severe. Traditional rule-based IDS struggle to handle new attacks, and pure hardware solutions are costly. This open-source project provides an end-to-end solution that combines three modules—network packet capture, machine learning classification, and visualization display—to help developers quickly build network intrusion detection prototypes.

1. Network Traffic Capture Layer: Scapy

Scapy is responsible for real-time network traffic capture, extracting key features such as packet size distribution, protocol type, connection duration and frequency, port scanning characteristics, and abnormal traffic patterns. It supports custom filtering rules.

2. Machine Learning Core: Random Forest

Reasons for choosing Random Forest: strong interpretability (provides feature importance ranking), fast training speed, good robustness (handles noise and imbalanced data), and no need for extensive parameter tuning. It uses supervised learning, trained with labeled datasets (such as KDD Cup99 or CICIDS2017) to distinguish between normal traffic and various types of attacks.

3. Visualization Interface: Streamlit

It provides real-time traffic monitoring panels, attack detection logs, statistical charts (traffic distribution, attack type proportion), and model performance metrics (accuracy, recall, F1 score) display. An interactive interface can be built with pure Python without front-end experience.

System operation flow: Network traffic → Scapy capture → Feature extraction → Random Forest classification → Streamlit display

Step-by-step explanation:

1. Scapy monitors the specified network interface and continuously captures packets;
2. Preprocesses raw packets and extracts numerical features;
3. Inputs feature vectors into the pre-trained Random Forest model to get classification results (normal/attack type);
4. Displays detection results in real-time via the Streamlit interface (alarm notifications, traffic statistics, attack details).

1. Teaching and Research

Suitable for students majoring in network security and machine learning, covering the complete process from data collection, feature engineering to model deployment. The code is clear and easy to learn and extend.

2. Small Network Monitoring

Can be deployed on gateways or key servers as a lightweight security monitoring layer, effectively detecting common attacks (such as port scanning, SYN Flood).

3. Model Validation and Prototype Development

Security researchers can replace Random Forest with other models (such as XGBoost, Isolation Forest) to quickly verify the effect of new algorithms.

Technical Highlights

• Mature tech stack: Scapy+scikit-learn+Streamlit are all benchmark tools in the field with perfect community support;
• Modular design: The three components have clear responsibilities, making it easy to upgrade or replace individually;
• MIT open-source license: Allows free use, modification, and commercial applications.

Potential Improvement Directions

• Real-time performance optimization: Pure Python may have bottlenecks in high-bandwidth networks; Cython or asynchronous IO optimization can be considered;
• Online model update: Explore incremental learning or hot update mechanisms to avoid restarting to load new models;
• Alarm mechanism: Add enterprise-level features such as email/SMS alarms and SIEM system integration;
• Expand attack types: Cover more new attack vectors.

This system demonstrates the idea of combining classic machine learning algorithms with modern Python toolchains to solve practical security problems, providing an understandable and extensible architecture template.

Suggestions for developers entering the network security + AI field: Start by reading the source code, understand the data flow and model decision logic, and try to add improvements (such as replacing deep learning models, integrating into security operation and maintenance processes). The sharing and collaboration of the open-source community make complex security technologies accessible.