Practical Comparison Between Neural Networks and Traditional Machine Learning in Network Intrusion Detection: Not All Attacks Are Detected Equally

This study systematically compares the performance of PyTorch Multilayer Perceptron (MLP) and traditional machine learning models (Logistic Regression, Random Forest) in flow-level network intrusion detection via the open-source project Network_ids-neural-vs-traditional-attack-eval, focusing on analyzing detection rate differences across different attack types and revealing security risk blind spots behind overall accuracy.

Traditional intrusion detection evaluations often focus on macro indicators like overall accuracy, but in security practice, some attack types are harder to detect (e.g., APT/internal threats disguise as normal traffic, while DDoS has obvious abnormal features). If a model's detection rate for a certain attack type is significantly low, even if overall indicators are good, there are still serious security blind spots. Core question of this study: Do the two types of models have systematic differences in detecting different attack types?

Experimental Design: Adopted a binary classification framework (normal vs. attack), comparing three types of models: traditional ML baselines (Logistic Regression, Random Forest), PyTorch MLP; based on CIC-IDS2017/CSE-CIC-IDS2018 benchmark datasets (including various attack types like DDoS, brute force attacks, etc.).

Data Preprocessing: Flow-level feature extraction (packet size, duration, etc.), missing value handling, label encoding, standardization, training/test split; retained attack type labels for subsequent analysis.

Key Insights:

1. Due to their unique traffic patterns, all models perform generally well on DDoS attacks;
2. Web attacks (SQL injection, XSS) are hidden in normal HTTP traffic, making detection more difficult;
3. Brute force attack features are between the two above;
4. False negative rate (missed detection) is a key indicator in security scenarios, and the harm of missed detection is greater than false positives; The visualization report attack_type_detection.png intuitively shows the detection rate differences of each model across different attack types.

Over-reliance on overall accuracy easily masks risks (e.g., predicting all as normal in a scenario with 95% normal traffic also yields a 95% accuracy). Recommended evaluation practices:

1. Hierarchical evaluation: Calculate indicators by attack type to identify weak points;
2. Confusion matrix analysis: Focus on off-diagonal error patterns;
3. Prioritize false negatives: The cost of missed detection is higher than false positives;
4. Handle imbalanced data: Adopt sampling or weighting strategies.

Technical Implementation: Modular framework (src/data.py for data processing, src/models.py for PyTorch MLP definition, src/train.py for experiment scripts, src/evaluate.py for evaluation analysis, reports/ for result output), easy to extend.

Limitations: Datasets have synthetic traces and distribution shifts, some attack samples are insufficient, limited generalization ability, detection only without blocking; Future directions: Complex neural networks (CNN/LSTM), online learning, real device integration testing.

Practical Value: Security teams can reproduce the comparison, verify model performance on their own data, identify detection deficiencies for specific attack types, balance complexity and performance, and establish internal evaluation systems.

Conclusion: Network security evaluation cannot only look at overall numbers; it needs to focus on attack type differences, false negative rates, and model limitations. Understanding the boundaries of capabilities is a core competency in security engineering.