Comparative Analysis of Neural Networks and SVM: Performance Study on MNIST and FashionMNIST Datasets

This article introduces the UTS-Data-Mining open-source project by GitHub user Arachnoida, which conducts a comparative analysis of Neural Networks (NN) and Support Vector Machines (SVM) on the MNIST and FashionMNIST datasets, aiming to provide empirical references for machine learning algorithm selection. The project covers core content such as dataset introduction, algorithm principles, experimental design, expected results, and practical recommendations.

Project Background

In the field of machine learning, algorithm selection is a key factor affecting model performance. The UTS-Data-Mining project analyzes the performance differences between NN and SVM through comparative experiments on MNIST and FashionMNIST, which has both academic value and practical application reference significance.

Dataset Introduction

• MNIST: 28×28 grayscale images of handwritten digits 0-9, with 60,000 training images and 10,000 test images. The task is 10-class classification, with high data quality and clear objectives.
• FashionMNIST: A substitute for MNIST, containing 28×28 grayscale images of 10 categories of fashion items (T-shirts, pants, etc.). It has large intra-class differences and high inter-class similarity, making it more challenging.

Neural Network (NN)

• Principle: Multi-layer structure (input/hidden/output layers), forward propagation + backpropagation optimization, nonlinear activation functions (e.g., ReLU).
• Advantages: Automatic learning of hierarchical features, nonlinear modeling, end-to-end training, strong scalability.

Support Vector Machine (SVM)

• Principle: Find the optimal hyperplane based on maximum margin, using support vectors and kernel tricks, with convex optimization ensuring global optimality.
• Features: Solid theoretical foundation, strong generalization ability (stable on small to medium datasets), flexible kernel functions, good interpretability.

Experimental Setup

• Data Preprocessing: Normalization, flattening (for SVM), possible data augmentation.
• Model Configuration:
  • NN: MLP/CNN architecture, Adam/SGD optimizer, regularization (Dropout/L2).
  • SVM: Kernel functions (RBF/polynomial/linear), regularization parameter C, kernel parameter gamma.

Evaluation Metrics

• Classification Performance: Accuracy, precision, recall, F1 score.
• Efficiency: Training time, inference time, memory usage.

Performance Expectations

• MNIST: NN accuracy ≥99%, SVM accuracy between 95-98%; both perform well, but NN is slightly better.
• FashionMNIST: NN accuracy between 90-95%, SVM accuracy between 85-90%; NN's advantage is more obvious.

Influencing Factors

• Data Complexity: MNIST has small intra-class differences, while FashionMNIST is more complex.
• Feature Representation: NN automatically learns features, while SVM relies on raw pixels or handcrafted features.
• Computational Resources: NN takes more time to train, while SVM trains quickly but has high complexity on large-scale data.

Scenarios for Choosing NN

• Large data volume and sufficient computational resources;
• Complex tasks requiring automatic feature learning;
• Pursuit of the highest accuracy.

Scenarios for Choosing SVM

• Small data volume and limited samples;
• Need for fast training and deployment;
• Requirement for model interpretability;
• Limited computational resources.

Other Considerations

Engineering practices (development cycle, maintenance cost), business scenarios (accuracy/real-time requirements, data update frequency), etc., also need to be comprehensively evaluated.

Open-Source Value

• Education: Helps understand algorithm principles, experimental design, and evaluation methods, providing code practice references.
• Research: Provides benchmark results, reproduction foundations, and verification of parameter impacts.
• Community: Supports contribution improvements, dataset expansion, and documentation refinement.

Future Directions

• Algorithm Expansion: Add Random Forest, ResNet, etc.;
• Dataset Expansion: CIFAR-10/100, EMNIST, etc.;
• Experiment Deepening: Hyperparameter optimization, cross-validation, statistical testing.