Application of Classical Machine Learning in MNIST Handwritten Digit Recognition: A Comparison Between Random Forest and SVM Methods

This project is based on scikit-learn and uses Random Forest and Support Vector Machine (SVM) to classify the MNIST handwritten digit dataset, including data preprocessing, PCA dimensionality reduction, model evaluation, and inference workflow. By comparing the performance of these two classical algorithms, we explore their unique advantages in scenarios such as resource-constrained environments and real-time inference, and return to the basics to understand the essence of machine learning algorithms.

In deep learning, CNN is the default choice for image classification, but classical machine learning algorithms have advantages such as fast training speed, simple hyperparameter tuning, strong interpretability, and low computational resource requirements. This project returns to the basics, compares the performance of Random Forest and SVM in handwritten digit recognition, understands the algorithm principles, and recognizes the applicability of traditional methods under specific constraints.

The MNIST dataset contains 70,000 28×28 pixel grayscale handwritten digit images, with 60,000 for training and 10,000 for testing, labeled from 0 to 9. The dataset considers real-world complexities such as writing styles and stroke thickness, and is relatively clean, making it an ideal choice for algorithm comparison and teaching demonstrations.

Data Preprocessing

1. Pixel value normalization (0-255 → 0-1)
2. Flatten 2D images into 784-dimensional vectors
3. Standard training/test set split

PCA Dimensionality Reduction

Through covariance matrix calculation, eigenvalue decomposition, and projection dimensionality reduction, retaining 50-100 principal components can preserve over 95% of the information, reducing computational overhead and noise.

Random Forest

Ensemble learning method: Bootstrap sampling to build training subsets, random feature selection for node splitting, and voting for decision-making. Advantages: Not prone to overfitting, fast training, and insensitive to feature scaling.

SVM

Based on statistical learning theory: Find the optimal hyperplane, use RBF kernel to handle nonlinear relationships, and adopt one-vs-all/one-vs-one strategy for multi-classification. Advantages: Strong generalization ability and good performance in high-dimensional spaces.

Evaluation Metrics

Accuracy, confusion matrix, precision/recall, F1 score

Algorithm Comparison

Feature	Random Forest	SVM
Training Time	Fast	Slower
Prediction Time	Fast	Fast (depends on number of support vectors)
Parameter Tuning	Relatively simple	Need to select kernel function and C parameter
Typical Accuracy	94-97%	95-98%
Interpretability	High	Medium
Memory Usage	Relatively large	Depends on number of support vectors

Note: CNN has higher accuracy, but classical algorithms are more computationally efficient.

Learning Value

1. Deeply understand the principles of Random Forest and SVM
2. Master the application of PCA in image data
3. Experience the end-to-end machine learning workflow
4. Learn to select algorithms based on tasks

Practical Application Significance

• Pragmatic choice for resource-constrained environments
• Fast prototyping without GPU
• Provide performance baseline for complex methods
• Clearly demonstrate basic machine learning concepts

Limitations

1. 784-dimensional pixels lose spatial structure information
2. Linear PCA cannot capture complex nonlinear relationships
3. Accuracy is lower than deep learning

Expansion Directions

• Introduce HOG/SIFT handcrafted features
• Try t-SNE/UMAP nonlinear dimensionality reduction
• Integrate predictions from both algorithms
• Model compression to reduce inference cost

This project demonstrates the feasibility of using classical machine learning algorithms to solve computer vision problems. Although deep learning dominates image recognition, Random Forest and SVM are still irreplaceable due to their fast training, ease of understanding, and resource-friendliness. Comparing the performance characteristics of the algorithms can help make informed technical choices in practical applications.