Comparative Study of Neural Network Activation Functions: Experimental Analysis on Handwritten Digit Recognition

This study is based on the PyTorch framework, taking handwritten digit recognition as the scenario, and systematically compares the performance of Sigmoid, Tanh, ReLU, and hybrid activation functions on the MNIST dataset and real-world datasets. Key findings include: ReLU performs best in accuracy (about 98.1%) and convergence speed; hybrid activation functions do not outperform pure ReLU; the model's performance drops significantly on real-world datasets; and ethical and reliability issues of AI systems are also discussed.

Activation functions are core components of neural networks, affecting expressive power, training speed, and generalization performance. The objectives of this study include: building a handwritten digit classification MLP as a benchmark; comparing multiple activation functions (including hybrid configurations); evaluating the model's generalization ability on real-world datasets; extending training to 15 epochs to observe changes; and analyzing risks and biases in model deployment.

Model Architecture: A multi-layer perceptron (MLP) is used, with an input layer of 784 neurons (flattened 28×28 images), 6 fully connected hidden layers, an output layer of 10 neurons (Softmax), Adam optimizer, and negative log-likelihood loss.

Datasets: MNIST benchmark dataset (60,000 training / 10,000 test) and real-world datasets (handwritten, online, paint_whitebg, etc.).

Activation Function Configurations: 7 configurations are tested: Sigmoid, Tanh, ReLU, Sigmoid-Tanh hybrid, Tanh-ReLU hybrid, Sigmoid-Tanh-ReLU hybrid.

MNIST Dataset Results (15 epochs of training): ReLU has the highest accuracy at about 98.1%, followed by Tanh at about 97.6% and Sigmoid at about 96.8%; hybrid activation functions are close but do not outperform ReLU.

Key Findings: ReLU converges faster (avoids gradient vanishing); extending training for Sigmoid/Tanh yields limited benefits; hybrid activation functions have no obvious advantages.

Generalization Ability: The model's performance drops significantly on real-world datasets, due to differences in writing styles, image noise, background complexity, etc.

Ethics and Reliability: Misclassification may lead to financial losses or delivery errors; the model may have biases towards mainstream writing styles; non-mainstream writing groups (e.g., the elderly) face recognition difficulties; system transparency and interpretability need to be improved.

Mitigation Strategies: Human-machine collaboration to verify low-confidence predictions; setting confidence thresholds; expanding diverse training data; continuously monitoring model performance.

Future Directions: Adopting convolutional neural networks (CNNs); applying data augmentation techniques; using larger-scale datasets; introducing uncertainty estimation (e.g., Bayesian neural networks).