MNIST Autoencoder Feature Extraction: A Systematic Study of RProp Optimization and Latent Space Dimensions

This article focuses on the MNIST handwritten digit recognition task and deeply analyzes a study using autoencoders for feature extraction. The core content of the study includes: using autoencoders to learn compact representations of data, custom implementation of the Batch RProp optimization algorithm, systematic comparison of the impact of different latent space dimensions on classification performance, and the application of k-autoencoder integration strategies and sigmoid neural networks. The study verifies the effectiveness of the method through rigorous experimental design and provides empirical support for the understanding of feature learning and optimization algorithms.

The MNIST dataset contains 60,000 training images and 10,000 test images, each being a 28x28 pixel grayscale handwritten digit (0-9). Although modern deep learning models can achieve an accuracy of over 99% on this dataset, MNIST remains an ideal testbed for validating new algorithms—its moderate data size, balanced classes, and simple preprocessing allow researchers to iterate and verify ideas quickly. The unique aspects of this study are: using autoencoders for feature extraction instead of raw pixels or hand-designed features; reproducing the classic RProp optimization algorithm instead of using off-the-shelf Adam or SGD; and systematically analyzing the impact of different latent space dimensions on downstream classification tasks.

Autoencoders are a type of neural network that learns effective encodings of data by encoding input data into low-dimensional representations and then decoding to reconstruct the original input. Their basic structure includes an encoder and a decoder: the encoder maps high-dimensional input to a low-dimensional latent space with the formula z = f(W_e · x + b_e); the decoder reconstructs the original input from the latent space representation with the formula x' = g(W_d · z + b_d). The training objective is to minimize reconstruction loss (e.g., mean squared error). The study uses a k-autoencoder strategy: training multiple autoencoders and combining their latent space representations to enhance diversity, improve robustness, expand dimensions, and explore configurations of different k values and latent space dimensions (m1-m5).

RProp (Resilient Backpropagation), proposed by Martin Riedmiller and Heinrich Braun in 1993, is an adaptive learning rate algorithm. Its core idea is to maintain a separate learning rate for each weight, dynamically adjusting it based on changes in the gradient sign: increase the learning rate when gradients are in the same direction, decrease it when opposite, and keep it unchanged when zero. The update rule does not directly use the gradient magnitude but its sign. Batch RProp improves standard RProp by using the average gradient of mini-batch samples, combining the advantages of adaptive step size and the stable convergence characteristics of batch gradient descent. The study implements this algorithm customly, demonstrating a deep understanding of optimization principles.

The study systematically compares the impact of different latent space dimensions from m1 to m5 (from low to high) on classification performance: too low a dimension (m1) may lose important features, moderate dimensions (m2-m4) balance compression ratio and information retention, and too high a dimension (m5) may retain excessive redundant information. Performance is evaluated using accuracy, confusion matrix, and per-class precision/recall. The results show that classification performance first improves with increasing dimensions, then stabilizes or slightly decreases, reflecting the law of diminishing marginal returns.

Data preprocessing normalizes MNIST image pixel values to the [0,1] interval. Training strategies include: autoencoder pre-training (unsupervised feature learning), classifier fine-tuning (training the classifier with a fixed encoder), and optional end-to-end joint training. Performance evaluation is conducted on the test set, focusing on overall accuracy, error analysis, and dimension sensitivity to verify the effects of different configurations.

Technical contributions: verifying the effectiveness of autoencoder feature extraction; reproducing the RProp algorithm to deepen understanding of neural network training processes; providing a methodology for systematic dimension analysis. Limitations: MNIST is relatively simple, so performance may not match convolutional neural networks; the network depth is shallow and does not explore deep potential; modern regularization techniques are not used. Improvement directions: using convolutional autoencoders or variational autoencoders; comparing RProp with modern optimizers; exploring deeper networks; introducing regularization techniques such as dropout and batch normalization.

This MNIST classification study demonstrates the complete workflow of classic neural network methods: from data preprocessing, model design, optimization algorithm implementation to systematic experimental analysis. Although the methods are traditional, their rigor and educational value are significant. For learners, reproducing the study can deepen understanding of neural network principles; for researchers, methodologies such as systematic hyperparameter comparison and error analysis are worth learning from. Although MNIST has been "solved", exploring classic methods can still bring lasting insights—returning to basics is sometimes more valuable than chasing new technologies.