Implementing a Neural Network with NumPy From Scratch: Deep Dive into the Core Principles of Deep Learning

In today's era where deep learning frameworks like TensorFlow and PyTorch are prevalent, many practitioners can build complex models but lack an intuitive understanding of the underlying mathematical principles and computation processes. This article provides an in-depth analysis of a neural network project implemented from scratch using only NumPy, covering core principles such as forward propagation, backpropagation, activation functions, and loss functions, helping readers build a solid understanding of the underlying mechanisms of deep learning. The project targets MNIST handwritten digit classification, follows minimalist design principles, retains the most essential components of a neural network, and serves as an ideal learning tool for mastering core concepts.

While the high-level APIs of current deep learning frameworks are convenient, they easily lead practitioners to overlook underlying principles. This project uses NumPy to implement the neural network because it provides the convenience of matrix operations while requiring developers to explicitly implement every step of computation. The project follows minimalist design, retaining only the essential components of a neural network and removing unnecessary complexity, making the code a learning tool where each line corresponds to textbook formulas, providing a friendly entry point for understanding core concepts like backpropagation and gradient descent.

The project adopts a classic multi-layer perceptron (MLP) architecture with two fully connected layers: the first layer maps the 784-dimensional MNIST input to a 128-dimensional hidden layer, and the second layer maps it to a 10-dimensional output. Data flows through forward propagation: the input undergoes a linear transformation (weight multiplication plus bias), then nonlinearity is introduced via an activation function. The hidden layer uses ReLU (f(x)=max(0,x)), which has advantages such as low computational cost, avoiding gradient vanishing, and introducing sparsity; the output layer uses Softmax, which converts log-odds into a normalized probability distribution, meeting the needs of classification tasks.

The project uses cross-entropy loss as the optimization target, which measures the difference between the predicted distribution and the true labels (formula: L = -Σy_i log(p_i)). Its combination with Softmax simplifies gradient calculation. Backpropagation uses the chain rule to calculate parameter gradients: first compute the output layer error, then propagate it forward layer by layer. The weight gradient is the outer product of the previous layer's activation and the current layer's error, and the bias gradient equals the error. The optimizer uses full-batch gradient descent, updating parameters along the negative gradient direction, whose principle is consistent with advanced optimizers.

Data preprocessing includes: feature normalization (scaling pixel values to 0-1), one-hot encoding (converting labels to 10-dimensional vectors), and training-test split (70% training / 30% testing). The training process is an iterative one: in each epoch, perform forward propagation to compute predictions, calculate loss and gradients, update parameters, and repeat until the loss converges or the number of iterations is reached. Although there are no advanced techniques like validation set monitoring, the core logic is consistent with production-level pipelines.

Implementing from scratch forces developers to face mathematical details such as matrix dimension matching and activation function derivative derivation, which are key to understanding deep learning. The project provides an experimental platform where readers can modify the architecture, adjust the learning rate, and try different activation functions. The intuition gained from hands-on practice is irreplaceable by reading tutorials.

Although this project has a small codebase, it covers core concepts such as forward propagation, backpropagation, and activation functions, helping readers build a solid understanding and lay the foundation for subsequent learning of complex architectures. In modern practice, implementing production models with NumPy is not practical, but training from scratch is irreplaceable for cultivating understanding. True mastery comes from in-depth understanding of basic principles rather than proficient API calls.