Implementing Neural Networks from Scratch: Handwriting a Feedforward Network and Backpropagation with NumPy

Deep learning frameworks simplify development but hide underlying details. This article analyzes a feedforward neural network project implemented purely with NumPy, helping learners understand the core mechanisms of forward propagation, backpropagation, and gradient descent, and build an intuitive understanding of the underlying principles of deep learning. The project covers components such as network architecture definition, activation functions, and training loops, which is a necessary path for in-depth learning.

Original Project Information: Author Rithvik007-04, source GitHub (link: https://github.com/Rithvik007-04/neural-network-from-numpy), published on August 11, 2026. The project implements a complete feedforward neural network using only NumPy, including components like architecture configuration, forward/backward propagation, activation functions, and training loops.

Neural Network Basics: A feedforward network consists of an input layer, hidden layers, and an output layer (input→hidden→output). Mathematical expression: The output of layer l is z^[l] = W^[l]·a^[l-1]+b^[l], and after activation, a^[l] =g(z^[l]) (g is the activation function).

Weight Initialization: Uses Xavier/Glorot initialization (code see original article), avoiding symmetry issues from zero initialization and gradient vanishing/explosion from random initialization.

Activation Functions: Implements Sigmoid (suitable for binary classification output, prone to gradient vanishing), ReLU (default for hidden layers, simple computation but has dead ReLU problem), Softmax (probability output for multi-class classification), with attached code.

Forward Propagation: Computes layer by layer from input to output, caches intermediate results (A_prev, W, b, Z) for backpropagation use, code see original article.

Loss Calculation: Uses cross-entropy loss for multi-class classification (code: compute_loss), adds 1e-8 to ensure numerical stability.

Backpropagation: Computes gradients via chain rule, traverses backward from the output layer, code see original article. Core formula: ∂L/∂W = ∂L/∂A · ∂A/∂Z · ∂Z/∂W.

Parameter Update: Updates weights and biases using gradient descent (code: update_parameters), learning rate needs to be chosen appropriately (too large causes oscillation, too small leads to slow convergence).

Common Pitfalls: Dimension mismatch (suggest printing shapes and using assertions to check), loss not decreasing (reasons: improper learning rate, wrong initialization, unnormalized data, etc.).

Debugging Tips: Gradient checking (compare numerical gradient with analytical gradient).

Framework Migration: Corresponding relationships between NumPy implementation and PyTorch (e.g., np.dot→nn.Linear, manual gradients→autograd), with attached concise PyTorch implementation code.

Extension Directions: Regularization (L2, Dropout), optimizers (Momentum, Adam), architecture improvements (Batch Normalization, residual connections).

Summary: Handwriting implementation helps understand the core mechanisms of deep learning (forward prediction, loss measurement, backward gradients, parameter update). Although frameworks are used in production, the experience of handwriting is key to understanding "why" and is the foundation for becoming an excellent engineer.