Building Neural Networks from Scratch: Understanding the Mathematical Essence of Deep Learning with Pure NumPy

The JimWid/Neural_Networks project aims to implement core neural network components from scratch using pure NumPy, helping learners deeply understand the mathematical principles of forward propagation, backpropagation, and various layer mechanisms. This project fills the gap between the convenience of modern frameworks (such as PyTorch/TensorFlow) and the lack of underlying understanding, adopting a dual-track strategy of NumPy low-level implementation and PyTorch high-level encapsulation, focusing on educational value rather than production performance optimization.

Tools like PyTorch and TensorFlow make it easy to build complex neural network models, but this convenience also leads many learners to lack an intuitive understanding of the mathematical principles and computation processes behind the models. This project does not pursue production-level performance optimization; instead, it helps learners establish a deep understanding of the working mechanisms of neural networks through clear code structure and detailed annotations.

The project adopts a dual-track parallel architecture:

1. NumPy Implementation (numpy_nn module)：Core educational content, including complete implementation from basic layers to advanced components, explicitly showing low-level details such as matrix operations and gradient calculation;
2. PyTorch Implementation (pytorch_nn module)：Demonstrates the high-level encapsulation of modern frameworks; comparing the two implementations allows clear insight into the underlying details hidden by the framework. This approach enables learners to gain both deep low-level understanding and master engineering best practices.

Implementation of Basic Layers

• Dense Layer: Forward propagation is output = W·input + b; backpropagation explicitly calculates weight gradients and updates parameters, using He initialization to alleviate gradient vanishing.

Activation Functions

Implements Tanh, Sigmoid, ReLU, Softmax, and Batch Normalization, including forward calculation and derivative computation (e.g., the gradient of ReLU in backpropagation is zero in the negative region).

Loss Functions

Implements MSE, binary cross-entropy, and categorical cross-entropy, showing formulas and derivative calculations (e.g., the derivative of categorical cross-entropy is y_pred - y_true).

Convolutional Layer

Uses SciPy to implement forward (cross-correlation operation) and backpropagation (kernel gradient and input gradient calculation) of 2D convolution, supporting padding and He initialization.

Training Process

Encapsulates training (mini-batch gradient descent), testing (accuracy calculation), and model save/load functions.

The PyTorch version code is more concise; the framework automatically handles gradient calculation (error.backward()) and parameter update (optimizer.step()). Additionally, it includes accuracy calculation in the validation phase, using torch.no_grad() to disable gradient calculation for resource saving, which is a standard practice in production-level training processes.

Learning Value

• Bridges mathematics and code, showing the precise mapping from formulas to code;
• Intuitively understands the gradient flow process in backpropagation;
• Deeply understands the underlying details of frameworks, aiding debugging and optimization;
• Modular design facilitates the expansion of new components.

Target Audience

Deep learning beginners, algorithm interview preparers, researchers, and educators.

This project is a rare educational resource that allows learners to glimpse the internal operating mechanisms of neural networks. Recommended learning path: first understand the NumPy version implementation, then compare it with the PyTorch version, and finally try to add new components or optimize existing implementations in the NumPy framework. True understanding comes from building by hand, not just calling ready-made tools.