Building Neural Networks from Scratch: Deep Learning Basics with Pure NumPy Implementation

The project named 'Neural-Network-from-Scratch' was created by Muntasir-Contractor. It implements a fully functional multi-layer perceptron (MLP) neural network using only Python's native NumPy library, including core components such as automatic differentiation, forward propagation, and backpropagation. The core goal is educational: by implementing each component by hand, it helps understand the underlying principles of deep learning (the mathematical essence and code implementation of automatic differentiation, backpropagation, gradient descent, etc.).

In today's era of numerous deep learning frameworks, building neural networks from scratch is still the best way to understand their underlying principles. This project aims to help developers break free from framework dependencies, handle every mathematical detail by hand, deeply grasp the core concepts behind frameworks, and avoid staying only at the parameter-tuning level.

The project adopts a modular three-layer architecture design:

1. Value Class: The core of automatic differentiation, storing scalar values, parent nodes (_children), operation types (_op), gradient cache (grad), and backpropagation closure (_backward). It supports gradient calculation for operations such as addition, multiplication, exponentiation, exponential functions, and tanh.
2. Backpropagation: Uses topological sorting algorithm to ensure the order of gradient calculation. Traverses the topological list in reverse order from the output node, calling each node's _backward function in sequence to complete the chain propagation of gradients.
3. Neuron and Layer: The Neuron class encapsulates weights (w), bias (b), and tanh activation function; the Layer class combines multiple neurons to process batch inputs and aggregates parameters within the layer.
4. MLP Class: Supports configuration of any number of layers, including functions such as forward propagation, prediction (predict), and training loop (fit). The training process is: clear gradients → forward calculation → loss calculation → backpropagation → parameter update.

• Advantages of Pure NumPy: Requires handling mathematical details by hand, understanding gradient flow and chain rule, and improving debugging ability (e.g., checking gradients layer by layer to locate convergence issues).
• Activation Function Selection: Uses tanh by default (compresses input to the (-1,1) interval, zero-centered, with milder gradient vanishing problem than Sigmoid). Implements the derivative formula: d/dx tanh(x) = 1 - tanh²(x).
• Training Example: Uses 4 three-dimensional input samples to build a 3→4→4→1 network structure (3 neurons in input layer, 4 neurons each in two hidden layers, 1 neuron in output layer), and completes training through 3000 iterations.

Target Audience:

• Deep Learning Beginners: Establish an intuitive understanding of the internal operation of neural networks, which is more profound than directly calling framework APIs.
• Interview Preparers: Refer to the derivation and implementation of backpropagation to prepare for technical interview assessments.
• Researchers: Quickly verify new ideas without being constrained by the complexity of large frameworks.
• Educators: Use as teaching materials to help students transition from mathematical formulas to runnable code.

Improvement Directions:

• Optimizers: Extend adaptive learning rate algorithms such as Adam and RMSprop.
• Regularization: Add L1/L2 regularization, Dropout, and other techniques to prevent overfitting.
• Batch Training: Support mini-batch training to improve efficiency.
• Activation Functions: Add modern activation functions such as ReLU, Leaky ReLU, and Swish.
• Visualization Tools: Add computational graph visualization and loss curve plotting functions.

The 'Neural-Network-from-Scratch' project demonstrates the essence of neural networks with concise code: core concepts such as gradient descent, backpropagation, and automatic differentiation do not require large frameworks—only a few hundred lines of Python code can express these elegant mathematical ideas. For developers who want to truly understand deep learning rather than just tune parameters, building neural networks from scratch is still an irreplaceable learning experience, and this project provides a clear and complete starting point.