Building Neural Networks from Scratch: A Hands-On Introduction to Deep Learning with Micrograd

This project is maintained by rishit836 and was published on GitHub on June 14, 2026 (Project link: https://github.com/rishit836/neural-network-from-scratch). It is a learning-oriented neural network toolkit that helps beginners gain a deep understanding of the underlying principles of deep learning (such as backpropagation and gradient descent) by implementing an automatic differentiation engine and a multi-layer perceptron from scratch, rather than just staying at the API calling level. The core components of the project include an automatic differentiation engine, neural network layer abstraction, training scripts, and visualization tools, aiming to fill the gap in developers' understanding of underlying principles.

In the current deep learning field, most developers rely on high-level APIs of frameworks like PyTorch and TensorFlow to improve efficiency, but lack a deep understanding of the underlying working principles of neural networks. This project is positioned as a learning and experimental tool (not a production-grade solution). Its "from scratch" philosophy allows learners to master the mathematical essence of concepts such as backpropagation and gradient descent by implementing core components themselves, which is crucial for subsequent model optimization, debugging, and new architecture design.

The project adopts a modular structure, with core components including an automatic differentiation engine, neural network layer abstraction, training scripts, and visualization tools. The automatic differentiation engine is the cornerstone, designed in a style similar to Micrograd: the Value class encapsulates scalar values and gradient information, supports basic operations such as addition and multiplication, and records the computational graph structure to automatically apply the chain rule for gradient calculation during backpropagation. Although this design is less efficient than tensor-based implementations, it is intuitive and easy to understand, making it suitable for learning.

Based on the automatic differentiation engine, the project implements complete neural network components: the Neuron class (including weights, biases, and activation functions), the Layer class (composed of multiple neurons, handling forward propagation of the layer), and the MLP class (connecting multiple layers in series to form a complete perceptron). The default architecture consists of an input layer followed by two 64-dimensional hidden layers, with a final output of 10 classification results. The structure is simple but can demonstrate the basic working principles of neural networks.

The training script covers the entire process of data loading, forward propagation, loss calculation, backpropagation, and parameter update. It uses a simple gradient descent optimizer with a learning rate of 0.01 and trains for 5 epochs by default. To ensure reproducibility, a fixed random seed is set. During training, the loss of each sample and the aggregated loss per epoch are printed, but there is room for improvement in the current loss calculation method.

The project has the following issues: loss calculation uses integer labels instead of one-hot encoding, and there is a lack of stable implementations of softmax activation and cross-entropy loss. Improvement suggestions include: using one-hot encoded labels for loss calculation, implementing a stable softmax + cross-entropy combination, adding configurable learning rates, introducing training/validation set splitting, and reporting accuracy. These improvements can enhance training quality and help learners gain a deeper understanding of the nature of classification problems.

Short-term goal: Optimize scalar operations to tensor operations, using NumPy vectorization to improve efficiency (while maintaining code readability). Long-term goal: Based on the optimized differentiation library, implement more types of neural network architectures by referring to research papers in the resources folder. This progressive path is suitable for deep learning beginners to learn in depth.