Implementing Micrograd from Scratch: Building an Automatic Differentiation Engine and Neural Network with Pure Python

micrograd-from-scratch is an educational open-source project based on Andrej Karpathy's Micrograd. It implements an automatic differentiation engine and neural network library from scratch using pure Python. Through concise code, the project demonstrates the core principles of backpropagation, helping learners gain an in-depth understanding of the underlying mechanisms of deep learning, making it an excellent learning resource.

Deep learning frameworks (such as PyTorch and TensorFlow) simplify the development process, but their high level of encapsulation leads to practitioners having only a superficial understanding of the underlying principles. Automatic differentiation is a core technology of these frameworks; understanding its principles helps in debugging and optimizing models, and is a necessary step to master backpropagation and gradient descent. The micrograd-from-scratch project was created for this purpose.

The project is implemented in pure Python with no external dependencies, following the "minimum viable implementation" philosophy, using a few hundred lines of code to demonstrate core mechanisms. Automatic differentiation is based on the chain rule; micrograd implements reverse-mode automatic differentiation, which is more efficient when calculating gradients of scalar functions with respect to multiple inputs, making it suitable for neural network training scenarios.

Value Class: Basic Unit of the Computational Graph

Each Value object encapsulates a scalar value, records parent nodes, operation type, and backpropagation logic.

Forward Propagation: Building the Computational Graph

When an operation is executed, a new Value node is created, operation information is recorded, and the computation tree is built recursively.

Backpropagation: Gradient Calculation

1. Topological sorting to determine the order of nodes; 2. Initialize the output gradient to 1; 3. Traverse in reverse order and call the _backward function; 4. Apply the chain rule to accumulate gradients for parent nodes.

Neuron Class: Single Neuron

Maintains weights and biases, computes the weighted sum, then outputs via tanh activation (non-linear and has a simple derivative).

Layer Class: Fully Connected Layer

Composed of multiple neurons; input is passed to all neurons to generate output.

MLP Class: Multilayer Perceptron

Allows specifying the size of each layer, automatically constructing the structure of input layer, hidden layers, and output layer.

The project includes training examples with the following steps:

1. Data Preparation: Create a binary classification dataset;
2. Model Construction: Initialize the MLP network;
3. Forward Propagation: Compute model output;
4. Loss Calculation: Use mean squared error;
5. Backpropagation: Call backward() to compute gradients;
6. Parameter Update: Update weights via gradient descent;
7. Iterative Optimization: Repeat until convergence.

Learning Value:

• Beginners: Master core concepts of automatic differentiation and neural networks;
• Experienced practitioners: Understand the internal mechanisms of frameworks and improve debugging capabilities;
• Researchers: A lightweight experimental platform to verify algorithms. Expansion Directions:
• Tensor support;
• More activation functions (ReLU, Sigmoid);
• Optimizers (SGD with Momentum, Adam);
• Convolutional layers;
• GPU acceleration (Numba/CuPy).

micrograd-from-scratch demonstrates the core technology of deep learning—automatic differentiation—in a concise way. By implementing this project, learners can understand the mathematical principles of backpropagation and the ingenuity of framework design. It is recommended for practitioners who want to deeply understand deep learning rather than just "using pre-built libraries".