Building a Deep Learning Library from Scratch: In-Depth Analysis of the ml-by-hand Project

ml-by-hand is an open-source deep learning library built from scratch. Its core goal is to reveal the inner workings of deep learning models by exposing every mathematical detail. The project not only implements an automatic differentiation engine but also includes complete implementations of complex models like GPT-2, Transformer, and ResNet, helping learners understand deep learning from first principles. Following Feynman's philosophy of "What I cannot create, I do not understand", this project is a valuable resource for deep learning education and practice.

Mainstream deep learning frameworks (e.g., PyTorch, TensorFlow) provide highly abstract APIs, which are convenient but lead to many practitioners lacking understanding of the mathematical principles and computational details behind the models. The ml-by-hand project was created to fill this gap. Adhering to Feynman's famous quote, it aims to make every mathematical operation and gradient calculation clearly visible by building a complete library from scratch, eliminating the "black box" and helping learners truly understand how deep learning works.

The core of ml-by-hand is the automatic differentiation (Autograd) engine, which is the cornerstone of modern deep learning frameworks. Automatic differentiation accurately calculates derivatives by tracking the computation graph and applying the chain rule, allowing neural networks to learn from errors and adjust parameters. Initially inspired by Micrograd, the project gradually added features. After implementing tensor-level basic operations, subsequent model building became natural, and the incremental development makes the code easier to understand and learn.

ml-by-hand follows four design principles:

1. Learning by Doing: All formulas and calculations are explicitly derived in code, with no hidden gradient computations, helping to understand backpropagation;
2. Learning Over Optimization: Focus on underlying mathematics and algorithms. Although it does not pursue speed optimization, it can still train GPT models on a single-core CPU;
3. PyTorch-like API: Reduces the learning threshold. Those familiar with PyTorch can switch seamlessly, making it easy to verify correctness;
4. Minimal Dependencies: Uses NumPy by default, with optional MLX/CuPy. PyTorch is only used for gradient verification in unit tests.

ml-by-hand includes a rich set of models and examples:

• Transformer/GPT Series: Complete implementations of the original Transformer, BPE tokenizer, and GPT-1/2;
• Traditional Tasks: Linear/polynomial regression, MNIST/CIFAR classification, binary classification on the breast cancer dataset;
• CNN and ResNet: Implementations of convolution/pooling layers, and ResNet skip connections to handle gradient vanishing;
• RNN/LSTM: Movie sentiment analysis, neural Turing machines, Seq2Seq summarization tasks; In addition, the linear regression training code example clearly shows the process of forward propagation, loss calculation, backpropagation, and parameter update, with annotations explaining the mathematical principles.

The greatest value of ml-by-hand is providing a transparent learning environment where users can see the details of gradient calculation and propagation (instead of just calling .backward()). It is of great value to deep learning beginners, researchers, educators, and engineers. The project proves that the learning method of building from scratch is effective, and it reminds us not to forget the underlying basics when pursuing abstraction and scale— to make better use of abstract tools, we must first understand their underlying working principles.