Building Deep Learning from Scratch: A Learner's Journey to Implementing a Micro-Gradient Engine

Cedric-kopp, an interdisciplinary learner, built a scalar-valued automatic differentiation engine and a multi-layer perceptron from scratch by reproducing Andrej Karpathy's "Neural Networks: Zero to Hero" series, gaining an in-depth understanding of the underlying principles of neural networks and backpropagation. The project documents the complete learning process (including intermediate steps and failed attempts) and uses PyTorch cross-validation to ensure implementation correctness.

The author comes from a quantitative analysis/audit background and is pursuing a master's degree in analytics and artificial intelligence, focusing on a thorough understanding of fundamental principles rather than just API calls. The core goal of the project is to establish a deep understanding of the underlying working mechanisms of neural networks and backpropagation, preserving intermediate steps, failed attempts, and dead ends as a learning log to show the real reasoning process.

Implemented a scalar automatic differentiation engine and multi-layer perceptron via Jupyter notebooks, with key technical points including:

1. Value class: Stores scalar values, operations, predecessor nodes, and gradients to build the computation graph;
2. Operation backpropagation: Manually implements gradient calculation logic for operations like addition and multiplication;
3. Topological sorting: Ensures gradients propagate from output to input in the correct order;
4. Gradient accumulation fix: Resolves the gradient overwriting issue when nodes are used multiple times;
5. Complete training loop: Implements forward propagation, loss calculation, backpropagation, and parameter updates.

Adopted rigorous learning methods:

• Feynman Learning Technique: Annotate derivation processes and error causes in one's own words;
• PyTorch cross-validation: Compare results to verify implementation correctness and understand industrial framework design;
• Preserve learning traces: Deliberately keep intermediate steps and errors to help other learners.

The project is positioned as a reproduction of Karpathy's educational materials (not original, for academic honesty); the tech stack uses Python's basic scientific computing stack, with advantages: minimal dependencies (no GPU required), transparency (no black-box operations), and strong portability.

Future plans include completing the makemore project and GPT notebook, with long-term interests in mechanistic interpretability and model alignment; insights for learners:

• Fundamental principles are the cornerstone of long-term development;
• Documenting the learning process helps with review and benefits others;
• Interdisciplinary perspectives enrich the diversity of the field.

Hands-on implementation is the key to bridging the gap between API calls and principle understanding. This project provides a valuable learning path for in-depth understanding of the underlying mechanisms of neural networks. The best way to learn is to implement concepts you understand by hand, and test the depth of your understanding through detailed problems.