Implementing a Neural Network from Scratch in C++: Deep Dive into the Mathematical Principles Behind Deep Learning

This article introduces an open-source project called neural-network.cpp, which is implemented entirely without relying on deep learning frameworks—using only C++ and mathematical formulas. Its goal is to help developers understand the mathematical essence of core algorithms like backpropagation and gradient descent. The project targets the MNIST handwritten digit recognition task; by building from scratch, readers can deeply grasp the underlying principles of deep learning.

This project was inspired by 3Blue1Brown's neural network video series. Its core philosophy is to build a neural network from scratch using only C++ and standard math libraries, without any off-the-shelf deep learning frameworks. The design goal is to recognize 28×28 pixel MNIST handwritten digit images; through this task, one can deeply understand core mechanisms like forward propagation, backpropagation, and gradient descent.

The network uses a 4-layer structure with the sigmoid activation function (formula: $\sigma(x) = \frac{1}{1 + e^{-x}}$). The input is a 28×28 grayscale image ($x \in [0,1]^{28×28}$), and the output is a 10-class predicted probability distribution ($\hat{y} \in [0,1]^{10}$). Forward propagation processes samples in batches via matrix operations: multiple sample activation vectors are stacked into a matrix, and matrix multiplication is used to optimize computational efficiency. The formula for the activation value of the i-th neuron in layer L is: $a_i^{(L)} = \sigma\left(\sum_{j=0}^{n-1} w_{i,j}^{(L)}a_j^{(L-1)} + b_i^{(L)}\right)$.

The project uses Mean Squared Error (MSE) as the cost function: the cost for a single sample is $C_x = \sum_{i=0}^{n_L-1} (a_i^{(L)} - y_i)^2$, and the average cost for a batch is $C = \frac{1}{|B|}\sum_{x \in B} C_x$. The core of backpropagation is using the chain rule to compute partial derivatives of the cost with respect to parameters. For example, the weight gradient: $\frac{\partial C_x}{\partial w_{i,j}^{(L)}} = 2(a_i^{(L)} - y_i) \cdot \sigma'(z_i^{(L)}) \cdot a_j^{(L-1)}$, where $\sigma'(z) = \frac{e^{-z}}{(1+e^{-z})^2}$.

Gradients for hidden layers need to be propagated recursively: the gradient of the activation value in layer L-1 is $\frac{\partial C_x}{\partial a_i^{(L-1)}} = \sum_{j=0}^{n_L-1} \frac{\partial C_x}{\partial a_j^{(L)}} \cdot \sigma'(z_j^{(L)}) \cdot w_{j,i}^{(L)}$. The project builds gradient vectors and iteratively applies gradient descent to update parameters (where $\nabla C$ contains partial derivatives of all weights and biases).

The value of this project includes: 1. Cultivating mathematical intuition: writing backpropagation by hand to understand gradient flow; 2. Understanding performance optimization: the importance of matrix batch processing; 3. Improving debugging skills: each step can be checked and verified; 4. Deepening framework usage: understanding the underlying layers allows for more informed decisions about framework architecture.

The neural-network.cpp project demonstrates that core deep learning concepts are based on solid mathematical foundations. Building from scratch not only teaches you how to construct a network but also helps you understand "why" it is constructed that way. For developers who want to deeply understand deep learning, this exercise is invaluable—it reminds us not to forget to return to the basics and understand the mathematical principles behind frameworks when using them.