From Scratch: Implementing Neural Networks in C - Deep Dive into Backpropagation and Leaky ReLU Activation Function

This GitHub project (Machine_Learning_C) developed by jacobbartzen implements the complete process from single-layer perceptrons to multi-layer neural networks from scratch using pure C. It demonstrates core mechanisms like forward propagation, backpropagation, and Leaky ReLU activation function, making it ideal for developers who want to deeply understand the underlying principles of neural networks. The project has no dependencies on external machine learning libraries and clearly presents the working principles of neural networks through concise code.

Original Author and Source

• Original Author/Maintainer: jacobbartzen
• Source Platform: GitHub
• Original Project Title: Machine_Learning_C
• Original Link: https://github.com/jacobbartzen/Machine_Learning_C
• Release Date: 2026-05-25

Project Overview

This is a neural network project implemented purely in C without any dependencies on external machine learning libraries. It includes two core files: perceptron.c (single-layer perceptron) and neuralNetwork.c (multi-layer neural network), helping developers clearly understand core concepts like forward propagation, backpropagation, and activation functions.

Basics of Single-Layer Perceptron

perceptron.c implements neurons that output continuous values (not traditional binary outputs), which can handle regression problems. Workflow: weighted sum of inputs + bias → activation function output; during training, adjust weights based on error (increase weight if prediction is too low, decrease if too high; the magnitude is proportional to input and learning rate).

Extension to Multi-Layer Networks

neuralNetwork.c flexibly configures the number of layers and neurons per layer via the neuronLayers array; modifying the array allows experimenting with different architectures.

Forward Propagation

Data flows layer by layer from the input layer to the output layer. Each layer's neurons receive outputs from the previous layer, perform weighted sum + bias + activation function operations, and pass the results to the next layer until the prediction is output.

Leaky ReLU Solves the "Dead Neuron" Problem

Traditional ReLU sets output to 0 when it's negative, leading to neurons being unable to update weights (death). Leaky ReLU introduces a negative slope of 0.01, allowing tiny weight updates even when output is negative, keeping neurons active, enabling higher learning rates, accelerating convergence, and improving performance.

Backpropagation is the core of training, allocating error responsibility layer by layer from the output layer backward:

1. Calculate total error (output layer responsibility);
2. Hidden layer neuron responsibility = sum of (responsibility of connected neurons in next layer × corresponding weight);
3. Responsibility × derivative of activation function (Leaky ReLU: negative value ×0.01, positive value remains unchanged).

Parameter update: bias is adjusted by learning rate × responsibility; weight is adjusted by learning rate × responsibility × output of previous layer.

• Educational Significance: Pure C code shows how mathematical operations are converted into code, no black-box operations, clearly presenting the essence of neural networks (matrix operations + calculus).
• Practical Functions: Includes early stopping, rudimentary Dropout, data normalization, maximum weight limit, etc.
• Inspirational: More suitable than advanced frameworks like PyTorch/TensorFlow for deep understanding of underlying principles, eliminating the mystery of machine learning.

Machine_Learning_C implements a fully functional neural network in less than a thousand lines of C code, covering from single-layer perceptrons to multi-layer networks, from ReLU to Leaky ReLU, from simple weight updates to complete backpropagation, demonstrating core mechanisms step by step. It is recommended that developers use it as learning material or implement their own neural networks based on it to deeply study the underlying principles.