Neural Network Learning Notes: Exploration of Implementation from Perceptron to Deep Networks

This GitHub project by Sergey-Dubinin (repo link: https://github.com/Sergey-Dubinin/Neural-Networks, updated 2026-05-30) records the implementation journey from basic perceptron to complex deep networks. It covers core concepts like forward propagation, backward propagation, activation functions, optimization algorithms, and regularization with Python code, providing a complete reference for learners to bridge theory and practice.

Neural networks are inspired by biological systems (neurons, synapses). The perceptron, the simplest network, uses a step function but can't solve nonlinear problems like XOR. Multilayer Perceptrons (MLP) add hidden layers to overcome this; the universal approximation theorem (1989) proves a single hidden layer with enough neurons can approximate any continuous function.

Forward Propagation: Computes output via matrix operations (code example included). Backward Propagation: Uses chain rule to calculate gradients for parameter updates (code example). Activation Functions: Sigmoid (smooth but gradient vanishing), tanh (zero-centered), ReLU (efficient, relieves gradient vanishing), Leaky ReLU (solves dead ReLU). Loss Functions: MSE (regression), cross-entropy (classification, paired with softmax).

Optimizers: Gradient descent variants (batch, SGD, mini-batch); advanced optimizers like Momentum (accelerate convergence), AdaGrad (adaptive learning rate), RMSprop (improved AdaGrad), Adam (combines Momentum & RMSprop). Regularization: L1/L2 (weight decay), Dropout (randomly drop neurons), Early Stopping (stop training when validation loss stops improving).

Architecture Design: Input size = feature dim, output size = class count/regression target; hidden layers usually decrease (e.g.,128→64→32). Weight Init: Xavier (stable variance for sigmoid/tanh), He (for ReLU). Batch Norm: Standardizes layer inputs to speed training. Learning Rate Scheduling: Decay or cosine annealing to adjust learning rate dynamically.

Debugging: Monitor loss/accuracy curves, gradient check (compare numerical & analytical gradients). Visualization: Weight visualization (understand learned features). Applications: MNIST handwritten digit recognition (classification), house price prediction (regression), customer churn prediction (binary classification).

Resources: Michael Nielsen's free book Neural Networks & Deep Learning, 3Blue1Brown's visualization series, Stanford CS231n. Advanced Directions: CNN (image processing), RNN (sequence data), Transformer (attention), GAN/VAE (generative models). Conclusion: Hands-on implementation helps internalize neural network principles; continuous learning is key to exploring advanced topics.