nn-timeline: An Exploration of the Evolution Timeline of Neural Network Architectures

The nn-timeline project is maintained by surafelml and was released on GitHub (June 9, 2026). It visually presents the historical evolution of neural network architectures from perceptrons to Transformers, helping researchers and learners understand the development context of deep learning, fill the gap of scattered knowledge, and explore its educational value, driving forces of technical evolution, and future directions.

The deep learning field has developed rapidly, undergoing multiple paradigm shifts—from AlexNet's breakthrough performance in the ImageNet competition in 2012, to the proposal of the Transformer architecture in 2017, and then to large models with hundreds of billions of parameters. New researchers need to understand the evolution context to grasp trends, but relevant knowledge is scattered across papers, blogs, and courses, lacking systematic organization. The nn-timeline project aims to fill this gap through visualization and help build a holistic understanding.

Perceptron Era (1950s-1960s)

Neural networks began with McCulloch and Pitts' mathematical model of neurons in 1943; Rosenblatt proposed the perceptron in 1958, laying the foundation for learning theory.

Multi-Layer Perceptron (MLP) and Backpropagation (1980s)

The backpropagation algorithm was popularized in 1986, making MLP training possible; LeNet achieved success in handwritten digit recognition in 1989.

Hibernation Period (1990s-2000s)

Shallow methods like SVM dominated, while technologies like RBM and DBN accumulated foundational knowledge.

Renaissance Period (2012-2014)

AlexNet won in 2012; VGGNet and GoogLeNet in 2014; ResNet solved the gradient vanishing problem in 2015.

Evolution of Sequence Modeling

RNN/LSTM/GRU dominated sequence tasks; the attention mechanism improved long-range dependencies; the Transformer abandoned the recurrent structure in 2017.

Transformer Era (2017-Present)

Cross-domain applications include BERT, GPT, ViT, multimodal models (CLIP/DALL-E), and AlphaFold2.

nn-timeline helps build a global perspective and understand the evolutionary logic of technical choices; identify patterns of technological development (hardware driving algorithms, profound impact of simple ideas); inspire innovation and avoid reinventing the wheel; connect the community and promote knowledge inheritance.

Driving Forces: Computing power (GPU/TPU/distributed frameworks), data scale (ImageNet, etc.), open-source culture (TensorFlow/PyTorch, etc.). Resistance: Challenges in interpretability, resource inequality, ethical considerations (abuse risks and social impacts).

Cutting-edge directions include efficiency optimization (sparse attention, model compression, neural architecture search), multimodal unification, integration of neural symbols, and continuous learning. nn-timeline reminds us that understanding the evolution process is an important lesson for becoming an excellent AI researcher.