Axon: An Educational Practice of Building a Neural Network Library from Scratch Using C++

Axon is a neural network library built entirely from scratch in C++ by mbbrueckner, hosted on GitHub (https://github.com/mbbrueckner/axon, released on 2026-06-14). Its core goal is to provide a clear reference for understanding the underlying principles of deep learning, covering tensor operations, automatic differentiation, layer modules, optimizers, and training loops—rather than competing with industrial frameworks like PyTorch or TensorFlow.

Modern deep learning frameworks (e.g., PyTorch) often abstract away underlying details, treating components like nn.Linear or optim.Adam as black boxes. Axon takes the opposite approach: it explicitly shows how tensors are stored in memory, how gradients propagate backward, and how optimizers update parameters—helping learners grasp the fundamentals behind these abstractions.

Axon's architecture follows classic deep learning framework design:

1. Tensor System: A fully implemented tensor class with explicit memory layout and basic math operations; the project is considering factory methods (e.g., Tensor::zeros()) for better API intuitiveness.
2. Automatic Differentiation: Implements computation graphs and backpropagation, allowing automatic gradient calculation without manual derivation.
3. Layers: Common layers (full connection, activation functions) with a unified interface (forward/backward propagation) for composability.
4. Model Container: Manages layer registration, forward flow, and parameter collection for easy network building.
5. Optimizers: Implements SGD, Adam, etc., to update parameters based on gradients.

Axon adopts solid engineering practices:

• CMake: Cross-platform build system for Linux, macOS, Windows.
• Code Formatting: .clang-format ensures consistent code style.
• Unit Tests: tests directory validates component correctness (critical for numerical libraries).
• CI: GitHub Actions workflows automate testing and code checks to maintain quality.

For learners, Axon offers a step-by-step path:

1. Tensor Operations: Understand memory layout, indexing, and basic math; compare results with NumPy.
2. Automatic Differentiation: Study computation graphs and backprop; verify with manual gradient derivation.
3. Layer Implementation: Build layers like linear, ReLU; validate forward/backward outputs.
4. Full Training: Combine components to train on simple datasets (XOR, MNIST) and debug issues.

Axon differs from industrial frameworks like PyTorch in key aspects:

Dimension	Axon	PyTorch
Goal	Education/Principle Understanding	Production/Fast Development
Performance	Unoptimized	Highly Optimized (CUDA/oneDNN)
Features	Basic Layers & Optimizers	Rich Pre-trained Models & Ecosystem
Usability	Requires底层 Knowledge	High-Level Abstraction
Debugging	Easy to Step Through	Needs Special Tools

They are complementary: PyTorch for productivity, Axon for deep understanding.

Axon has room for growth:

• Convolution Layers: Add Conv2D to handle image data (involves im2col, GEMM optimizations).
• GPU Acceleration: Use CUDA/OpenCL for faster training.
• Sequence Models: Implement RNN/LSTM/Transformer for time-series data.
• Distributed Training: Support multi-GPU/node training for large-scale systems.

Axon is an effective educational tool for understanding the underlying principles of deep learning. In an era of rapid AI iteration, knowing fundamentals is more valuable than API proficiency. For learners, a 'double-track' approach—using PyTorch for projects while reading Axon's code—balances productivity and deep knowledge, laying a solid foundation for future ML engineering.