Self-Normalizing Neural Networks and Explainable AI: Building a Transparent Gradient Flow Automatic Differentiation Engine from Scratch

The igor-pw/SNN-XAI-Engine project (released on GitHub on June 15, 2026) combines Self-Normalizing Neural Networks (SNN) with a transparent automatic differentiation engine to address the interpretability issue in deep learning. Key features include: SNN enables deep network training without batch normalization via the SELU activation function and special weight initialization; the directed graph-based automatic differentiation engine provides fully transparent gradient flow, supporting explainable AI applications such as sensitivity analysis and feature attribution.

Deep learning is widely applied in critical fields like healthcare and autonomous driving, but the non-interpretability of 'black-box' models has become a hidden risk. This project approaches the problem from two aspects: 1. Self-Normalizing Neural Networks (SNN): Automatically maintain stable mean and variance of activation values through special activation functions and weight initialization; 2. Transparent Gradient Flow: The directed graph-based automatic differentiation engine makes gradient calculation at each layer clearly visible.The combination of the two provides a unique perspective for understanding the internal mechanisms of networks.

Normalization Issues in Deep Networks

In deep networks, the shift in activation value distribution leads to problems like gradient vanishing/explosion and slow convergence. Traditional batch normalization relies on batch statistics and has drawbacks such as complex deployment.

Core Mechanisms of SNN

• SELU Activation Function: The formula is selu(x)=λ*x (x>0) or λ*α*(exp(x)-1) (x≤0), where λ≈1.0507 and α≈1.6733, which can make the output tend to zero mean and unit variance.
• Weight Initialization: Orthogonal initialization or Gaussian initialization with specific variance is required to ensure stable variance of activation values.

Advantages of SNN

No need for batch normalization, suitable for deep networks, solid theoretical foundation, consistent training and inference behavior.

Computational Graph and Backpropagation

• Directed Graph Representation: Nodes represent tensors/operations, edges represent data dependencies, supporting flexible topology and visualization.
• Backpropagation Steps: Forward computation → Gradient initialization → Reverse topological traversal → Local gradient calculation → Chain rule propagation → Gradient accumulation.

Transparent Gradient Flow (XAI)

• Features: Track the magnitude and direction of gradients layer by layer, analyze contribution paths, detect gradient anomalies.
• Applications: Sensitivity analysis (partial derivatives of input features to output), feature attribution (e.g., Integrated Gradients), network profiling, adversarial sample detection.

Core Components

• Tensor Class: Stores multi-dimensional arrays, supports automatic gradient tracking and association with computational graphs.
• Operation Class: Encapsulates mathematical operations, including forward computation and backpropagation logic.
• Engine Class: Manages computational graphs, topological sorting, and executes forward/backward propagation.

Modular Design

Divided into core (core implementation), nn (neural network layers), optim (optimizers), and viz (visualization tools) modules.

Usage Examples

• Building a Network: Combine Linear and SELU layers via Sequential, apply SNN initialization.
• Training and Gradient Check: Compute output and loss via forward propagation, check the mean gradient of each layer after backpropagation.
• Visualization: Draw computational graphs, generate gradient heatmaps.

• Education and Research: Teaching demonstrations of backpropagation, validating new algorithms, exploring decision mechanisms.
• Model Debugging: Locating gradient issues, analyzing the impact of layer configurations, optimizing hyperparameters.
• Security and Auditing: Verifying the rationality of decisions, detecting biases, enhancing adversarial defense capabilities.

Challenges

• Computational Efficiency: Lack of GPU optimization, memory reuse mechanisms, and operator fusion; suitable for small-scale experiments.
• Functional Completeness: No support for automatic parallelism, distributed training, or advanced optimizers.
• SNN Limitations: Mainly applicable to fully connected networks and sensitive to initialization.

Future Directions

• Functional Expansion: Support for convolutional layers, recurrent layers, and attention mechanisms.
• Performance Optimization: Numba acceleration, GPU support, graph optimization.
• XAI Enhancement: Interactive visualization, comparative analysis, support for Concept Activation Vectors.

This project demonstrates the practical value of SNN and the educational significance of transparent automatic differentiation. For learners, it is an excellent resource to understand backpropagation and network mechanisms; for researchers, it provides an extensible experimental platform. In today's era of increasing AI complexity, efforts toward transparency are an essential path to building trustworthy AI systems—we should not only pursue accuracy but also not ignore the understanding and control of model behavior.